{"thread":{"id":"42904","subject":"[PATCH v1 0/3] Git filter protocol","startedAt":"2016-07-22T15:49:16Z","lastAt":"2016-08-07T00:13:38Z","messageCount":77,"participants":["larsxschneider@gmail.com","Junio C Hamano","Torsten Bögershausen","Ramsay Jones","Jakub Narębski","Eric Wong","Lars Schneider","Duy Nguyen","Remi Galan Alfonso","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"291963","messageId":"20160722154900.19477-1-larsxschneider@gmail.com","threadId":"42904","inReplyTo":null,"subject":"[PATCH v1 0/3] Git filter protocol","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-22T15:48:57Z","receivedAt":"2016-07-22T15:49:16Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nHi,\n\nGit's clean/smudge mechanism invokes an external filter process for every\nsingle blob that is affected by a filter. If Git filters a lot of blobs\nthen the startup time of the external filter processes can become a\nsignificant part of the overall Git execution time. This patch series\naddresses this problem.\n\nThe first two patches are cleanup patches which are not really necessary\nfor the feature. The third patch is the relevant one :-)\n\nYou will notice that I do not check the exact number of \"clean\" filter\ninvocations in the tests. The reason is that Git calls \"clean\" multiple\ntimes (up to 4 times for the same blob!). I posted a patch to demonstrate\nthe problem and I would prefer to tackle the problem in a seperate patch\nseries: http://thread.gmane.org/gmane.comp.version-control.git/300028/\n\nThe main reason for this Git core patch is to speed up Git large file\nextensions such as git-annex or Git LFS. I proposed an according Git LFS\npatch here: https://github.com/github/git-lfs/pull/1382\n\nIn addition to the Git core tests, all Git LFS integration tests run clean\non my machine.\n\nThanks,\nLars\n\n\nLars Schneider (3):\n convert: quote filter names in error messages\n convert: modernize tests\n convert: add filter..useProtocol option\n\n Documentation/gitattributes.txt | 41 ++++++-\n convert.c | 222 +++++++++++++++++++++++++++++++++++---\n t/t0021-conversion.sh | 232 ++++++++++++++++++++++++++++++++++------\n t/t0021/rot13.pl | 80 ++++++++++++++\n 4 files changed, 531 insertions(+), 44 deletions(-)\n create mode 100755 t/t0021/rot13.pl\n\n--\n2.9.0\n\n"},{"id":"291964","messageId":"20160722154900.19477-2-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160722154900.19477-1-larsxschneider@gmail.com","subject":"[PATCH v1 1/3] convert: quote filter names in error messages","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-22T15:48:58Z","receivedAt":"2016-07-22T15:49:20Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nGit filter with spaces (e.g. `filter.sh foo`) are hard to read in\nerror messages. Quote them to improve the readability.\n\nSigned-off-by: Lars Schneider \n---\n convert.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex b1614bf..522e2c5 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -397,7 +397,7 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \tchild_process.out = out;\n \n \tif (start_command(&child_process))\n-\t\treturn error(\"cannot fork to run external filter %s\", params->cmd);\n+\t\treturn error(\"cannot fork to run external filter '%s'\", params->cmd);\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n@@ -415,13 +415,13 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \tif (close(child_process.in))\n \t\twrite_err = 1;\n \tif (write_err)\n-\t\terror(\"cannot feed the input to external filter %s\", params->cmd);\n+\t\terror(\"cannot feed the input to external filter '%s'\", params->cmd);\n \n \tsigchain_pop(SIGPIPE);\n \n \tstatus = finish_command(&child_process);\n \tif (status)\n-\t\terror(\"external filter %s failed %d\", params->cmd, status);\n+\t\terror(\"external filter '%s' failed %d\", params->cmd, status);\n \n \tstrbuf_release(&cmd);\n \treturn (write_err || status);\n@@ -462,15 +462,15 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \t\treturn 0;\t/* error was already reported */\n \n \tif (strbuf_read(&nbuf, async.out, len) < 0) {\n-\t\terror(\"read from external filter %s failed\", cmd);\n+\t\terror(\"read from external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \tif (close(async.out)) {\n-\t\terror(\"read from external filter %s failed\", cmd);\n+\t\terror(\"read from external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \tif (finish_async(&async)) {\n-\t\terror(\"external filter %s failed\", cmd);\n+\t\terror(\"external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \n-- \n2.9.0\n\n"},{"id":"291965","messageId":"20160722154900.19477-3-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160722154900.19477-1-larsxschneider@gmail.com","subject":"[PATCH v1 2/3] convert: modernize tests","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-22T15:48:59Z","receivedAt":"2016-07-22T15:49:23Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nUse `test_config` to set the config, check that files are empty with\n`test_must_be_empty`, compare files with `test_cmp`, and remove spaces\nafter \">\".\n\nSigned-off-by: Lars Schneider \n---\n t/t0021-conversion.sh | 62 +++++++++++++++++++++++++--------------------------\n 1 file changed, 31 insertions(+), 31 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 7bac2bc..a05a8d2 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -13,8 +13,8 @@ EOF\n chmod +x rot13.sh\n \n test_expect_success setup '\n-\tgit config filter.rot13.smudge ./rot13.sh &&\n-\tgit config filter.rot13.clean ./rot13.sh &&\n+\ttest_config filter.rot13.smudge ./rot13.sh &&\n+\ttest_config filter.rot13.clean ./rot13.sh &&\n \n \t{\n \t echo \"*.t filter=rot13\"\n@@ -38,8 +38,8 @@ script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n \n test_expect_success check '\n \n-\tcmp test.o test &&\n-\tcmp test.o test.t &&\n+\ttest_cmp test.o test &&\n+\ttest_cmp test.o test.t &&\n \n \t# ident should be stripped in the repository\n \tgit diff --raw --exit-code :test :test.i &&\n@@ -47,10 +47,10 @@ test_expect_success check '\n \tembedded=$(sed -ne \"$script\" test.i) &&\n \ttest \"z$id\" = \"z$embedded\" &&\n \n-\tgit cat-file blob :test.t > test.r &&\n+\tgit cat-file blob :test.t >test.r &&\n \n-\t./rot13.sh < test.o > test.t &&\n-\tcmp test.r test.t\n+\t./rot13.sh < test.o >test.t &&\n+\ttest_cmp test.r test.t\n '\n \n # If an expanded ident ever gets into the repository, we want to make sure that\n@@ -130,7 +130,7 @@ test_expect_success 'filter shell-escaped filenames' '\n \n \t# delete the files and check them out again, using a smudge filter\n \t# that will count the args and echo the command-line back to us\n-\tgit config filter.argc.smudge \"sh ./argc.sh %f\" &&\n+\ttest_config filter.argc.smudge \"sh ./argc.sh %f\" &&\n \trm \"$normal\" \"$special\" &&\n \tgit checkout -- \"$normal\" \"$special\" &&\n \n@@ -141,7 +141,7 @@ test_expect_success 'filter shell-escaped filenames' '\n \ttest_cmp expect \"$special\" &&\n \n \t# do the same thing, but with more args in the filter expression\n-\tgit config filter.argc.smudge \"sh ./argc.sh %f --my-extra-arg\" &&\n+\ttest_config filter.argc.smudge \"sh ./argc.sh %f --my-extra-arg\" &&\n \trm \"$normal\" \"$special\" &&\n \tgit checkout -- \"$normal\" \"$special\" &&\n \n@@ -154,9 +154,9 @@ test_expect_success 'filter shell-escaped filenames' '\n '\n \n test_expect_success 'required filter should filter data' '\n-\tgit config filter.required.smudge ./rot13.sh &&\n-\tgit config filter.required.clean ./rot13.sh &&\n-\tgit config filter.required.required true &&\n+\ttest_config filter.required.smudge ./rot13.sh &&\n+\ttest_config filter.required.clean ./rot13.sh &&\n+\ttest_config filter.required.required true &&\n \n \techo \"*.r filter=required\" >.gitattributes &&\n \n@@ -165,17 +165,17 @@ test_expect_success 'required filter should filter data' '\n \n \trm -f test.r &&\n \tgit checkout -- test.r &&\n-\tcmp test.o test.r &&\n+\ttest_cmp test.o test.r &&\n \n \t./rot13.sh expected &&\n \tgit cat-file blob :test.r >actual &&\n-\tcmp expected actual\n+\ttest_cmp expected actual\n '\n \n test_expect_success 'required filter smudge failure' '\n-\tgit config filter.failsmudge.smudge false &&\n-\tgit config filter.failsmudge.clean cat &&\n-\tgit config filter.failsmudge.required true &&\n+\ttest_config filter.failsmudge.smudge false &&\n+\ttest_config filter.failsmudge.clean cat &&\n+\ttest_config filter.failsmudge.required true &&\n \n \techo \"*.fs filter=failsmudge\" >.gitattributes &&\n \n@@ -186,9 +186,9 @@ test_expect_success 'required filter smudge failure' '\n '\n \n test_expect_success 'required filter clean failure' '\n-\tgit config filter.failclean.smudge cat &&\n-\tgit config filter.failclean.clean false &&\n-\tgit config filter.failclean.required true &&\n+\ttest_config filter.failclean.smudge cat &&\n+\ttest_config filter.failclean.clean false &&\n+\ttest_config filter.failclean.required true &&\n \n \techo \"*.fc filter=failclean\" >.gitattributes &&\n \n@@ -197,8 +197,8 @@ test_expect_success 'required filter clean failure' '\n '\n \n test_expect_success 'filtering large input to small output should use little memory' '\n-\tgit config filter.devnull.clean \"cat >/dev/null\" &&\n-\tgit config filter.devnull.required true &&\n+\ttest_config filter.devnull.clean \"cat >/dev/null\" &&\n+\ttest_config filter.devnull.required true &&\n \tfor i in $(test_seq 1 30); do printf \"%1048576d\" 1; done >30MB &&\n \techo \"30MB filter=devnull\" >.gitattributes &&\n \tGIT_MMAP_LIMIT=1m GIT_ALLOC_LIMIT=1m git add 30MB\n@@ -207,7 +207,7 @@ test_expect_success 'filtering large input to small output should use little mem\n test_expect_success 'filter that does not read is fine' '\n \ttest-genrandom foo $((128 * 1024 + 1)) >big &&\n \techo \"big filter=epipe\" >.gitattributes &&\n-\tgit config filter.epipe.clean \"echo xyzzy\" &&\n+\ttest_config filter.epipe.clean \"echo xyzzy\" &&\n \tgit add big &&\n \tgit cat-file blob :big >actual &&\n \techo xyzzy >expect &&\n@@ -215,20 +215,20 @@ test_expect_success 'filter that does not read is fine' '\n '\n \n test_expect_success EXPENSIVE 'filter large file' '\n-\tgit config filter.largefile.smudge cat &&\n-\tgit config filter.largefile.clean cat &&\n+\ttest_config filter.largefile.smudge cat &&\n+\ttest_config filter.largefile.clean cat &&\n \tfor i in $(test_seq 1 2048); do printf \"%1048576d\" 1; done >2GB &&\n \techo \"2GB filter=largefile\" >.gitattributes &&\n \tgit add 2GB 2>err &&\n-\t! test -s err &&\n+\ttest_must_be_empty err &&\n \trm -f 2GB &&\n \tgit checkout -- 2GB 2>err &&\n-\t! test -s err\n+\ttest_must_be_empty err\n '\n \n test_expect_success \"filter: clean empty file\" '\n-\tgit config filter.in-repo-header.clean \"echo cleaned && cat\" &&\n-\tgit config filter.in-repo-header.smudge \"sed 1d\" &&\n+\ttest_config filter.in-repo-header.clean \"echo cleaned && cat\" &&\n+\ttest_config filter.in-repo-header.smudge \"sed 1d\" &&\n \n \techo \"empty-in-worktree filter=in-repo-header\" >>.gitattributes &&\n \t>empty-in-worktree &&\n@@ -240,8 +240,8 @@ test_expect_success \"filter: clean empty file\" '\n '\n \n test_expect_success \"filter: smudge empty file\" '\n-\tgit config filter.empty-in-repo.clean \"cat >/dev/null\" &&\n-\tgit config filter.empty-in-repo.smudge \"echo smudged && cat\" &&\n+\ttest_config filter.empty-in-repo.clean \"cat >/dev/null\" &&\n+\ttest_config filter.empty-in-repo.smudge \"echo smudged && cat\" &&\n \n \techo \"empty-in-repo filter=empty-in-repo\" >>.gitattributes &&\n \techo dead data walking >empty-in-repo &&\n-- \n2.9.0\n\n"},{"id":"291966","messageId":"20160722154900.19477-4-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160722154900.19477-1-larsxschneider@gmail.com","subject":"[PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-22T15:49:00Z","receivedAt":"2016-07-22T15:49:27Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nGit's clean/smudge mechanism invokes an external filter process for every\nsingle blob that is affected by a filter. If Git filters a lot of blobs\nthen the startup time of the external filter processes can become a\nsignificant part of the overall Git execution time.\n\nThis patch adds the filter..useProtocol option which, if enabled,\nkeeps the external filter process running and processes all blobs with\nthe following protocol over stdin/stdout.\n\n1. Git starts the filter on first usage and expects a welcome message\nwith protocol version number:\n\tGit <-- Filter: \"git-filter-protocol\\n\"\n\tGit <-- Filter: \"version 1\"\n\n2. Git sends the command (either \"smudge\" or \"clean\"), the filename, the\ncontent size in bytes, and the content separated by a newline character:\n\tGit --> Filter: \"smudge\\n\"\n\tGit --> Filter: \"testfile.dat\\n\"\n\tGit --> Filter: \"7\\n\"\n\tGit --> Filter: \"CONTENT\"\n\n3. The filter is expected to answer with the result content size in\nbytes and the result content separated by a newline character:\n\tGit <-- Filter: \"15\\n\"\n\tGit <-- Filter: \"SMUDGED_CONTENT\"\n\n4. The filter is expected to wait for the next file in step 2, again.\n\nPlease note that the protocol filters do not support stream processing\nwith this implemenatation because the filter needs to know the length of\nthe result in advance. A protocol version 2 could address this in a\nfuture patch.\n\nSigned-off-by: Lars Schneider \nHelped-by: Martin-Louis Bright \n---\n Documentation/gitattributes.txt | 41 +++++++-\n convert.c | 210 ++++++++++++++++++++++++++++++++++++++--\n t/t0021-conversion.sh | 170 ++++++++++++++++++++++++++++++++\n t/t0021/rot13.pl | 80 +++++++++++++++\n 4 files changed, 494 insertions(+), 7 deletions(-)\n create mode 100755 t/t0021/rot13.pl\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 8882a3e..7026d62 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -300,7 +300,10 @@ checkout, when the `smudge` command is specified, the command is\n fed the blob object from its standard input, and its standard\n output is used to update the worktree file. Similarly, the\n `clean` command is used to convert the contents of worktree file\n-upon checkin.\n+upon checkin. By default these commands process only a single\n+blob and terminate. If the setting filter..useProtocol is\n+enabled then Git can process all blobs with a single filter command\n+invocation (see filter protocol below).\n \n One use of the content filtering is to massage the content into a shape\n that is more convenient for the platform, filesystem, and the user to use.\n@@ -375,6 +378,42 @@ substitution. For example:\n ------------------------\n \n \n+Filter Protocol\n+^^^^^^^^^^^^^^^\n+\n+If the setting filter..useProtocol is enabled then Git\n+can process all blobs with a single filter command invocation\n+by talking with the following protocol over stdin/stdout.\n+\n+Git starts the filter on first usage and expects a welcome\n+message with protocol version number:\n+------------------------\n+Git < Filter: \"git-filter-protocol\\n\"\n+Git < Filter: \"version 1\"\n+------------------------\n+\n+Afterwards Git sends a blob command (either \"smudge\" or \"clean\"),\n+the filename, the content size in bytes, and the content separated\n+by a newline character:\n+------------------------\n+Git > Filter: \"smudge\\n\"\n+Git > Filter: \"testfile.dat\\n\"\n+Git > Filter: \"7\\n\"\n+Git > Filter: \"CONTENT\"\n+------------------------\n+\n+The filter is expected to answer with the result content size in\n+bytes and the result content separated by a newline character:\n+------------------------\n+Git < Filter: \"15\\n\"\n+Git < Filter: \"SMUDGED_CONTENT\"\n+------------------------\n+\n+Afterwards the filter is expected to wait for the next blob process\n+command. A demo implementation can be found in `t/t0021/rot13.pl`\n+located in the Git core repository.\n+\n+\n Interaction between checkin/checkout attributes\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \ndiff --git a/convert.c b/convert.c\nindex 522e2c5..91ce86f 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -481,12 +481,188 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \treturn ret;\n }\n \n+static int cmd_process_map_init = 0;\n+static struct hashmap cmd_process_map;\n+\n+struct cmd2process {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tconst char *cmd;\n+\tlong protocol;\n+\tstruct child_process process;\n+};\n+\n+static int cmd2process_cmp(const struct cmd2process *e1,\n+\t\t\t\t\t\t\tconst struct cmd2process *e2,\n+\t\t\t\t\t\t\tconst void *unused)\n+{\n+\treturn strcmp(e1->cmd, e2->cmd);\n+}\n+\n+static struct cmd2process *find_protocol_filter_entry(const char *cmd)\n+{\n+\tstruct cmd2process k;\n+\thashmap_entry_init(&k, strhash(cmd));\n+\tk.cmd = cmd;\n+\treturn hashmap_get(&cmd_process_map, &k, NULL);\n+}\n+\n+static void stop_protocol_filter(struct cmd2process *entry) {\n+\tif (!entry)\n+\t\treturn;\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tclose(entry->process.in);\n+\tclose(entry->process.out);\n+\tsigchain_pop(SIGPIPE);\n+\tfinish_command(&entry->process);\n+\tchild_process_clear(&entry->process);\n+\thashmap_remove(&cmd_process_map, entry, NULL);\n+\tfree(entry);\n+}\n+\n+static struct cmd2process *start_protocol_filter(const char *cmd)\n+{\n+\tint ret = 1;\n+\tstruct cmd2process *entry = NULL;\n+\tstruct child_process *process = NULL;\n+\tstruct strbuf nbuf = STRBUF_INIT;\n+\tstruct string_list split = STRING_LIST_INIT_NODUP;\n+\tconst char *argv[] = { NULL, NULL };\n+\tconst char *header = \"git-filter-protocol\\nversion\";\n+\n+\tentry = xmalloc(sizeof(*entry));\n+\thashmap_entry_init(entry, strhash(cmd));\n+\tentry->cmd = cmd;\n+\tprocess = &entry->process;\n+\n+\tchild_process_init(process);\n+\targv[0] = cmd;\n+\tprocess->argv = argv;\n+\tprocess->use_shell = 1;\n+\tprocess->in = -1;\n+\tprocess->out = -1;\n+\n+\tif (start_command(process)) {\n+\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n+\t\treturn NULL;\n+\t}\n+\tstrbuf_reset(&nbuf);\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n+\tsigchain_pop(SIGPIPE);\n+\n+\tstrbuf_stripspace(&nbuf, 0);\n+\tstring_list_split_in_place(&split, nbuf.buf, ' ', 2);\n+\tret &= split.nr > 1;\n+\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n+\tif (ret) {\n+\t\tentry->protocol = strtol(split.items[1].string, NULL, 10);\n+\t\tswitch (entry->protocol) {\n+\t\t\tcase 1:\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\t\tret = 0;\n+\t\t\t\terror(\"unsupported protocol version %s for external persistent filter '%s'\",\n+\t\t\t\t\tnbuf.buf, cmd);\n+\t\t}\n+\t}\n+\tstring_list_clear(&split, 0);\n+\tstrbuf_release(&nbuf);\n+\n+\tif (!ret) {\n+\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n+\t\treturn NULL;\n+\t}\n+\n+\thashmap_add(&cmd_process_map, entry);\n+\treturn entry;\n+}\n+\n+static int apply_protocol_filter(const char *path, const char *src, size_t len,\n+\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n+\t\t\t\t\t\tconst char *filter_type)\n+{\n+\tint ret = 1;\n+\tstruct cmd2process *entry = NULL;\n+\tstruct child_process *process = NULL;\n+\tstruct stat fileStat;\n+\tstruct strbuf nbuf = STRBUF_INIT;\n+\tsize_t nbuf_len;\n+\tchar *strtol_end;\n+\tchar c;\n+\n+\tif (!cmd || !*cmd)\n+\t\treturn 0;\n+\n+\tif (!dst)\n+\t\treturn 1;\n+\n+\tif (!cmd_process_map_init) {\n+\t\tcmd_process_map_init = 1;\n+\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n+\t} else {\n+\t\tentry = find_protocol_filter_entry(cmd);\n+\t}\n+\n+\tif (!entry){\n+\t\tentry = start_protocol_filter(cmd);\n+\t\tif (!entry) {\n+\t\t\tstop_protocol_filter(entry);\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\tprocess = &entry->process;\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tswitch (entry->protocol) {\n+\t\tcase 1:\n+\t\t\tif (fd >= 0 && !src) {\n+\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n+\t\t\t\tlen = fileStat.st_size;\n+\t\t\t}\n+\t\t\tstrbuf_reset(&nbuf);\n+\t\t\tstrbuf_addf(&nbuf, \"%s\\n%s\\n%zu\\n\", filter_type, path, len);\n+\t\t\tret &= write_str_in_full(process->in, nbuf.buf) > 1;\n+\t\t\tif (len > 0) {\n+\t\t\t\tif (src)\n+\t\t\t\t\tret &= write_in_full(process->in, src, len) == len;\n+\t\t\t\telse if (fd >= 0)\n+\t\t\t\t\tret &= copy_fd(fd, process->in) == 0;\n+\t\t\t\telse\n+\t\t\t\t\tret &= 0;\n+\t\t\t}\n+\n+\t\t\tstrbuf_reset(&nbuf);\n+\t\t\twhile (xread(process->out, &c, 1) == 1 && c != '\\n')\n+\t\t\t\tstrbuf_addchars(&nbuf, c, 1);\n+\t\t\tnbuf_len = (size_t)strtol(nbuf.buf, &strtol_end, 10);\n+\t\t\tret &= (strtol_end != nbuf.buf && errno != ERANGE);\n+\t\t\tstrbuf_reset(&nbuf);\n+\t\t\tif (nbuf_len > 0)\n+\t\t\t\tret &= strbuf_read_once(&nbuf, process->out, nbuf_len) == nbuf_len;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tret = 0;\n+\t}\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (ret) {\n+\t\tstrbuf_swap(dst, &nbuf);\n+\t} else {\n+\t\t// Something went wrong with the protocol filter. Force shutdown!\n+\t\tstop_protocol_filter(entry);\n+\t}\n+\tstrbuf_release(&nbuf);\n+\treturn ret;\n+}\n+\n static struct convert_driver {\n \tconst char *name;\n \tstruct convert_driver *next;\n \tconst char *smudge;\n \tconst char *clean;\n \tint required;\n+\tint use_protocol;\n } *user_convert, **user_convert_tail;\n \n static int read_convert_config(const char *var, const char *value, void *cb)\n@@ -526,6 +702,11 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(\"clean\", key))\n \t\treturn git_config_string(&drv->clean, var, value);\n \n+\tif (!strcmp(\"useprotocol\", key)) {\n+\t\tdrv->use_protocol = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(\"required\", key)) {\n \t\tdrv->required = git_config_bool(var, value);\n \t\treturn 0;\n@@ -823,7 +1004,10 @@ int would_convert_to_git_filter_fd(const char *path)\n \tif (!ca.drv->required)\n \t\treturn 0;\n \n-\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n+\tif (ca.drv->use_protocol)\n+\t\treturn apply_protocol_filter(path, NULL, 0, -1, NULL, ca.drv->clean, \"clean\");\n+\telse\n+\t\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n }\n \n const char *get_convert_attr_ascii(const char *path)\n@@ -857,16 +1041,20 @@ int convert_to_git(const char *path, const char *src, size_t len,\n {\n \tint ret = 0;\n \tconst char *filter = NULL;\n-\tint required = 0;\n+\tint required = 0, use_protocol = 0;\n \tstruct conv_attrs ca;\n \n \tconvert_attrs(&ca, path);\n \tif (ca.drv) {\n \t\tfilter = ca.drv->clean;\n \t\trequired = ca.drv->required;\n+\t\tuse_protocol = ca.drv->use_protocol;\n \t}\n \n-\tret |= apply_filter(path, src, len, -1, dst, filter);\n+\tif (use_protocol)\n+\t\tret |= apply_protocol_filter(path, src, len, -1, dst, filter, \"clean\");\n+\telse\n+\t\tret |= apply_filter(path, src, len, -1, dst, filter);\n \tif (!ret && required)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n@@ -885,13 +1073,19 @@ int convert_to_git(const char *path, const char *src, size_t len,\n void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n \t\t\t enum safe_crlf checksafe)\n {\n+\tint ret = 0;\n \tstruct conv_attrs ca;\n \tconvert_attrs(&ca, path);\n \n \tassert(ca.drv);\n \tassert(ca.drv->clean);\n \n-\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))\n+\tif (ca.drv->use_protocol)\n+\t\tret = apply_protocol_filter(path, NULL, 0, fd, dst, ca.drv->clean, \"clean\");\n+\telse\n+\t\tret = apply_filter(path, NULL, 0, fd, dst, ca.drv->clean);\n+\n+\tif (!ret)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n@@ -904,13 +1098,14 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n {\n \tint ret = 0, ret_filter = 0;\n \tconst char *filter = NULL;\n-\tint required = 0;\n+\tint required = 0, use_protocol = 0;\n \tstruct conv_attrs ca;\n \n \tconvert_attrs(&ca, path);\n \tif (ca.drv) {\n \t\tfilter = ca.drv->smudge;\n \t\trequired = ca.drv->required;\n+\t\tuse_protocol = ca.drv->use_protocol;\n \t}\n \n \tret |= ident_to_worktree(path, src, len, dst, ca.ident);\n@@ -930,7 +1125,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n-\tret_filter = apply_filter(path, src, len, -1, dst, filter);\n+\tif (use_protocol)\n+\t\tret_filter = apply_protocol_filter(path, src, len, -1, dst, filter, \"smudge\");\n+\telse\n+\t\tret_filter |= apply_filter(path, src, len, -1, dst, filter);\n \tif (!ret_filter && required)\n \t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n \ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex a05a8d2..d9077ea 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -268,4 +268,174 @@ test_expect_success 'disable filter with empty override' '\n \ttest_must_be_empty err\n '\n \n+test_expect_success 'required protocol filter should filter data' '\n+\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n+\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n+\ttest_config_global filter.protocol.useprotocol true &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\tgit branch empty &&\n+\n+\t\tcat ../test.o >test.r &&\n+\t\techo \"test22\" >test2.r &&\n+\t\techo \"test333\" >test3.r &&\n+\n+\t\trm -f output.log &&\n+\t\tgit add . &&\n+\t\tsort output.log | uniq -c | sed \"s/^[ ]*//\" >uniq_output.log &&\n+\t\tcat >expected_add.log <<-\\EOF &&\n+\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 [OK]\n+\t\t\t1 IN: clean test2.r 7 [OK] -- OUT: 7 [OK]\n+\t\t\t1 IN: clean test3.r 8 [OK] -- OUT: 8 [OK]\n+\t\t\t1 start\n+\t\t\t1 wrote version\n+\t\tEOF\n+\t\ttest_cmp expected_add.log uniq_output.log &&\n+\n+\t\tprintf \"\" >output.log &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\tsort output.log | uniq -c | sed \"s/^[ ]*//\" |\n+\t\t\tsed \"s/^\\([0-9]\\) IN: clean/x IN: clean/\" >uniq_output.log &&\n+\t\tcat >expected_commit.log <<-\\EOF &&\n+\t\t\tx IN: clean test.r 57 [OK] -- OUT: 57 [OK]\n+\t\t\tx IN: clean test2.r 7 [OK] -- OUT: 7 [OK]\n+\t\t\tx IN: clean test3.r 8 [OK] -- OUT: 8 [OK]\n+\t\t\t1 start\n+\t\t\t1 wrote version\n+\t\tEOF\n+\t\ttest_cmp expected_commit.log uniq_output.log &&\n+\n+\t\tprintf \"\" >output.log &&\n+\t\trm -f test?.r &&\n+\t\tgit checkout . &&\n+\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n+\t\tcat >expected_checkout.log <<-\\EOF &&\n+\t\t\tstart\n+\t\t\twrote version\n+\t\t\tIN: smudge test2.r 7 [OK] -- OUT: 7 [OK]\n+\t\t\tIN: smudge test3.r 8 [OK] -- OUT: 8 [OK]\n+\t\tEOF\n+\t\ttest_cmp expected_checkout.log smudge_output.log &&\n+\n+\t\tgit checkout empty &&\n+\n+\t\tprintf \"\" >output.log &&\n+\t\tgit checkout master &&\n+\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n+\t\tcat >expected_checkout_master.log <<-\\EOF &&\n+\t\t\tstart\n+\t\t\twrote version\n+\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 [OK]\n+\t\t\tIN: smudge test2.r 7 [OK] -- OUT: 7 [OK]\n+\t\t\tIN: smudge test3.r 8 [OK] -- OUT: 8 [OK]\n+\t\tEOF\n+\t\ttest_cmp expected_checkout_master.log smudge_output.log\n+\t)\n+'\n+\n+test_expect_success EXPENSIVE 'protocol filter large file' '\n+\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n+\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"2GB filter=largefile\" >.gitattributes &&\n+\t\tfor i in $(test_seq 1 2048); do printf \"%1048576d\" 1; done >2GB &&\n+\t\tgit add 2GB 2>err &&\n+\t\ttest_must_be_empty err &&\n+\t\trm -f 2GB &&\n+\t\tgit checkout -- 2GB 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_expect_success 'required protocol filter should fail with clean' '\n+\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n+\ttest_config_global filter.protocol.useprotocol true &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcat ../test.o >test.r &&\n+\t\techo \"this is going to fail\" >clean-write-fail.r &&\n+\t\techo \"test333\" >test3.r &&\n+\n+\t\t# Note: There are three clean paths in convert.c we just test one here.\n+\t\ttest_must_fail git add .\n+\t)\n+'\n+\n+test_expect_success 'protocol filter should restart after failure' '\n+\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n+\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n+\ttest_config_global filter.protocol.useprotocol true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcat ../test.o >test.r &&\n+\t\techo \"1234567\" >test2.o &&\n+\t\tcat test2.o >test2.r &&\n+\t\techo \"this is going to fail\" >smudge-write-fail.o &&\n+\t\tcat smudge-write-fail.o >smudge-write-fail.r &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\trm -f *.r &&\n+\n+\t\tprintf \"\" >output.log &&\n+\t\tgit checkout . &&\n+\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n+\t\tcat >expected_checkout_master.log <<-\\EOF &&\n+\t\t\tstart\n+\t\t\twrote version\n+\t\t\tIN: smudge smudge-write-fail.r 22 [OK] -- OUT: 22 [FAIL]\n+\t\t\tstart\n+\t\t\twrote version\n+\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 [OK]\n+\t\t\tIN: smudge test2.r 8 [OK] -- OUT: 8 [OK]\n+\t\tEOF\n+\t\ttest_cmp expected_checkout_master.log smudge_output.log &&\n+\n+\t\ttest_cmp ../test.o test.r &&\n+\t\t./../rot13.sh <../test.o >expected &&\n+\t\tgit cat-file blob :test.r >actual &&\n+\t\ttest_cmp expected actual\n+\n+\t\ttest_cmp test2.o test2.r &&\n+\t\t./../rot13.sh expected &&\n+\t\tgit cat-file blob :test2.r >actual &&\n+\t\ttest_cmp expected actual\n+\n+\t\ttest_cmp test2.o test2.r &&\n+\t\t./../rot13.sh expected &&\n+\t\tgit cat-file blob :test2.r >actual &&\n+\t\ttest_cmp expected actual\n+\n+\t\t! test_cmp smudge-write-fail.o smudge-write-fail.r && # Smudge failed!\n+\t\t./../rot13.sh expected &&\n+\t\tgit cat-file blob :smudge-write-fail.r >actual &&\n+\t\ttest_cmp expected actual\t\t\t\t\t\t\t # Clean worked!\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t0021/rot13.pl b/t/t0021/rot13.pl\nnew file mode 100755\nindex 0000000..f2d7a03\n--- /dev/null\n+++ b/t/t0021/rot13.pl\n@@ -0,0 +1,80 @@\n+#!/usr/bin/env perl\n+#\n+# Example implementation for the Git filter protocol version 1\n+# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n+#\n+\n+use strict;\n+use warnings;\n+use autodie;\n+\n+sub rot13 {\n+ my ($str) = @_;\n+ $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n+ return $str;\n+}\n+\n+$| = 1; # autoflush STDOUT\n+\n+open my $debug, \">>\", \"output.log\";\n+$debug->autoflush(1);\n+\n+print $debug \"start\\n\";\n+\n+print STDOUT \"git-filter-protocol\\nversion 1\";\n+print $debug \"wrote version\\n\";\n+\n+while (1) {\n+ my $command = ;\n+ unless (defined($command)) {\n+ exit();\n+ }\n+ chomp $command;\n+ print $debug \"IN: $command\";\n+ my $filename = ;\n+ chomp $filename;\n+ print $debug \" $filename\";\n+ my $filelen = ;\n+ chomp $filelen;\n+ print $debug \" $filelen\";\n+\n+ $filelen = int($filelen);\n+ my $output;\n+\n+ if ( $filelen > 0 ) {\n+ my $input;\n+ {\n+ binmode(STDIN);\n+ my $bytes_read = 0;\n+ $bytes_read = read STDIN, $input, $filelen;\n+ if ( $bytes_read != $filelen ) {\n+ die \"not enough to read\";\n+ }\n+ print $debug \" [OK] -- \";\n+ }\n+\n+ if ( $command eq \"clean\") {\n+ $output = rot13($input);\n+ }\n+ elsif ( $command eq \"smudge\" ) {\n+ $output = rot13($input);\n+ }\n+ else {\n+ die \"bad command\\n\";\n+ }\n+ }\n+\n+ my $output_len = length($output);\n+ print STDOUT \"$output_len\\n\";\n+ print $debug \"OUT: $output_len\";\n+ if ( $output_len > 0 ) {\n+ if ( ($command eq \"clean\" and $filename eq \"clean-write-fail.r\") or\n+ ($command eq \"smudge\" and $filename eq \"smudge-write-fail.r\") ) {\n+ print STDOUT \"fail\";\n+ print $debug \" [FAIL]\\n\"\n+ } else {\n+ print STDOUT $output;\n+ print $debug \" [OK]\\n\";\n+ }\n+ }\n+}\n-- \n2.9.0\n\n"},{"id":"292018","messageId":"xmqqshv18i8i.fsf@gitster.mtv.corp.google.com","threadId":"42904","inReplyTo":"20160722154900.19477-1-larsxschneider@gmail.com","subject":"Re: [PATCH v1 0/3] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-22T21:39:57Z","receivedAt":"2016-07-22T21:40:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"larsxschneider@gmail.com writes:\n\n> The first two patches are cleanup patches which are not really necessary\n> for the feature.\n\nThese two looked trivially good.\n\nI think I can agree with what 3/3 wants to do in principle, but\n\n * \"protocol\" is not quite the right word. The current way to\n interact with clean and smudge filters can be considered using a\n different \"protocol\", that conveys the data and the options via\n the command line and pipe. The most distinguishing feature that\n differentiates the old way and the new style this change allows\n is that it allows you to have a single instance of the process\n running that can be reused?\n\n * I am not sure what's the pros-and-cons in forcing people writing\n a single program that can do both cleaning and smudging. You\n cannot have only \"smudge\" side that uses the long-running process\n while \"clean\" side that runs single-shot invocation with this\n design, which I'd imagine would be a downside. If you are going\n to use a long-running process interface for both sides, this\n design allows you to do it with fewer number of processes, which\n may be an upside.\n\n * The way the serialized access to these long-running processes\n work in 3/3 would make it harder or impossible to later\n parallelize conversion? I am imagining a far future where we\n would run \"git checkout .\" using (say) two threads, one\n responsible for active_cache[0..active_nr/2] and the other\n responsible for the remainder.\n\n> You will notice that I do not check the exact number of \"clean\" filter\n> invocations in the tests.\n\nThat is a good thing to do. You shouldn't really care for the\nproper operation of the feature, reducing the number of them would\nbe an independent topic (see the work of Peff earlier today), and we\nmay even find a need to make _more_ calls for correctness (again,\nsee the work of Peff earlier today -- to a person who wants to keep\nthe number of requests to the attribute system low, the change may\nlook like a regression, but it is necessary for the overall system;\nyou may find a similar need to running \"clean\" more for some need of\nthe overall system that you do not anticipate right now).\n\n"},{"id":"292020","messageId":"32d8feda-0fff-6c8c-1ac3-9cc3d783d0ef@web.de","threadId":"42904","inReplyTo":"20160722154900.19477-4-larsxschneider@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-07-22T22:32:46Z","receivedAt":"2016-07-22T22:33:38Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\nOn 07/22/2016 05:49 PM, larsxschneider@gmail.com wrote:\n> From: Lars Schneider \n>\n> Git's clean/smudge mechanism invokes an external filter process for every\n> single blob that is affected by a filter. If Git filters a lot of blobs\n> then the startup time of the external filter processes can become a\n> significant part of the overall Git execution time.\n>\n> This patch adds the filter..useProtocol option which, if enabled,\n> keeps the external filter process running and processes all blobs with\n> the following protocol over stdin/stdout.\n>\n> 1. Git starts the filter on first usage and expects a welcome message\n> with protocol version number:\n> \tGit <-- Filter: \"git-filter-protocol\\n\"\n> \tGit <-- Filter: \"version 1\"\nIs there no terminator here ?\nHow long should the reading side wait without a '\\n', or should it read\n\"version 1\\n\" ?\n\n>\n> 2. Git sends the command (either \"smudge\" or \"clean\"), the filename, the\n> content size in bytes, and the content separated by a newline character:\n> \tGit --> Filter: \"smudge\\n\"\n> \tGit --> Filter: \"testfile.dat\\n\"\n> \tGit --> Filter: \"7\\n\"\n> \tGit --> Filter: \"CONTENT\"\n>\n> 3. The filter is expected to answer with the result content size in\n> bytes and the result content separated by a newline character:\n> \tGit <-- Filter: \"15\\n\"\n> \tGit <-- Filter: \"SMUDGED_CONTENT\"\n>\n> 4. The filter is expected to wait for the next file in step 2, again.\n>\n> Please note that the protocol filters do not support stream processing\n> with this implemenatation because the filter needs to know the length of\n ^^^^^^^^^^^^^^^^typo\n> the result in advance. A protocol version 2 could address this in a\n> future patch.\n>\n> Signed-off-by: Lars Schneider \n> Helped-by: Martin-Louis Bright \n> ---\n> Documentation/gitattributes.txt | 41 +++++++-\n> convert.c | 210 ++++++++++++++++++++++++++++++++++++++--\n> t/t0021-conversion.sh | 170 ++++++++++++++++++++++++++++++++\n> t/t0021/rot13.pl | 80 +++++++++++++++\n> 4 files changed, 494 insertions(+), 7 deletions(-)\n> create mode 100755 t/t0021/rot13.pl\n>\n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index 8882a3e..7026d62 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -300,7 +300,10 @@ checkout, when the `smudge` command is specified, the command is\n> fed the blob object from its standard input, and its standard\n> output is used to update the worktree file. Similarly, the\n> `clean` command is used to convert the contents of worktree file\n> -upon checkin.\n> +upon checkin. By default these commands process only a single\n> +blob and terminate. If the setting filter..useProtocol is\n> +enabled then Git can process all blobs with a single filter command\n> +invocation (see filter protocol below).\n>\n> One use of the content filtering is to massage the content into a shape\n> that is more convenient for the platform, filesystem, and the user to use.\n> @@ -375,6 +378,42 @@ substitution. For example:\n> ------------------------\n>\n>\n> +Filter Protocol\n> +^^^^^^^^^^^^^^^\n> +\n> +If the setting filter..useProtocol is enabled then Git\n> +can process all blobs with a single filter command invocation\n> +by talking with the following protocol over stdin/stdout.\n> +\n> +Git starts the filter on first usage and expects a welcome\n> +message with protocol version number:\n> +------------------------\n> +Git < Filter: \"git-filter-protocol\\n\"\n> +Git < Filter: \"version 1\"\n> +------------------------\n> +\n> +Afterwards Git sends a blob command (either \"smudge\" or \"clean\"),\n> +the filename, the content size in bytes, and the content separated\n> +by a newline character:\n> +------------------------\n> +Git > Filter: \"smudge\\n\"\n> +Git > Filter: \"testfile.dat\\n\"\n> +Git > Filter: \"7\\n\"\n> +Git > Filter: \"CONTENT\"\n> +------------------------\n> +\n> +The filter is expected to answer with the result content size in\n> +bytes and the result content separated by a newline character:\n> +------------------------\n> +Git < Filter: \"15\\n\"\n> +Git < Filter: \"SMUDGED_CONTENT\"\n> +------------------------\n> +\n> +Afterwards the filter is expected to wait for the next blob process\n> +command. A demo implementation can be found in `t/t0021/rot13.pl`\n> +located in the Git core repository.\n> +\n> +\n> Interaction between checkin/checkout attributes\n> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>\n> diff --git a/convert.c b/convert.c\n> index 522e2c5..91ce86f 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -481,12 +481,188 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n> \treturn ret;\n> }\n>\n> +static int cmd_process_map_init = 0;\n> +static struct hashmap cmd_process_map;\n> +\n> +struct cmd2process {\n> +\tstruct hashmap_entry ent; /* must be the first member! */\n> +\tconst char *cmd;\n> +\tlong protocol;\n> +\tstruct child_process process;\n> +};\n> +\n> +static int cmd2process_cmp(const struct cmd2process *e1,\n> +\t\t\t\t\t\t\tconst struct cmd2process *e2,\n> +\t\t\t\t\t\t\tconst void *unused)\n> +{\n> +\treturn strcmp(e1->cmd, e2->cmd);\n> +}\n> +\n> +static struct cmd2process *find_protocol_filter_entry(const char *cmd)\n> +{\n> +\tstruct cmd2process k;\n> +\thashmap_entry_init(&k, strhash(cmd));\n> +\tk.cmd = cmd;\n> +\treturn hashmap_get(&cmd_process_map, &k, NULL);\n> +}\n> +\n> +static void stop_protocol_filter(struct cmd2process *entry) {\n> +\tif (!entry)\n> +\t\treturn;\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tclose(entry->process.in);\n> +\tclose(entry->process.out);\n> +\tsigchain_pop(SIGPIPE);\n> +\tfinish_command(&entry->process);\n> +\tchild_process_clear(&entry->process);\n> +\thashmap_remove(&cmd_process_map, entry, NULL);\n> +\tfree(entry);\n> +}\n> +\n> +static struct cmd2process *start_protocol_filter(const char *cmd)\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry = NULL;\n> +\tstruct child_process *process = NULL;\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n> +\tconst char *argv[] = { NULL, NULL };\n> +\tconst char *header = \"git-filter-protocol\\nversion\";\n> +\n> +\tentry = xmalloc(sizeof(*entry));\n> +\thashmap_entry_init(entry, strhash(cmd));\n> +\tentry->cmd = cmd;\n> +\tprocess = &entry->process;\n> +\n> +\tchild_process_init(process);\n> +\targv[0] = cmd;\n> +\tprocess->argv = argv;\n> +\tprocess->use_shell = 1;\n> +\tprocess->in = -1;\n> +\tprocess->out = -1;\n> +\n> +\tif (start_command(process)) {\n> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n> +\t\treturn NULL;\n> +\t}\n> +\tstrbuf_reset(&nbuf);\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tstrbuf_stripspace(&nbuf, 0);\n> +\tstring_list_split_in_place(&split, nbuf.buf, ' ', 2);\n> +\tret &= split.nr > 1;\n> +\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n> +\tif (ret) {\n> +\t\tentry->protocol = strtol(split.items[1].string, NULL, 10);\n> +\t\tswitch (entry->protocol) {\n> +\t\t\tcase 1:\n> +\t\t\t\tbreak;\n> +\t\t\tdefault:\n> +\t\t\t\tret = 0;\n> +\t\t\t\terror(\"unsupported protocol version %s for external persistent filter '%s'\",\n> +\t\t\t\t\tnbuf.buf, cmd);\n> +\t\t}\n> +\t}\n> +\tstring_list_clear(&split, 0);\n> +\tstrbuf_release(&nbuf);\n> +\n> +\tif (!ret) {\n> +\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\thashmap_add(&cmd_process_map, entry);\n> +\treturn entry;\n> +}\n> +\n> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n> +\t\t\t\t\t\tconst char *filter_type)\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry = NULL;\n> +\tstruct child_process *process = NULL;\n> +\tstruct stat fileStat;\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n> +\tsize_t nbuf_len;\n> +\tchar *strtol_end;\n> +\tchar c;\n> +\n> +\tif (!cmd || !*cmd)\n> +\t\treturn 0;\n> +\n> +\tif (!dst)\n> +\t\treturn 1;\n> +\n> +\tif (!cmd_process_map_init) {\n> +\t\tcmd_process_map_init = 1;\n> +\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n> +\t} else {\n> +\t\tentry = find_protocol_filter_entry(cmd);\n> +\t}\n> +\n> +\tif (!entry){\n> +\t\tentry = start_protocol_filter(cmd);\n> +\t\tif (!entry) {\n> +\t\t\tstop_protocol_filter(entry);\n> +\t\t\treturn 0;\n> +\t\t}\n> +\t}\n> +\tprocess = &entry->process;\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tswitch (entry->protocol) {\n> +\t\tcase 1:\n> +\t\t\tif (fd >= 0 && !src) {\n> +\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n> +\t\t\t\tlen = fileStat.st_size;\n> +\t\t\t}\n> +\t\t\tstrbuf_reset(&nbuf);\n> +\t\t\tstrbuf_addf(&nbuf, \"%s\\n%s\\n%zu\\n\", filter_type, path, len);\n> +\t\t\tret &= write_str_in_full(process->in, nbuf.buf) > 1;\n> +\t\t\tif (len > 0) {\n> +\t\t\t\tif (src)\n> +\t\t\t\t\tret &= write_in_full(process->in, src, len) == len;\n> +\t\t\t\telse if (fd >= 0)\n> +\t\t\t\t\tret &= copy_fd(fd, process->in) == 0;\n> +\t\t\t\telse\n> +\t\t\t\t\tret &= 0;\n> +\t\t\t}\n> +\n> +\t\t\tstrbuf_reset(&nbuf);\n> +\t\t\twhile (xread(process->out, &c, 1) == 1 && c != '\\n')\n> +\t\t\t\tstrbuf_addchars(&nbuf, c, 1);\n> +\t\t\tnbuf_len = (size_t)strtol(nbuf.buf, &strtol_end, 10);\n> +\t\t\tret &= (strtol_end != nbuf.buf && errno != ERANGE);\n> +\t\t\tstrbuf_reset(&nbuf);\n> +\t\t\tif (nbuf_len > 0)\n> +\t\t\t\tret &= strbuf_read_once(&nbuf, process->out, nbuf_len) == nbuf_len;\n> +\t\t\tbreak;\n> +\t\tdefault:\n> +\t\t\tret = 0;\n> +\t}\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tif (ret) {\n> +\t\tstrbuf_swap(dst, &nbuf);\n> +\t} else {\n> +\t\t// Something went wrong with the protocol filter. Force shutdown!\n> +\t\tstop_protocol_filter(entry);\n> +\t}\n> +\tstrbuf_release(&nbuf);\n> +\treturn ret;\n> +}\n> +\n> static struct convert_driver {\n> \tconst char *name;\n> \tstruct convert_driver *next;\n> \tconst char *smudge;\n> \tconst char *clean;\n> \tint required;\n> +\tint use_protocol;\n> } *user_convert, **user_convert_tail;\n>\n> static int read_convert_config(const char *var, const char *value, void *cb)\n> @@ -526,6 +702,11 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n> \tif (!strcmp(\"clean\", key))\n> \t\treturn git_config_string(&drv->clean, var, value);\n>\n> +\tif (!strcmp(\"useprotocol\", key)) {\n> +\t\tdrv->use_protocol = git_config_bool(var, value);\n> +\t\treturn 0;\n> +\t}\n> +\n> \tif (!strcmp(\"required\", key)) {\n> \t\tdrv->required = git_config_bool(var, value);\n> \t\treturn 0;\n> @@ -823,7 +1004,10 @@ int would_convert_to_git_filter_fd(const char *path)\n> \tif (!ca.drv->required)\n> \t\treturn 0;\n>\n> -\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n> +\tif (ca.drv->use_protocol)\n> +\t\treturn apply_protocol_filter(path, NULL, 0, -1, NULL, ca.drv->clean, \"clean\");\n> +\telse\n> +\t\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n> }\n>\n> const char *get_convert_attr_ascii(const char *path)\n> @@ -857,16 +1041,20 @@ int convert_to_git(const char *path, const char *src, size_t len,\n> {\n> \tint ret = 0;\n> \tconst char *filter = NULL;\n> -\tint required = 0;\n> +\tint required = 0, use_protocol = 0;\n> \tstruct conv_attrs ca;\n>\n> \tconvert_attrs(&ca, path);\n> \tif (ca.drv) {\n> \t\tfilter = ca.drv->clean;\n> \t\trequired = ca.drv->required;\n> +\t\tuse_protocol = ca.drv->use_protocol;\n> \t}\n>\n> -\tret |= apply_filter(path, src, len, -1, dst, filter);\n> +\tif (use_protocol)\n> +\t\tret |= apply_protocol_filter(path, src, len, -1, dst, filter, \"clean\");\n> +\telse\n> +\t\tret |= apply_filter(path, src, len, -1, dst, filter);\n> \tif (!ret && required)\n> \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n>\n> @@ -885,13 +1073,19 @@ int convert_to_git(const char *path, const char *src, size_t len,\n> void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n> \t\t\t enum safe_crlf checksafe)\n> {\n> +\tint ret = 0;\n> \tstruct conv_attrs ca;\n> \tconvert_attrs(&ca, path);\n>\n> \tassert(ca.drv);\n> \tassert(ca.drv->clean);\n>\n> -\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))\n> +\tif (ca.drv->use_protocol)\n> +\t\tret = apply_protocol_filter(path, NULL, 0, fd, dst, ca.drv->clean, \"clean\");\n> +\telse\n> +\t\tret = apply_filter(path, NULL, 0, fd, dst, ca.drv->clean);\n> +\n> +\tif (!ret)\n> \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n>\n> \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n> @@ -904,13 +1098,14 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n> {\n> \tint ret = 0, ret_filter = 0;\n> \tconst char *filter = NULL;\n> -\tint required = 0;\n> +\tint required = 0, use_protocol = 0;\n> \tstruct conv_attrs ca;\n>\n> \tconvert_attrs(&ca, path);\n> \tif (ca.drv) {\n> \t\tfilter = ca.drv->smudge;\n> \t\trequired = ca.drv->required;\n> +\t\tuse_protocol = ca.drv->use_protocol;\n> \t}\n>\n> \tret |= ident_to_worktree(path, src, len, dst, ca.ident);\n> @@ -930,7 +1125,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n> \t\t}\n> \t}\n>\n> -\tret_filter = apply_filter(path, src, len, -1, dst, filter);\n> +\tif (use_protocol)\n> +\t\tret_filter = apply_protocol_filter(path, src, len, -1, dst, filter, \"smudge\");\n> +\telse\n> +\t\tret_filter |= apply_filter(path, src, len, -1, dst, filter);\n> \tif (!ret_filter && required)\n> \t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n>\n> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n> index a05a8d2..d9077ea 100755\n> --- a/t/t0021-conversion.sh\n> +++ b/t/t0021-conversion.sh\n> @@ -268,4 +268,174 @@ test_expect_success 'disable filter with empty override' '\n> \ttest_must_be_empty err\n> '\n>\n> +test_expect_success 'required protocol filter should filter data' '\n> +\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> +\ttest_config_global filter.protocol.useprotocol true &&\n> +\ttest_config_global filter.protocol.required true &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\t\tgit add . &&\n> +\t\tgit commit . -m \"test commit\" &&\n> +\t\tgit branch empty &&\n> +\n> +\t\tcat ../test.o >test.r &&\n> +\t\techo \"test22\" >test2.r &&\n> +\t\techo \"test333\" >test3.r &&\n> +\n> +\t\trm -f output.log &&\n> +\t\tgit add . &&\n> +\t\tsort output.log | uniq -c | sed \"s/^[ ]*//\" >uniq_output.log &&\n> +\t\tcat >expected_add.log <<-\\EOF &&\n> +\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 [OK]\n> +\t\t\t1 IN: clean test2.r 7 [OK] -- OUT: 7 [OK]\n> +\t\t\t1 IN: clean test3.r 8 [OK] -- OUT: 8 [OK]\n> +\t\t\t1 start\n> +\t\t\t1 wrote version\n> +\t\tEOF\n> +\t\ttest_cmp expected_add.log uniq_output.log &&\n> +\n> +\t\tprintf \"\" >output.log &&\n> +\t\tgit commit . -m \"test commit\" &&\n> +\t\tsort output.log | uniq -c | sed \"s/^[ ]*//\" |\n> +\t\t\tsed \"s/^\\([0-9]\\) IN: clean/x IN: clean/\" >uniq_output.log &&\n> +\t\tcat >expected_commit.log <<-\\EOF &&\n> +\t\t\tx IN: clean test.r 57 [OK] -- OUT: 57 [OK]\n> +\t\t\tx IN: clean test2.r 7 [OK] -- OUT: 7 [OK]\n> +\t\t\tx IN: clean test3.r 8 [OK] -- OUT: 8 [OK]\n> +\t\t\t1 start\n> +\t\t\t1 wrote version\n> +\t\tEOF\n> +\t\ttest_cmp expected_commit.log uniq_output.log &&\n> +\n> +\t\tprintf \"\" >output.log &&\n> +\t\trm -f test?.r &&\n> +\t\tgit checkout . &&\n> +\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n> +\t\tcat >expected_checkout.log <<-\\EOF &&\n> +\t\t\tstart\n> +\t\t\twrote version\n> +\t\t\tIN: smudge test2.r 7 [OK] -- OUT: 7 [OK]\n> +\t\t\tIN: smudge test3.r 8 [OK] -- OUT: 8 [OK]\n> +\t\tEOF\n> +\t\ttest_cmp expected_checkout.log smudge_output.log &&\n> +\n> +\t\tgit checkout empty &&\n> +\n> +\t\tprintf \"\" >output.log &&\n> +\t\tgit checkout master &&\n> +\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n> +\t\tcat >expected_checkout_master.log <<-\\EOF &&\n> +\t\t\tstart\n> +\t\t\twrote version\n> +\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 [OK]\n> +\t\t\tIN: smudge test2.r 7 [OK] -- OUT: 7 [OK]\n> +\t\t\tIN: smudge test3.r 8 [OK] -- OUT: 8 [OK]\n> +\t\tEOF\n> +\t\ttest_cmp expected_checkout_master.log smudge_output.log\n> +\t)\n> +'\n> +\n> +test_expect_success EXPENSIVE 'protocol filter large file' '\n> +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> +\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"2GB filter=largefile\" >.gitattributes &&\n> +\t\tfor i in $(test_seq 1 2048); do printf \"%1048576d\" 1; done >2GB &&\nSide question:\nIs there a way to \"re-use\" the 2GB test file through t0021?\nIt takes a long time to produce it, especially on my 32 Bit systems ;-)\nBut this may be a different patch.\n\n> +\t\tgit add 2GB 2>err &&\n> +\t\ttest_must_be_empty err &&\n> +\t\trm -f 2GB &&\n> +\t\tgit checkout -- 2GB 2>err &&\n> +\t\ttest_must_be_empty err\n> +\t)\n> +'\n> +\n> +test_expect_success 'required protocol filter should fail with clean' '\n> +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> +\ttest_config_global filter.protocol.useprotocol true &&\n> +\ttest_config_global filter.protocol.required true &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\n> +\t\tcat ../test.o >test.r &&\n> +\t\techo \"this is going to fail\" >clean-write-fail.r &&\n> +\t\techo \"test333\" >test3.r &&\n> +\n> +\t\t# Note: There are three clean paths in convert.c we just test one here.\n> +\t\ttest_must_fail git add .\n> +\t)\n> +'\n> +\n> +test_expect_success 'protocol filter should restart after failure' '\n> +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> +\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> +\ttest_config_global filter.protocol.useprotocol true &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\n> +\t\tcat ../test.o >test.r &&\n> +\t\techo \"1234567\" >test2.o &&\n> +\t\tcat test2.o >test2.r &&\n> +\t\techo \"this is going to fail\" >smudge-write-fail.o &&\n> +\t\tcat smudge-write-fail.o >smudge-write-fail.r &&\n> +\t\tgit add . &&\n> +\t\tgit commit . -m \"test commit\" &&\n> +\t\trm -f *.r &&\n> +\n> +\t\tprintf \"\" >output.log &&\nIs this the same as\n >output.log\nto produce an empty file ?\n\n> +\t\tgit checkout . &&\n> +\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n> +\t\tcat >expected_checkout_master.log <<-\\EOF &&\n> +\t\t\tstart\n> +\t\t\twrote version\n> +\t\t\tIN: smudge smudge-write-fail.r 22 [OK] -- OUT: 22 [FAIL]\n> +\t\t\tstart\n> +\t\t\twrote version\n> +\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 [OK]\n> +\t\t\tIN: smudge test2.r 8 [OK] -- OUT: 8 [OK]\n> +\t\tEOF\n> +\t\ttest_cmp expected_checkout_master.log smudge_output.log &&\n> +\n> +\t\ttest_cmp ../test.o test.r &&\n> +\t\t./../rot13.sh <../test.o >expected &&\n> +\t\tgit cat-file blob :test.r >actual &&\n> +\t\ttest_cmp expected actual\n> +\n> +\t\ttest_cmp test2.o test2.r &&\n> +\t\t./../rot13.sh expected &&\n> +\t\tgit cat-file blob :test2.r >actual &&\n> +\t\ttest_cmp expected actual\n> +\n> +\t\ttest_cmp test2.o test2.r &&\n> +\t\t./../rot13.sh expected &&\n> +\t\tgit cat-file blob :test2.r >actual &&\n> +\t\ttest_cmp expected actual\n> +\n> +\t\t! test_cmp smudge-write-fail.o smudge-write-fail.r && # Smudge failed!\n> +\t\t./../rot13.sh expected &&\n> +\t\tgit cat-file blob :smudge-write-fail.r >actual &&\n> +\t\ttest_cmp expected actual\t\t\t\t\t\t\t # Clean worked!\n> +\t)\n> +'\n> +\n> test_done\n> diff --git a/t/t0021/rot13.pl b/t/t0021/rot13.pl\n> new file mode 100755\n> index 0000000..f2d7a03\n> --- /dev/null\n> +++ b/t/t0021/rot13.pl\n> @@ -0,0 +1,80 @@\n> +#!/usr/bin/env perl\nShould this be\n\"$PERL_PATH\" ?\nAnd do we need to protect the TC with\ntest_have_prereq PERL or similar ?\n\n\nAn other solution could be to write a filter in C-language.\n> +#\n> +# Example implementation for the Git filter protocol version 1\n> +# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n> +#\n> +\n> +use strict;\n> +use warnings;\n> +use autodie;\n> +\n> +sub rot13 {\n> + my ($str) = @_;\n> + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n> + return $str;\n> +}\n> +\n> +$| = 1; # autoflush STDOUT\n> +\n> +open my $debug, \">>\", \"output.log\";\n> +$debug->autoflush(1);\n> +\n> +print $debug \"start\\n\";\n> +\n> +print STDOUT \"git-filter-protocol\\nversion 1\";\nAgain, I don't like the missing terminator here.\nWhat if we step up to protocol \"version 10\" ?\nCould it work to use one and only one line,\nwith one terminator, like this ?\nprint STDOUT \"git-filter-protocol version 1\\1\";\n> +print $debug \"wrote version\\n\";\n> +\n> +while (1) {\n> + my $command = ;\n> + unless (defined($command)) {\n> + exit();\n> + }\n> + chomp $command;\n> + print $debug \"IN: $command\";\n> + my $filename = ;\n> + chomp $filename;\n> + print $debug \" $filename\";\n> + my $filelen = ;\n> + chomp $filelen;\n> + print $debug \" $filelen\";\n> +\n> + $filelen = int($filelen);\n> + my $output;\n> +\n> + if ( $filelen > 0 ) {\n> + my $input;\n> + {\n> + binmode(STDIN);\n> + my $bytes_read = 0;\n> + $bytes_read = read STDIN, $input, $filelen;\n> + if ( $bytes_read != $filelen ) {\n> + die \"not enough to read\";\n> + }\n> + print $debug \" [OK] -- \";\n> + }\n> +\n> + if ( $command eq \"clean\") {\n> + $output = rot13($input);\n> + }\n> + elsif ( $command eq \"smudge\" ) {\n> + $output = rot13($input);\n> + }\n> + else {\n> + die \"bad command\\n\";\n> + }\n> + }\n> +\n> + my $output_len = length($output);\n> + print STDOUT \"$output_len\\n\";\n> + print $debug \"OUT: $output_len\";\n> + if ( $output_len > 0 ) {\n> + if ( ($command eq \"clean\" and $filename eq \"clean-write-fail.r\") or\n> + ($command eq \"smudge\" and $filename eq \"smudge-write-fail.r\") ) {\n> + print STDOUT \"fail\";\n> + print $debug \" [FAIL]\\n\"\n> + } else {\n> + print STDOUT $output;\n> + print $debug \" [OK]\\n\";\n> + }\n> + }\n> +}\n>\n"},{"id":"292023","messageId":"9f47cf44-7163-a7a7-c1f0-87ebdee65b37@ramsayjones.plus.com","threadId":"42904","inReplyTo":"20160722154900.19477-4-larsxschneider@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2016-07-22T23:19:44Z","receivedAt":"2016-07-22T23:19:56Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 22/07/16 16:49, larsxschneider@gmail.com wrote:\n> From: Lars Schneider \n> \n> Git's clean/smudge mechanism invokes an external filter process for every\n> single blob that is affected by a filter. If Git filters a lot of blobs\n> then the startup time of the external filter processes can become a\n> significant part of the overall Git execution time.\n> \n> This patch adds the filter..useProtocol option which, if enabled,\n> keeps the external filter process running and processes all blobs with\n> the following protocol over stdin/stdout.\n> \n> 1. Git starts the filter on first usage and expects a welcome message\n> with protocol version number:\n> \tGit <-- Filter: \"git-filter-protocol\\n\"\n> \tGit <-- Filter: \"version 1\"\n\nHmm, I was a bit surprised to see a 'filter' talk first (but so long as the\ninteraction is fully defined, I guess it doesn't matter).\n\n[If you wanted to check for a version, you could add a \"version\" command\ninstead, just like \"clean\" and \"smudge\".]\n\n[snip]\n\n> diff --git a/convert.c b/convert.c\n> index 522e2c5..91ce86f 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -481,12 +481,188 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n> \treturn ret;\n> }\n> \n> +static int cmd_process_map_init = 0;\n> +static struct hashmap cmd_process_map;\n> +\n> +struct cmd2process {\n> +\tstruct hashmap_entry ent; /* must be the first member! */\n> +\tconst char *cmd;\n> +\tlong protocol;\n> +\tstruct child_process process;\n> +};\n> +\n> +static int cmd2process_cmp(const struct cmd2process *e1,\n> +\t\t\t\t\t\t\tconst struct cmd2process *e2,\n> +\t\t\t\t\t\t\tconst void *unused)\n> +{\n> +\treturn strcmp(e1->cmd, e2->cmd);\n> +}\n> +\n> +static struct cmd2process *find_protocol_filter_entry(const char *cmd)\n> +{\n> +\tstruct cmd2process k;\n> +\thashmap_entry_init(&k, strhash(cmd));\n> +\tk.cmd = cmd;\n> +\treturn hashmap_get(&cmd_process_map, &k, NULL);\n> +}\n> +\n> +static void stop_protocol_filter(struct cmd2process *entry) {\n> +\tif (!entry)\n> +\t\treturn;\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tclose(entry->process.in);\n> +\tclose(entry->process.out);\n> +\tsigchain_pop(SIGPIPE);\n> +\tfinish_command(&entry->process);\n> +\tchild_process_clear(&entry->process);\n> +\thashmap_remove(&cmd_process_map, entry, NULL);\n> +\tfree(entry);\n> +}\n> +\n> +static struct cmd2process *start_protocol_filter(const char *cmd)\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry = NULL;\n> +\tstruct child_process *process = NULL;\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n> +\tconst char *argv[] = { NULL, NULL };\n> +\tconst char *header = \"git-filter-protocol\\nversion\";\n> +\n> +\tentry = xmalloc(sizeof(*entry));\n> +\thashmap_entry_init(entry, strhash(cmd));\n> +\tentry->cmd = cmd;\n> +\tprocess = &entry->process;\n> +\n> +\tchild_process_init(process);\n> +\targv[0] = cmd;\n> +\tprocess->argv = argv;\n> +\tprocess->use_shell = 1;\n> +\tprocess->in = -1;\n> +\tprocess->out = -1;\n> +\n> +\tif (start_command(process)) {\n> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n> +\t\treturn NULL;\n> +\t}\n> +\tstrbuf_reset(&nbuf);\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n\nHmm, how much will be read into nbuf by this single call?\nSince strbuf_read_once() makes a single call to xread(), with\na len argument that will probably be 8192, you can not really\ntell how much it will read, in general. (xread() does not\nguarantee how many bytes it will read.)\n\nIn particular, it could be less than strlen(header).\n\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tstrbuf_stripspace(&nbuf, 0);\n> +\tstring_list_split_in_place(&split, nbuf.buf, ' ', 2);\n> +\tret &= split.nr > 1;\n> +\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n> +\tif (ret) {\n> +\t\tentry->protocol = strtol(split.items[1].string, NULL, 10);\n> +\t\tswitch (entry->protocol) {\n> +\t\t\tcase 1:\n> +\t\t\t\tbreak;\n> +\t\t\tdefault:\n> +\t\t\t\tret = 0;\n> +\t\t\t\terror(\"unsupported protocol version %s for external persistent filter '%s'\",\n> +\t\t\t\t\tnbuf.buf, cmd);\n> +\t\t}\n> +\t}\n> +\tstring_list_clear(&split, 0);\n> +\tstrbuf_release(&nbuf);\n> +\n> +\tif (!ret) {\n> +\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\thashmap_add(&cmd_process_map, entry);\n> +\treturn entry;\n> +}\n> +\n> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n> +\t\t\t\t\t\tconst char *filter_type)\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry = NULL;\n> +\tstruct child_process *process = NULL;\n> +\tstruct stat fileStat;\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n> +\tsize_t nbuf_len;\n> +\tchar *strtol_end;\n> +\tchar c;\n> +\n> +\tif (!cmd || !*cmd)\n> +\t\treturn 0;\n> +\n> +\tif (!dst)\n> +\t\treturn 1;\n> +\n> +\tif (!cmd_process_map_init) {\n> +\t\tcmd_process_map_init = 1;\n> +\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n> +\t} else {\n> +\t\tentry = find_protocol_filter_entry(cmd);\n> +\t}\n> +\n> +\tif (!entry){\n> +\t\tentry = start_protocol_filter(cmd);\n> +\t\tif (!entry) {\n> +\t\t\tstop_protocol_filter(entry);\n> +\t\t\treturn 0;\n> +\t\t}\n> +\t}\n> +\tprocess = &entry->process;\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tswitch (entry->protocol) {\n> +\t\tcase 1:\n> +\t\t\tif (fd >= 0 && !src) {\n> +\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n> +\t\t\t\tlen = fileStat.st_size;\n> +\t\t\t}\n> +\t\t\tstrbuf_reset(&nbuf);\n> +\t\t\tstrbuf_addf(&nbuf, \"%s\\n%s\\n%zu\\n\", filter_type, path, len);\n> +\t\t\tret &= write_str_in_full(process->in, nbuf.buf) > 1;\n\nwhy not write_in_full(process->in, nbuf.buf, nbuf.len) ?\n\n> +\t\t\tif (len > 0) {\n> +\t\t\t\tif (src)\n> +\t\t\t\t\tret &= write_in_full(process->in, src, len) == len;\n> +\t\t\t\telse if (fd >= 0)\n> +\t\t\t\t\tret &= copy_fd(fd, process->in) == 0;\n> +\t\t\t\telse\n> +\t\t\t\t\tret &= 0;\n> +\t\t\t}\n> +\n> +\t\t\tstrbuf_reset(&nbuf);\n> +\t\t\twhile (xread(process->out, &c, 1) == 1 && c != '\\n')\n> +\t\t\t\tstrbuf_addchars(&nbuf, c, 1);\n> +\t\t\tnbuf_len = (size_t)strtol(nbuf.buf, &strtol_end, 10);\n> +\t\t\tret &= (strtol_end != nbuf.buf && errno != ERANGE);\n> +\t\t\tstrbuf_reset(&nbuf);\n> +\t\t\tif (nbuf_len > 0)\n> +\t\t\t\tret &= strbuf_read_once(&nbuf, process->out, nbuf_len) == nbuf_len;\n\nAgain, how many bytes will be read?\nNote, that in the default configuration, a _maximum_ of\nMAX_IO_SIZE (8MB or SSIZE_MAX, whichever is smaller) bytes\nwill be read.\n\nATB,\nRamsay Jones\n\n"},{"id":"292025","messageId":"d32793ea-2b78-a16e-2d77-42e2b9d36ce8@ramsayjones.plus.com","threadId":"42904","inReplyTo":"9f47cf44-7163-a7a7-c1f0-87ebdee65b37@ramsayjones.plus.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2016-07-22T23:28:45Z","receivedAt":"2016-07-22T23:28:54Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\nHi Lars,\n\nOn 23/07/16 00:19, Ramsay Jones wrote:\n> \n> \n> On 22/07/16 16:49, larsxschneider@gmail.com wrote:\n>> From: Lars Schneider \n>>\n>> Git's clean/smudge mechanism invokes an external filter process for every\n>> single blob that is affected by a filter. If Git filters a lot of blobs\n>> then the startup time of the external filter processes can become a\n>> significant part of the overall Git execution time.\n>>\n>> This patch adds the filter..useProtocol option which, if enabled,\n>> keeps the external filter process running and processes all blobs with\n>> the following protocol over stdin/stdout.\n>>\n>> 1. Git starts the filter on first usage and expects a welcome message\n>> with protocol version number:\n>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>> \tGit <-- Filter: \"version 1\"\n> \n> Hmm, I was a bit surprised to see a 'filter' talk first (but so long as the\n> interaction is fully defined, I guess it doesn't matter).\n> \n> [If you wanted to check for a version, you could add a \"version\" command\n> instead, just like \"clean\" and \"smudge\".]\n> \n> [snip]\n> \n>> diff --git a/convert.c b/convert.c\n>> index 522e2c5..91ce86f 100644\n>> --- a/convert.c\n>> +++ b/convert.c\n>> @@ -481,12 +481,188 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n>> \treturn ret;\n>> }\n>> \n>> +static int cmd_process_map_init = 0;\n>> +static struct hashmap cmd_process_map;\n>> +\n>> +struct cmd2process {\n>> +\tstruct hashmap_entry ent; /* must be the first member! */\n>> +\tconst char *cmd;\n>> +\tlong protocol;\n>> +\tstruct child_process process;\n>> +};\n>> +\n>> +static int cmd2process_cmp(const struct cmd2process *e1,\n>> +\t\t\t\t\t\t\tconst struct cmd2process *e2,\n>> +\t\t\t\t\t\t\tconst void *unused)\n>> +{\n>> +\treturn strcmp(e1->cmd, e2->cmd);\n>> +}\n>> +\n>> +static struct cmd2process *find_protocol_filter_entry(const char *cmd)\n>> +{\n>> +\tstruct cmd2process k;\n>> +\thashmap_entry_init(&k, strhash(cmd));\n>> +\tk.cmd = cmd;\n>> +\treturn hashmap_get(&cmd_process_map, &k, NULL);\n>> +}\n>> +\n>> +static void stop_protocol_filter(struct cmd2process *entry) {\n>> +\tif (!entry)\n>> +\t\treturn;\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tclose(entry->process.in);\n>> +\tclose(entry->process.out);\n>> +\tsigchain_pop(SIGPIPE);\n>> +\tfinish_command(&entry->process);\n>> +\tchild_process_clear(&entry->process);\n>> +\thashmap_remove(&cmd_process_map, entry, NULL);\n>> +\tfree(entry);\n>> +}\n>> +\n>> +static struct cmd2process *start_protocol_filter(const char *cmd)\n>> +{\n>> +\tint ret = 1;\n>> +\tstruct cmd2process *entry = NULL;\n>> +\tstruct child_process *process = NULL;\n>> +\tstruct strbuf nbuf = STRBUF_INIT;\n>> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n>> +\tconst char *argv[] = { NULL, NULL };\n>> +\tconst char *header = \"git-filter-protocol\\nversion\";\n>> +\n>> +\tentry = xmalloc(sizeof(*entry));\n>> +\thashmap_entry_init(entry, strhash(cmd));\n>> +\tentry->cmd = cmd;\n>> +\tprocess = &entry->process;\n>> +\n>> +\tchild_process_init(process);\n>> +\targv[0] = cmd;\n>> +\tprocess->argv = argv;\n>> +\tprocess->use_shell = 1;\n>> +\tprocess->in = -1;\n>> +\tprocess->out = -1;\n>> +\n>> +\tif (start_command(process)) {\n>> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n>> +\t\treturn NULL;\n>> +\t}\n>> +\tstrbuf_reset(&nbuf);\n>> +\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n> \n> Hmm, how much will be read into nbuf by this single call?\n> Since strbuf_read_once() makes a single call to xread(), with\n> a len argument that will probably be 8192, you can not really\n> tell how much it will read, in general. (xread() does not\n> guarantee how many bytes it will read.)\n> \n> In particular, it could be less than strlen(header).\n\nPlease ignore this email, it's late ... ;-)\n\nSorry for the noise.\n\n[Off to bed now]\n\nATB,\nRamsay Jones\n\n"},{"id":"292026","messageId":"5792B622.5040306@gmail.com","threadId":"42904","inReplyTo":"20160722154900.19477-4-larsxschneider@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-23T00:11:14Z","receivedAt":"2016-07-23T00:11:31Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-22 o 17:49, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n\nNb. this line is only needed if you want author name and/or date\ndifferent from the email sender, or if you have sender line misconfigured\n(e.g. lacking the human readable name).\n\n> \n> Git's clean/smudge mechanism invokes an external filter process for every\n> single blob that is affected by a filter. If Git filters a lot of blobs\n> then the startup time of the external filter processes can become a\n> significant part of the overall Git execution time.\n\nDo I understand it correctly (from the commit description) that with\nthis new option you start one filter process for the whole life of\nthe Git command (e.g. `git add .`), which may perform more than one\ncleanup, but do not create a daemon or daemon-like process which\nwould live for several commands invocations?\n\n> \n> This patch adds the filter..useProtocol option which, if enabled,\n> keeps the external filter process running and processes all blobs with\n> the following protocol over stdin/stdout.\n\nI agree with Junio that the name \"useProtocol\" is bad, and not quite\nright. Perhaps \"persistent\" would be better? Also, what is the value\nof `filter..useProtocol`: boolean? or a script name?\n\nI also agree that we might wat to be able to keep clean and smudge\nfilters separate, but be able to run a single program if they are\nboth the same. I think there is a special case for filter unset,\nand/or filter being \"cat\" -- we would want to keep that.\n\nMy proposal is to use `filter..persistent` as an addition\nto 'clean' and 'smudge' variables, with the following possible\nvalues:\n\n * none (the default)\n * clean\n * smudge\n * both\n\nI assume that either Git would have to start multiple filter\ncommands for multi-threaded operation, or the protocol would have\nto be extended to make persistent filter fork itself.\n\n\nBTW. what would happen in your original proposal if the user had\n*both* filter..useProtocol and filter..smudge\n(and/or filter..clean) set?\n\n> \n> 1. Git starts the filter on first usage and expects a welcome message\n> with protocol version number:\n> \tGit <-- Filter: \"git-filter-protocol\\n\"\n> \tGit <-- Filter: \"version 1\"\n\nI was wondering how Git would know that filter executable was started,\nbut then I realized it was once-per-command invocation, not a daemon.\n\nI agree with Torsten that there should be a terminator after the\nversion number.\n\nAlso, for future extendability this should be probably followed by\npossibly empty list of script capabilities, that is:\n\n \tGit <-- Filter: \"git-filter-protocol\\n\"\n \tGit <-- Filter: \"version 1.1\\n\"\n \tGit <-- Filter: \"capabilities clean smudge\\n\"\n\nOr we can add capabilities in later version...\n\nBTW. why not follow e.g. HTTP protocol example, and use\n\n \tGit <-- Filter: \"git-filter-protocol/1\\n\"\n\n> 2. Git sends the command (either \"smudge\" or \"clean\"), the filename, the\n> content size in bytes, and the content separated by a newline character:\n> \tGit --> Filter: \"smudge\\n\"\n\nWould it help (for some cases) to pass the name of filter that\nis being invoked?\n\n> \tGit --> Filter: \"testfile.dat\\n\"\n\nUnfortunately, while sane filenames should not contain newlines[1],\nthe unfortunate fact is that *filenames can include newlines*, and\nyou need to be able to handle that[2]. Therefore you need either to\nchoose a different separator (the only one that can be safely used\nis \"\\0\", i.e. the NUL character - but it is not something easy to\nhandle by shell scripts), or C-quote filenames as needed, or always\nC-quote filenames. C-quoting at minimum should include quoting newline\ncharacter, and the escape character itself.\n\nBTW. is it the basename of a file, or a full pathname?\n\n[1]: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n[2]: http://www.dwheeler.com/essays/filenames-in-shell.html\n\n> \tGit --> Filter: \"7\\n\"\n\nThat's the content size in bytes written as an ASCII number.\n\n> \tGit --> Filter: \"CONTENT\"\n\nCan filter ignore the content size, and just read all what it was\nsent, that is until eof or something?\n\n> \n> 3. The filter is expected to answer with the result content size in\n> bytes and the result content separated by a newline character:\n> \tGit <-- Filter: \"15\\n\"\n> \tGit <-- Filter: \"SMUDGED_CONTENT\"\n\nI wonder how hard would be to write filters for this protocol...\n\n> \n> 4. The filter is expected to wait for the next file in step 2, again.\n> \n> Please note that the protocol filters do not support stream processing\n> with this implemenatation because the filter needs to know the length of\n ^^^^^^^^~^^^^^^\n\nimplementation\n\n> the result in advance. A protocol version 2 could address this in a\n> future patch.\n> \n> Signed-off-by: Lars Schneider \n> Helped-by: Martin-Louis Bright \n> ---\n> Documentation/gitattributes.txt | 41 +++++++-\n> convert.c | 210 ++++++++++++++++++++++++++++++++++++++--\n> t/t0021-conversion.sh | 170 ++++++++++++++++++++++++++++++++\n\nWouldn't it be better to name the test case something more\ndescriptive, for example\n\n t/t0021-filter-driver-useProtocol.sh\n\nThe name of test should be adjusted to final name of the feature,\nof course.\n\n> t/t0021/rot13.pl | 80 +++++++++++++++\n> 4 files changed, 494 insertions(+), 7 deletions(-)\n> create mode 100755 t/t0021/rot13.pl\n> \n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index 8882a3e..7026d62 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -300,7 +300,10 @@ checkout, when the `smudge` command is specified, the command is\n> fed the blob object from its standard input, and its standard\n> output is used to update the worktree file. Similarly, the\n> `clean` command is used to convert the contents of worktree file\n> -upon checkin.\n> +upon checkin. By default these commands process only a single\n> +blob and terminate. If the setting filter..useProtocol is\n> +enabled then Git can process all blobs with a single filter command\n> +invocation (see filter protocol below).\n\nThis does not tell the precedence between `smudge`, `clean` and\nfilter..useProtocol, see above. Also, discrepancy in how\nconfig variables are referenced.\n\n> \n> One use of the content filtering is to massage the content into a shape\n> that is more convenient for the platform, filesystem, and the user to use.\n> @@ -375,6 +378,42 @@ substitution. For example:\n> ------------------------\n> \n> \n> +Filter Protocol\n> +^^^^^^^^^^^^^^^\n> +\n> +If the setting filter..useProtocol is enabled then Git\n\nThis seems to tell that `useProtocol` is boolean-valued (?)\n\n> +can process all blobs with a single filter command invocation\n> +by talking with the following protocol over stdin/stdout.\n\nShould we use stdin/stdout shortcut, or spell standard input\nand standard output in full?\n\n> diff --git a/convert.c b/convert.c\n> index 522e2c5..91ce86f 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -481,12 +481,188 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n> \treturn ret;\n> }\n> \n> +static int cmd_process_map_init = 0;\n> +static struct hashmap cmd_process_map;\n> +\n> +struct cmd2process {\n> +\tstruct hashmap_entry ent; /* must be the first member! */\n> +\tconst char *cmd;\n> +\tlong protocol;\n> +\tstruct child_process process;\n> +};\n[...]\n> +static struct cmd2process *find_protocol_filter_entry(const char *cmd)\n> +{\n> +\tstruct cmd2process k;\n> +\thashmap_entry_init(&k, strhash(cmd));\n> +\tk.cmd = cmd;\n> +\treturn hashmap_get(&cmd_process_map, &k, NULL);\n\nShould we use global variable cmd_process_map, or pass it as parameter?\nThe same question apply for other procedures and functions.\n\nNote that I am not saying that it is a bad thing to use global\nvariable here.\n\n[...]\n> +static struct cmd2process *start_protocol_filter(const char *cmd)\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry = NULL;\n> +\tstruct child_process *process = NULL;\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n> +\tconst char *argv[] = { NULL, NULL };\n> +\tconst char *header = \"git-filter-protocol\\nversion\";\n> +\n> +\tentry = xmalloc(sizeof(*entry));\n> +\thashmap_entry_init(entry, strhash(cmd));\n> +\tentry->cmd = cmd;\n> +\tprocess = &entry->process;\n> +\n> +\tchild_process_init(process);\n> +\targv[0] = cmd;\n> +\tprocess->argv = argv;\n> +\tprocess->use_shell = 1;\n> +\tprocess->in = -1;\n> +\tprocess->out = -1;\n> +\n> +\tif (start_command(process)) {\n> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n> +\t\treturn NULL;\n> +\t}\n> +\tstrbuf_reset(&nbuf);\n\nIs strbuf_reset needed here? We have not used nbuf variable yet.\n\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tstrbuf_stripspace(&nbuf, 0);\n> +\tstring_list_split_in_place(&split, nbuf.buf, ' ', 2);\n> +\tret &= split.nr > 1;\n> +\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n> +\tif (ret) {\n> +\t\tentry->protocol = strtol(split.items[1].string, NULL, 10);\n\nThis does not handle at least some errors in version number parsing,\nfor example junk after version number. Don't we have some helper\nfunctions for this?\n\nNb. this code makes it so that the version number must be integer.\n\n> +\t\tswitch (entry->protocol) {\n> +\t\t\tcase 1:\n> +\t\t\t\tbreak;\n> +\t\t\tdefault:\n> +\t\t\t\tret = 0;\n> +\t\t\t\terror(\"unsupported protocol version %s for external persistent filter '%s'\",\n> +\t\t\t\t\tnbuf.buf, cmd);\n> +\t\t}\n> +\t}\n> +\tstring_list_clear(&split, 0);\n> +\tstrbuf_release(&nbuf);\n> +\n> +\tif (!ret) {\n> +\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n> +\t\treturn NULL;\n> +\t}\n\nDo we handle persistent filter command being killed before it finishes?\nOr exiting with error? I don't know this Git API...\n\n> +\n> +\thashmap_add(&cmd_process_map, entry);\n> +\treturn entry;\n> +}\n[...]\n\n> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n> index a05a8d2..d9077ea 100755\n> --- a/t/t0021-conversion.sh\n> +++ b/t/t0021-conversion.sh\n> @@ -268,4 +268,174 @@ test_expect_success 'disable filter with empty override' '\n> \ttest_must_be_empty err\n> '\n> \n> +test_expect_success 'required protocol filter should filter data' '\n> +\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n\nPerhaps align it?\n\n +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n\n\n> diff --git a/t/t0021/rot13.pl b/t/t0021/rot13.pl\n\nThat's bit more than rot13... but it might be O.K. for a filename here.\n\n> new file mode 100755\n> index 0000000..f2d7a03\n> --- /dev/null\n> +++ b/t/t0021/rot13.pl\n> @@ -0,0 +1,80 @@\n> +#!/usr/bin/env perl\n\nDon't we use other way to specify perl path for Git, and for its\ntest suite?\n\n> +#\n> +# Example implementation for the Git filter protocol version 1\n> +# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n> +#\n> +\n> +use strict;\n> +use warnings;\n> +use autodie;\n\nautodie?\n\n> +\n> +sub rot13 {\n> + my ($str) = @_;\n> + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n> + return $str;\n> +}\n> +\n> +$| = 1; # autoflush STDOUT\n\nPerhaps *STDOUT->autoflush(1), if I remember my Perl correctly?\nShould this matter? Why it is needed?\n\n> +\n> +open my $debug, \">>\", \"output.log\";\n> +$debug->autoflush(1);\n> +\n> +print $debug \"start\\n\";\n> +\n> +print STDOUT \"git-filter-protocol\\nversion 1\";\n> +print $debug \"wrote version\\n\";\n> +\n> +while (1) {\n> + my $command = ;\n> + unless (defined($command)) {\n> + exit();\n> + }\n> + chomp $command;\n> + print $debug \"IN: $command\";\n> + my $filename = ;\n> + chomp $filename;\n> + print $debug \" $filename\";\n> + my $filelen = ;\n> + chomp $filelen;\n> + print $debug \" $filelen\";\n> +\n> + $filelen = int($filelen);\n> + my $output;\n> +\n> + if ( $filelen > 0 ) {\n\nInconsistent style. You use\n\n unless (defined($command)) {\n\nwithout extra whitespace after and before parentheses, but\n\n if ( $filelen > 0 ) {\n\ninstead of simply\n\n if ($filelen > 0) {\n\n> + my $input;\n> + {\n> + binmode(STDIN);\n> + my $bytes_read = 0;\n> + $bytes_read = read STDIN, $input, $filelen;\n> + if ( $bytes_read != $filelen ) {\n> + die \"not enough to read\";\n\nI know it's only a test script (well, a part of one), but we would probably\nwant to have more information in the case of a real filter.\n\n> + }\n> + print $debug \" [OK] -- \";\n> + }\n> +\n> + if ( $command eq \"clean\") {\n> + $output = rot13($input);\n> + }\n> + elsif ( $command eq \"smudge\" ) {\n\nStyle; I think we use\n\n + } elsif ( $command eq \"smudge\" ) {\n\n> + $output = rot13($input);\n> + }\n> + else {\n> + die \"bad command\\n\";\n\nSame here (both about style, and error message).\n\n> + }\n> + }\n\nWhat happens if $filelen is zero, or negative? Ah, I see that $output\nwould be undef... which is bad, I think.\n\n> +\n> + my $output_len = length($output);\n> + print STDOUT \"$output_len\\n\";\n> + print $debug \"OUT: $output_len\";\n> + if ( $output_len > 0 ) {\n> + if ( ($command eq \"clean\" and $filename eq \"clean-write-fail.r\") or\n> + ($command eq \"smudge\" and $filename eq \"smudge-write-fail.r\") ) {\n\nHardcoded filenames, without it being described in the file header?\n\n> + print STDOUT \"fail\";\n\nThis is not defined in the protocol description! Unless anything that\ndoes not conform to the specification would work here, but at least it\nis a recommended practice to be described in the documentation, don't\nyou think?\n\nWhat would happen in $output_len is 4?\n\n> + print $debug \" [FAIL]\\n\"\n> + } else {\n> + print STDOUT $output;\n> + print $debug \" [OK]\\n\";\n> + }\n> + }\n> +}\n\n-- \nJakub Narębski\n \n\n"},{"id":"292030","messageId":"20160723072721.GA20875@starla","threadId":"42904","inReplyTo":"5792B622.5040306@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-23T07:27:21Z","receivedAt":"2016-07-23T07:27:28Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jakub Narębski wrote:\n> W dniu 2016-07-22 o 17:49, larsxschneider@gmail.com pisze:\n> > +use strict;\n> > +use warnings;\n> > +use autodie;\n> \n> autodie?\n\n\"set -e\" for Perl (man autodie)\n\nIt's been a part of Perl for ages, but I've never used it\nmyself, either; I suppose it's fine for tests...\n\n> > +$| = 1; # autoflush STDOUT\n> \n> Perhaps *STDOUT->autoflush(1), if I remember my Perl correctly?\n> Should this matter? Why it is needed?\n\nIt's better to always disable automatic output buffering when\nwriting to pipes or sockets for IPC. Otherwise output may be\nbuffered indefinitely because the buffering mechanism doesn't\nknow a reader is stalled.\n\nSame problem with using stdio.h functions for IPC in C.\n"},{"id":"292032","messageId":"20160723081447.GA24318@starla","threadId":"42904","inReplyTo":"20160722154900.19477-4-larsxschneider@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-23T08:14:47Z","receivedAt":"2016-07-23T08:14:54Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"larsxschneider@gmail.com wrote:\n> Please note that the protocol filters do not support stream processing\n> with this implemenatation because the filter needs to know the length of\n> the result in advance. A protocol version 2 could address this in a\n> future patch.\n\nWould it be prudent to reuse pkt-line for this?\n\n> +static void stop_protocol_filter(struct cmd2process *entry) {\n> +\tif (!entry)\n> +\t\treturn;\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tclose(entry->process.in);\n> +\tclose(entry->process.out);\n> +\tsigchain_pop(SIGPIPE);\n> +\tfinish_command(&entry->process);\n> +\tchild_process_clear(&entry->process);\n> +\thashmap_remove(&cmd_process_map, entry, NULL);\n> +\tfree(entry);\n> +}\n> +\n> +static struct cmd2process *start_protocol_filter(const char *cmd)\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry = NULL;\n> +\tstruct child_process *process = NULL;\n\nThese are unconditionally set below, so initializing to NULL\nmay hide future bugs.\n\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n> +\tconst char *argv[] = { NULL, NULL };\n> +\tconst char *header = \"git-filter-protocol\\nversion\";\n\n\tstatic const char header[] = \"git-filter-protocol\\nversion\";\n\n...might be smaller by avoiding the extra pointer\n(but compilers ought to be able to optimize it)\n\n> +\tentry = xmalloc(sizeof(*entry));\n> +\thashmap_entry_init(entry, strhash(cmd));\n> +\tentry->cmd = cmd;\n> +\tprocess = &entry->process;\n\n\n\n> +\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n\nstarts_with() is probably more readable, here.\n\n> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n> +\t\t\t\t\t\tconst char *filter_type)\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry = NULL;\n> +\tstruct child_process *process = NULL;\n\nI would leave process initialized, here, since it should\nalways be set below:\n\n> +\tstruct stat fileStat;\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n> +\tsize_t nbuf_len;\n> +\tchar *strtol_end;\n> +\tchar c;\n> +\n> +\tif (!cmd || !*cmd)\n> +\t\treturn 0;\n> +\n> +\tif (!dst)\n> +\t\treturn 1;\n> +\n> +\tif (!cmd_process_map_init) {\n> +\t\tcmd_process_map_init = 1;\n> +\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n> +\t} else {\n> +\t\tentry = find_protocol_filter_entry(cmd);\n> +\t}\n> +\n> +\tif (!entry){\n> +\t\tentry = start_protocol_filter(cmd);\n> +\t\tif (!entry) {\n> +\t\t\tstop_protocol_filter(entry);\n\nstop_protocol_filter is a no-op, here, since entry is NULL\n\n> +\t\t\treturn 0;\n> +\t\t}\n> +\t}\n> +\tprocess = &entry->process;\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tswitch (entry->protocol) {\n> +\t\tcase 1:\n> +\t\t\tif (fd >= 0 && !src) {\n> +\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n> +\t\t\t\tlen = fileStat.st_size;\n\nThere's a truncation bug when sizeof(size_t) < sizeof(off_t)\n(and mixedCase is inconsistent with our style)\n\n> + my $filelen = ;\n> + chomp $filelen;\n> + print $debug \" $filelen\";\n> +\n> + $filelen = int($filelen);\n\nCalling int() here is unnecessary and may hide bugs if you\nforget to check $debug. Perhaps a regexp check is safer:\n\n\t$filelen =~ /\\A\\d+\\z/ or die \"bad filelen: $filelen\\n\";\n"},{"id":"292055","messageId":"7DABD0C7-8838-4FEC-B6E5-36F82E521312@gmail.com","threadId":"42904","inReplyTo":"xmqqshv18i8i.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v1 0/3] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-24T11:24:29Z","receivedAt":"2016-07-24T11:24:34Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 22 Jul 2016, at 23:39, Junio C Hamano wrote:\n\n> larsxschneider@gmail.com writes:\n> \n>> The first two patches are cleanup patches which are not really necessary\n>> for the feature.\n> \n> These two looked trivially good.\nThanks!\n\n\n> I think I can agree with what 3/3 wants to do in principle, but\n> \n> * \"protocol\" is not quite the right word. The current way to\n> interact with clean and smudge filters can be considered using a\n> different \"protocol\", that conveys the data and the options via\n> the command line and pipe. The most distinguishing feature that\n> differentiates the old way and the new style this change allows\n> is that it allows you to have a single instance of the process\n> running that can be reused?\nI agree that the name is not ideal. When I started working on the\nfeatured I called it \"streaming\" but then I read your comment in\n$gmane/299863 and realized that this would be a misleading name.\nAfterwards I called it \"persistent\"/\"long running\" but then I thought \nthis term could trick people into thinking that this is some kind of \ndaemon. Somehow I want to convey that the filter is persistent for \none Git invocation only.\n\nWhat if we would keep the config option \"protocol\" and make it an \"int\"? \nUndefined or version \"1\" would describe the existing clean/smudge \nprotocol via command line and pipe. Version \"2\" would be the new protocol?\n\n\n> * I am not sure what's the pros-and-cons in forcing people writing\n> a single program that can do both cleaning and smudging. You\n> cannot have only \"smudge\" side that uses the long-running process\n> while \"clean\" side that runs single-shot invocation with this\n> design, which I'd imagine would be a downside. If you are going\n> to use a long-running process interface for both sides, this\n> design allows you to do it with fewer number of processes, which\n> may be an upside.\nWe could define the protocol for clean and smudge individually. However,\nif you have implemented the more complicated long-running protocol already\nfor one filter, then you could reuse the code for the other filter, too, as\nthis protocol is, as far as I can see, always more efficient (assuming you \nhave source code access to both filters). Another argument could be that we \ndon't define the \"required\" flag for the filters individually either.\n\n\n> * The way the serialized access to these long-running processes\n> work in 3/3 would make it harder or impossible to later\n> parallelize conversion? I am imagining a far future where we\n> would run \"git checkout .\" using (say) two threads, one\n> responsible for active_cache[0..active_nr/2] and the other\n> responsible for the remainder.\nI hope this future is not too far away :-) \nHowever, I don't think that would be a problem as we could start the\nlong-running process once for each checkout thread, no?\n\n\nThank you,\nLars\n"},{"id":"292056","messageId":"D4012F2A-9774-408B-A355-B7442A4FBF26@gmail.com","threadId":"42904","inReplyTo":"32d8feda-0fff-6c8c-1ac3-9cc3d783d0ef@web.de","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-24T12:09:33Z","receivedAt":"2016-07-24T12:09:40Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 23 Jul 2016, at 00:32, Torsten Bögershausen wrote:\n\n> On 07/22/2016 05:49 PM, larsxschneider@gmail.com wrote:\n>> From: Lars Schneider \n>> \n>> [...]\n>> \n>> 1. Git starts the filter on first usage and expects a welcome message\n>> with protocol version number:\n>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>> \tGit <-- Filter: \"version 1\"\n> Is there no terminator here ?\n> How long should the reading side wait without a '\\n', or should it read\n> \"version 1\\n\" ?\nI agree, I will add the \"\\n\" terminator!\n\n\n>> [...]\n>> \n>> Please note that the protocol filters do not support stream processing\n>> with this implemenatation because the filter needs to know the length of\n> ^^^^^^^^^^^^^^^^typo\nThanks!\n\n\n>> [...]\n>> \n>> +\n>> +test_expect_success EXPENSIVE 'protocol filter large file' '\n>> +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n>> +\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n>> +\trm -rf repo &&\n>> +\tmkdir repo &&\n>> +\t(\n>> +\t\tcd repo &&\n>> +\t\tgit init &&\n>> +\n>> +\t\techo \"2GB filter=largefile\" >.gitattributes &&\n>> +\t\tfor i in $(test_seq 1 2048); do printf \"%1048576d\" 1; done >2GB &&\n> Side question:\n> Is there a way to \"re-use\" the 2GB test file through t0021?\n> It takes a long time to produce it, especially on my 32 Bit systems ;-)\n> But this may be a different patch.\nI would like to keep the tests as unentangled as possible and therefore a direct\nreuse might not be ideal. However, I could add a new \"EXPENSIVE setup test\" that \nprepares the file for both tests.\n\n\n>> [...]\n>> +\n>> +\t\tprintf \"\" >output.log &&\n> Is this the same as\n> >output.log\n> to produce an empty file ?\nYes, thank you :-)\n\n\n>> [...]\n>> +++ b/t/t0021/rot13.pl\n>> @@ -0,0 +1,80 @@\n>> +#!/usr/bin/env perl\n> Should this be\n> \"$PERL_PATH\" ?\nI think we can't use this variable directly in the script. I could create the script file \nfor the test and set the shebang to this value. However, no other \"Perl file test\" does it\nand therefore I wonder if it is necessary:\nt/t0202/test.pl\nt/t9000/test.pl\nt/t9700/test.pl\nAccording to the documentation this is useful to avoid trouble on Windows. I will check\nthis test on Windows.\n\nI also just noticed that all other Perl tests use \"#!/usr/bin/perl\". Should I change mine\nto match those?\n\n\n> And do we need to protect the TC with\n> test_have_prereq PERL or similar ?\nProbably not as the documentation states \"Even without the PERL prerequisite, tests can \nassume there is a usable perl interpreter\". However, all other Perl file tests do the same\nand therefore I think it is a good idea.\n\n\n>> [...]\n>> +\n>> +print STDOUT \"git-filter-protocol\\nversion 1\";\n> Again, I don't like the missing terminator here.\n> What if we step up to protocol \"version 10\" ?\n> Could it work to use one and only one line,\n> with one terminator, like this ?\n> print STDOUT \"git-filter-protocol version 1\\1\";\nThe missing terminator was a mistake. As mentioned above, \nI will add it!\n\n\nThanks for the review,\nLars"},{"id":"292061","messageId":"1A0C148F-C7C3-4FAF-BAEE-58B11A2324FF@gmail.com","threadId":"42904","inReplyTo":"9f47cf44-7163-a7a7-c1f0-87ebdee65b37@ramsayjones.plus.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-24T17:16:45Z","receivedAt":"2016-07-24T17:16:56Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 23 Jul 2016, at 01:19, Ramsay Jones wrote:\n\n> On 22/07/16 16:49, larsxschneider@gmail.com wrote:\n>> From: Lars Schneider \n>> \n>> Git's clean/smudge mechanism invokes an external filter process for every\n>> single blob that is affected by a filter. If Git filters a lot of blobs\n>> then the startup time of the external filter processes can become a\n>> significant part of the overall Git execution time.\n>> \n>> This patch adds the filter..useProtocol option which, if enabled,\n>> keeps the external filter process running and processes all blobs with\n>> the following protocol over stdin/stdout.\n>> \n>> 1. Git starts the filter on first usage and expects a welcome message\n>> with protocol version number:\n>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>> \tGit <-- Filter: \"version 1\"\n> \n> Hmm, I was a bit surprised to see a 'filter' talk first (but so long as the\n> interaction is fully defined, I guess it doesn't matter).\n> \n> [If you wanted to check for a version, you could add a \"version\" command\n> instead, just like \"clean\" and \"smudge\".]\n\nIt was a conscious decision to have the `filter` talk first. My reasoning was:\n\n(1) I want a reliable way to distinguish the existing filter protocol (\"single-shot \ninvocation\") from the new one (\"long running\"). I don't think there would be a\nsituation where the existing protocol would talk first. Therefore the users would\nnot accidentally mix them with a possibly half working, undetermined, outcome.\n\n(2) In the future we could extend the pipe protocol (see $gmane/297994, it's very\ninteresting). A filter could check Git's version and then pick the most appropriate\nfilter protocol on startup.\n\n\n> [...]\n>> +static struct cmd2process *start_protocol_filter(const char *cmd)\n>> +{\n>> +\tint ret = 1;\n>> +\tstruct cmd2process *entry = NULL;\n>> +\tstruct child_process *process = NULL;\n>> +\tstruct strbuf nbuf = STRBUF_INIT;\n>> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n>> +\tconst char *argv[] = { NULL, NULL };\n>> +\tconst char *header = \"git-filter-protocol\\nversion\";\n>> +\n>> +\tentry = xmalloc(sizeof(*entry));\n>> +\thashmap_entry_init(entry, strhash(cmd));\n>> +\tentry->cmd = cmd;\n>> +\tprocess = &entry->process;\n>> +\n>> +\tchild_process_init(process);\n>> +\targv[0] = cmd;\n>> +\tprocess->argv = argv;\n>> +\tprocess->use_shell = 1;\n>> +\tprocess->in = -1;\n>> +\tprocess->out = -1;\n>> +\n>> +\tif (start_command(process)) {\n>> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n>> +\t\treturn NULL;\n>> +\t}\n>> +\tstrbuf_reset(&nbuf);\n>> +\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n> \n> Hmm, how much will be read into nbuf by this single call?\n> Since strbuf_read_once() makes a single call to xread(), with\n> a len argument that will probably be 8192, you can not really\n> tell how much it will read, in general. (xread() does not\n> guarantee how many bytes it will read.)\n> \n> In particular, it could be less than strlen(header).\n\nAs mentioned to Torsten in $gmane/300156, I will add a newline\nand then read until I find the second newline. That should solve\nthe problem, right?\n\n(You wrote in $gmane/300119 that I should ignore your email but\nI think you have a valid point here ;-)\n\n\n>> [...]\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tswitch (entry->protocol) {\n>> +\t\tcase 1:\n>> +\t\t\tif (fd >= 0 && !src) {\n>> +\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n>> +\t\t\t\tlen = fileStat.st_size;\n>> +\t\t\t}\n>> +\t\t\tstrbuf_reset(&nbuf);\n>> +\t\t\tstrbuf_addf(&nbuf, \"%s\\n%s\\n%zu\\n\", filter_type, path, len);\n>> +\t\t\tret &= write_str_in_full(process->in, nbuf.buf) > 1;\n> \n> why not write_in_full(process->in, nbuf.buf, nbuf.len) ?\nOK, this would save a \"strlen\" call. Do you think such a function could be of general\nuse? If yes, then I would add:\n\nstatic inline ssize_t write_strbuf_in_full(int fd, struct strbuf *str)\n{\n\treturn write_in_full(fd, str->buf, str->len);\n}\n\n\n>> +\t\t\tif (len > 0) {\n>> +\t\t\t\tif (src)\n>> +\t\t\t\t\tret &= write_in_full(process->in, src, len) == len;\n>> +\t\t\t\telse if (fd >= 0)\n>> +\t\t\t\t\tret &= copy_fd(fd, process->in) == 0;\n>> +\t\t\t\telse\n>> +\t\t\t\t\tret &= 0;\n>> +\t\t\t}\n>> +\n>> +\t\t\tstrbuf_reset(&nbuf);\n>> +\t\t\twhile (xread(process->out, &c, 1) == 1 && c != '\\n')\n>> +\t\t\t\tstrbuf_addchars(&nbuf, c, 1);\n>> +\t\t\tnbuf_len = (size_t)strtol(nbuf.buf, &strtol_end, 10);\n>> +\t\t\tret &= (strtol_end != nbuf.buf && errno != ERANGE);\n>> +\t\t\tstrbuf_reset(&nbuf);\n>> +\t\t\tif (nbuf_len > 0)\n>> +\t\t\t\tret &= strbuf_read_once(&nbuf, process->out, nbuf_len) == nbuf_len;\n> \n> Again, how many bytes will be read?\n> Note, that in the default configuration, a _maximum_ of\n> MAX_IO_SIZE (8MB or SSIZE_MAX, whichever is smaller) bytes\n> will be read.\nWould something like this be more appropriate?\n\nstrbuf_reset(&nbuf);\nif (nbuf_len > 0) {\n strbuf_grow(&nbuf, nbuf_len);\n ret &= read_in_full(process->out, nbuf.buf, nbuf_len) == nbuf_len;\n}\n\n\nThanks for the review,\nLars\n\n"},{"id":"292066","messageId":"0FB8A921-3465-4081-A36D-D2F19A8ED63D@gmail.com","threadId":"42904","inReplyTo":"5792B622.5040306@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-24T18:36:50Z","receivedAt":"2016-07-24T18:37:15Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 23 Jul 2016, at 02:11, Jakub Narębski wrote:\n\n> W dniu 2016-07-22 o 17:49, larsxschneider@gmail.com pisze:\n>> From: Lars Schneider \n> \n> Nb. this line is only needed if you want author name and/or date\n> different from the email sender, or if you have sender line misconfigured\n> (e.g. lacking the human readable name).\n\nI use \"git format-patch\" to generate these emails:\n\ngit format-patch --cover-letter --subject-prefix=\"PATCH ...\" -M $BASE -o $OUTPUT\n\nHow would I disable this line? (I already checked the man page to no avail).\nPlus, what does \"Nb\" stand for? :-)\n\n\n>> Git's clean/smudge mechanism invokes an external filter process for every\n>> single blob that is affected by a filter. If Git filters a lot of blobs\n>> then the startup time of the external filter processes can become a\n>> significant part of the overall Git execution time.\n> \n> Do I understand it correctly (from the commit description) that with\n> this new option you start one filter process for the whole life of\n> the Git command (e.g. `git add .`), which may perform more than one\n> cleanup, but do not create a daemon or daemon-like process which\n> would live for several commands invocations?\n\nCorrect!\n\n\n>> This patch adds the filter..useProtocol option which, if enabled,\n>> keeps the external filter process running and processes all blobs with\n>> the following protocol over stdin/stdout.\n> \n> I agree with Junio that the name \"useProtocol\" is bad, and not quite\n> right. Perhaps \"persistent\" would be better? Also, what is the value\n> of `filter..useProtocol`: boolean? or a script name?\n\nI agree that the name is not ideal. \"UseProtocol\" as it is would be a boolean. \nI thought about \"persistent\" but this name wouldn't convey the scope of the \npersistency (\"persistent for one Git operation\" vs. \"persistent for many Git \noperations\"). What do you think about the protocol as int version idea\ndescribed in $gmane/300155 ?\n\n\n> I also agree that we might wat to be able to keep clean and smudge\n> filters separate, but be able to run a single program if they are\n> both the same. I think there is a special case for filter unset,\n> and/or filter being \"cat\" -- we would want to keep that.\n\nSince 1a8630d there is a more efficient way to unset a filter ;-)\nCan you think of other cases where the separation would be useful?\n\n\n> My proposal is to use `filter..persistent` as an addition\n> to 'clean' and 'smudge' variables, with the following possible\n> values:\n> \n> * none (the default)\n> * clean\n> * smudge\n> * both\n\nThat could work. However, I am not convinced, yet, that separate\nfilters are an actual use case.\n\n\n> I assume that either Git would have to start multiple filter\n> commands for multi-threaded operation, or the protocol would have\n> to be extended to make persistent filter fork itself.\n\nI think it would be better to have Git start multiple filter commands\nto keep the protocol as simple and error free as possible.\n\n\n> BTW. what would happen in your original proposal if the user had\n> *both* filter..useProtocol and filter..smudge\n> (and/or filter..clean) set?\n\nThat wouldn't be an issue as \"useProtocol\" is just a boolean that\ntells Git how to talk to \"filter..smudge\" and \"filter..clean\".\nI need to make this more clear in the documentation.\n\n\n>> 1. Git starts the filter on first usage and expects a welcome message\n>> with protocol version number:\n>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>> \tGit <-- Filter: \"version 1\"\n> \n> I was wondering how Git would know that filter executable was started,\n> but then I realized it was once-per-command invocation, not a daemon.\n> \n> I agree with Torsten that there should be a terminator after the\n> version number.\n\nI agree, too :)\n\n\n> Also, for future extendability this should be probably followed by\n> possibly empty list of script capabilities, that is:\n> \n> \tGit <-- Filter: \"git-filter-protocol\\n\"\n> \tGit <-- Filter: \"version 1.1\\n\"\n> \tGit <-- Filter: \"capabilities clean smudge\\n\"\n> \n> Or we can add capabilities in later version...\n\nThat is an interesting idea. My initial thought was to make the capabilities\nof a certain version fix. If we want to add new capabilities then we would \nbump the version. I wonder what others think about your suggestion!\n\n\n> BTW. why not follow e.g. HTTP protocol example, and use\n> \n> \tGit <-- Filter: \"git-filter-protocol/1\\n\"\n\nI think my proposal is a bit more explicit as it states \"version\". If\nyou feel strongly about it, I could be convinced otherwise.\n\n\n>> 2. Git sends the command (either \"smudge\" or \"clean\"), the filename, the\n>> content size in bytes, and the content separated by a newline character:\n>> \tGit --> Filter: \"smudge\\n\"\n> \n> Would it help (for some cases) to pass the name of filter that\n> is being invoked?\n\nInteresting thought! Can you imagine a case where this would be useful?\n\n\n>> \tGit --> Filter: \"testfile.dat\\n\"\n> \n> Unfortunately, while sane filenames should not contain newlines[1],\n> the unfortunate fact is that *filenames can include newlines*, and\n> you need to be able to handle that[2]. Therefore you need either to\n> choose a different separator (the only one that can be safely used\n> is \"\\0\", i.e. the NUL character - but it is not something easy to\n> handle by shell scripts), or C-quote filenames as needed, or always\n> C-quote filenames. C-quoting at minimum should include quoting newline\n> character, and the escape character itself.\n> \n> BTW. is it the basename of a file, or a full pathname?\n> \n> [1]: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n> [2]: http://www.dwheeler.com/essays/filenames-in-shell.html\n\nThanks for this explanation. A bash version of the protocol is not\ntrivial (I tried it but ended up using Perl). Therefore I think \"\\0\"\nwould be a good choice?\n\n\n>> \tGit --> Filter: \"7\\n\"\n> \n> That's the content size in bytes written as an ASCII number.\n\nCorrect.\n\n\n>> \tGit --> Filter: \"CONTENT\"\n> \n> Can filter ignore the content size, and just read all what it was\n> sent, that is until eof or something?\n\nWhat would that something be? Since CONTENT is binary it can contain\nany character (even \"\\0\")...\n\n\n>> 3. The filter is expected to answer with the result content size in\n>> bytes and the result content separated by a newline character:\n>> \tGit <-- Filter: \"15\\n\"\n>> \tGit <-- Filter: \"SMUDGED_CONTENT\"\n> \n> I wonder how hard would be to write filters for this protocol...\n\nEasy :-) Plus you can look at a Perl (see t/t0021) and a golang implementation\nalready (https://github.com/github/git-lfs/pull/1382).\n\n\n>> 4. The filter is expected to wait for the next file in step 2, again.\n>> \n>> Please note that the protocol filters do not support stream processing\n>> with this implemenatation because the filter needs to know the length of\n> ^^^^^^^^~^^^^^^\n> \n> implementation\nThanks!\n\n\n>> the result in advance. A protocol version 2 could address this in a\n>> future patch.\n>> \n>> Signed-off-by: Lars Schneider \n>> Helped-by: Martin-Louis Bright \n>> ---\n>> Documentation/gitattributes.txt | 41 +++++++-\n>> convert.c | 210 ++++++++++++++++++++++++++++++++++++++--\n>> t/t0021-conversion.sh | 170 ++++++++++++++++++++++++++++++++\n> \n> Wouldn't it be better to name the test case something more\n> descriptive, for example\n> \n> t/t0021-filter-driver-useProtocol.sh\n> \n> The name of test should be adjusted to final name of the feature,\n> of course.\n\nI think the prefix numbers should be unique, no? And t0022 is already taken.\n\n\n>> t/t0021/rot13.pl | 80 +++++++++++++++\n>> 4 files changed, 494 insertions(+), 7 deletions(-)\n>> create mode 100755 t/t0021/rot13.pl\n>> \n>> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n>> index 8882a3e..7026d62 100644\n>> --- a/Documentation/gitattributes.txt\n>> +++ b/Documentation/gitattributes.txt\n>> @@ -300,7 +300,10 @@ checkout, when the `smudge` command is specified, the command is\n>> fed the blob object from its standard input, and its standard\n>> output is used to update the worktree file. Similarly, the\n>> `clean` command is used to convert the contents of worktree file\n>> -upon checkin.\n>> +upon checkin. By default these commands process only a single\n>> +blob and terminate. If the setting filter..useProtocol is\n>> +enabled then Git can process all blobs with a single filter command\n>> +invocation (see filter protocol below).\n> \n> This does not tell the precedence between `smudge`, `clean` and\n> filter..useProtocol, see above. Also, discrepancy in how\n> config variables are referenced.\n\nAs mentioned above \"useProtocol\" is a boolean. Therefore precedence shouldn't\nbe a problem. What do you mean by \"discrepancy in how config variables are \nreferenced\"?\n\n\n>> One use of the content filtering is to massage the content into a shape\n>> that is more convenient for the platform, filesystem, and the user to use.\n>> @@ -375,6 +378,42 @@ substitution. For example:\n>> ------------------------\n>> \n>> \n>> +Filter Protocol\n>> +^^^^^^^^^^^^^^^\n>> +\n>> +If the setting filter..useProtocol is enabled then Git\n> \n> This seems to tell that `useProtocol` is boolean-valued (?)\n\nCorrect.\n\n\n>> +can process all blobs with a single filter command invocation\n>> +by talking with the following protocol over stdin/stdout.\n> \n> Should we use stdin/stdout shortcut, or spell standard input\n> and standard output in full?\n\nI think the documentation is not consistent here but spelled in full\nseems to be used more often. I will change it!\n\n\n>> diff --git a/convert.c b/convert.c\n>> index 522e2c5..91ce86f 100644\n>> --- a/convert.c\n>> +++ b/convert.c\n>> @@ -481,12 +481,188 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n>> \treturn ret;\n>> }\n>> \n>> +static int cmd_process_map_init = 0;\n>> +static struct hashmap cmd_process_map;\n>> +\n>> +struct cmd2process {\n>> +\tstruct hashmap_entry ent; /* must be the first member! */\n>> +\tconst char *cmd;\n>> +\tlong protocol;\n>> +\tstruct child_process process;\n>> +};\n> [...]\n>> +static struct cmd2process *find_protocol_filter_entry(const char *cmd)\n>> +{\n>> +\tstruct cmd2process k;\n>> +\thashmap_entry_init(&k, strhash(cmd));\n>> +\tk.cmd = cmd;\n>> +\treturn hashmap_get(&cmd_process_map, &k, NULL);\n> \n> Should we use global variable cmd_process_map, or pass it as parameter?\n> The same question apply for other procedures and functions.\n> \n> Note that I am not saying that it is a bad thing to use global\n> variable here.\n\nPassing it would be nicer as this would make at least a few functions \"pure\".\nI will change that!\n\n\n> [...]\n>> +static struct cmd2process *start_protocol_filter(const char *cmd)\n>> +{\n>> +\tint ret = 1;\n>> +\tstruct cmd2process *entry = NULL;\n>> +\tstruct child_process *process = NULL;\n>> +\tstruct strbuf nbuf = STRBUF_INIT;\n>> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n>> +\tconst char *argv[] = { NULL, NULL };\n>> +\tconst char *header = \"git-filter-protocol\\nversion\";\n>> +\n>> +\tentry = xmalloc(sizeof(*entry));\n>> +\thashmap_entry_init(entry, strhash(cmd));\n>> +\tentry->cmd = cmd;\n>> +\tprocess = &entry->process;\n>> +\n>> +\tchild_process_init(process);\n>> +\targv[0] = cmd;\n>> +\tprocess->argv = argv;\n>> +\tprocess->use_shell = 1;\n>> +\tprocess->in = -1;\n>> +\tprocess->out = -1;\n>> +\n>> +\tif (start_command(process)) {\n>> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n>> +\t\treturn NULL;\n>> +\t}\n>> +\tstrbuf_reset(&nbuf);\n> \n> Is strbuf_reset needed here? We have not used nbuf variable yet.\n\nAgreed, not needed!\n\n\n>> +\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n>> +\tsigchain_pop(SIGPIPE);\n>> +\n>> +\tstrbuf_stripspace(&nbuf, 0);\n>> +\tstring_list_split_in_place(&split, nbuf.buf, ' ', 2);\n>> +\tret &= split.nr > 1;\n>> +\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n>> +\tif (ret) {\n>> +\t\tentry->protocol = strtol(split.items[1].string, NULL, 10);\n> \n> This does not handle at least some errors in version number parsing,\n> for example junk after version number. Don't we have some helper\n> functions for this?\n\nI am not sure. I haven't found one.\n\n> Nb. this code makes it so that the version number must be integer.\n\nNb? :-)\n\n\n>> +\t\tswitch (entry->protocol) {\n>> +\t\t\tcase 1:\n>> +\t\t\t\tbreak;\n>> +\t\t\tdefault:\n>> +\t\t\t\tret = 0;\n>> +\t\t\t\terror(\"unsupported protocol version %s for external persistent filter '%s'\",\n>> +\t\t\t\t\tnbuf.buf, cmd);\n>> +\t\t}\n>> +\t}\n>> +\tstring_list_clear(&split, 0);\n>> +\tstrbuf_release(&nbuf);\n>> +\n>> +\tif (!ret) {\n>> +\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n>> +\t\treturn NULL;\n>> +\t}\n> \n> Do we handle persistent filter command being killed before it finishes?\n> Or exiting with error? I don't know this Git API...\n\nIf the \"apply_filter\" function fails then Git would proceed and just not\nfilter the content. If you define the \"required\" flag for the filter then\nGit would error in that case.\n\n\n>> +\n>> +\thashmap_add(&cmd_process_map, entry);\n>> +\treturn entry;\n>> +}\n> [...]\n> \n>> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n>> index a05a8d2..d9077ea 100755\n>> --- a/t/t0021-conversion.sh\n>> +++ b/t/t0021-conversion.sh\n>> @@ -268,4 +268,174 @@ test_expect_success 'disable filter with empty override' '\n>> \ttest_must_be_empty err\n>> '\n>> \n>> +test_expect_success 'required protocol filter should filter data' '\n>> +\ttest_config_global filter.protocol.smudge \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n>> +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n> \n> Perhaps align it?\n> \n> +\ttest_config_global filter.protocol.clean \\\"$TEST_DIRECTORY/t0021/rot13.pl\\\" &&\n\nOK.\n\n\n>> diff --git a/t/t0021/rot13.pl b/t/t0021/rot13.pl\n> \n> That's bit more than rot13... but it might be O.K. for a filename here.\n\n\"rot13-$FEATURE_NAME.pl\" ?\n\n\n>> new file mode 100755\n>> index 0000000..f2d7a03\n>> --- /dev/null\n>> +++ b/t/t0021/rot13.pl\n>> @@ -0,0 +1,80 @@\n>> +#!/usr/bin/env perl\n> \n> Don't we use other way to specify perl path for Git, and for its\n> test suite?\n\nOther tests use \"#!/usr/bin/perl\" - I will change that.\nSee $gmane/300156.\n\n\n>> +#\n>> +# Example implementation for the Git filter protocol version 1\n>> +# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n>> +#\n>> +\n>> +use strict;\n>> +use warnings;\n>> +use autodie;\n> \n> autodie?\n\nSee $gmane/300124.\n\n\n>> +\n>> +sub rot13 {\n>> + my ($str) = @_;\n>> + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n>> + return $str;\n>> +}\n>> +\n>> +$| = 1; # autoflush STDOUT\n> \n> Perhaps *STDOUT->autoflush(1), if I remember my Perl correctly?\n> Should this matter? Why it is needed?\n\nAs recommended by Eric in $gmane/300124 I will run flush explicitly.\n\n\n>> +\n>> +open my $debug, \">>\", \"output.log\";\n>> +$debug->autoflush(1);\n>> +\n>> +print $debug \"start\\n\";\n>> +\n>> +print STDOUT \"git-filter-protocol\\nversion 1\";\n>> +print $debug \"wrote version\\n\";\n>> +\n>> +while (1) {\n>> + my $command = ;\n>> + unless (defined($command)) {\n>> + exit();\n>> + }\n>> + chomp $command;\n>> + print $debug \"IN: $command\";\n>> + my $filename = ;\n>> + chomp $filename;\n>> + print $debug \" $filename\";\n>> + my $filelen = ;\n>> + chomp $filelen;\n>> + print $debug \" $filelen\";\n>> +\n>> + $filelen = int($filelen);\n>> + my $output;\n>> +\n>> + if ( $filelen > 0 ) {\n> \n> Inconsistent style. You use\n> \n> unless (defined($command)) {\n> \n> without extra whitespace after and before parentheses, but\n> \n> if ( $filelen > 0 ) {\n> \n> instead of simply\n> \n> if ($filelen > 0) {\n\nAgreed. I will fix it.\n\n\n>> + my $input;\n>> + {\n>> + binmode(STDIN);\n>> + my $bytes_read = 0;\n>> + $bytes_read = read STDIN, $input, $filelen;\n>> + if ( $bytes_read != $filelen ) {\n>> + die \"not enough to read\";\n> \n> I know it's only a test script (well, a part of one), but we would probably\n> want to have more information in the case of a real filter.\n\nTrue. Do you think there is anything to change in the script, though?\n\n\n> \n>> + }\n>> + print $debug \" [OK] -- \";\n>> + }\n>> +\n>> + if ( $command eq \"clean\") {\n>> + $output = rot13($input);\n>> + }\n>> + elsif ( $command eq \"smudge\" ) {\n> \n> Style; I think we use\n> \n> + } elsif ( $command eq \"smudge\" ) {\n\nOK.\n\n\n> \n>> + $output = rot13($input);\n>> + }\n>> + else {\n>> + die \"bad command\\n\";\n> \n> Same here (both about style, and error message).\n> \n>> + }\n>> + }\n> \n> What happens if $filelen is zero, or negative? Ah, I see that $output\n> would be undef... which is bad, I think.\n\nIs this something we need to consider in the test script?\n\n\n>> +\n>> + my $output_len = length($output);\n>> + print STDOUT \"$output_len\\n\";\n>> + print $debug \"OUT: $output_len\";\n>> + if ( $output_len > 0 ) {\n>> + if ( ($command eq \"clean\" and $filename eq \"clean-write-fail.r\") or\n>> + ($command eq \"smudge\" and $filename eq \"smudge-write-fail.r\") ) {\n> \n> Hardcoded filenames, without it being described in the file header?\n\nGood point! I will add a comment!\n\n> \n>> + print STDOUT \"fail\";\n> \n> This is not defined in the protocol description! Unless anything that\n> does not conform to the specification would work here, but at least it\n> is a recommended practice to be described in the documentation, don't\n> you think?\n> \n> What would happen in $output_len is 4?\n\nThen it would work :D\nI understand your point. However, this is not a reference implementation.\nIt is a test script that is supposed to trigger bad behavior which we can test. \nTherefore, I would argue that such a return value is OK. I will document it in \nthe header, though. \n\n\nThanks a lot for your extensive review,\nLars"},{"id":"292070","messageId":"121F8453-3A12-47AE-AE10-021EADC4D995@gmail.com","threadId":"42904","inReplyTo":"20160723081447.GA24318@starla","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-24T19:11:42Z","receivedAt":"2016-07-24T19:11:51Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 23 Jul 2016, at 10:14, Eric Wong wrote:\n\n> larsxschneider@gmail.com wrote:\n>> Please note that the protocol filters do not support stream processing\n>> with this implemenatation because the filter needs to know the length of\n>> the result in advance. A protocol version 2 could address this in a\n>> future patch.\n> \n> Would it be prudent to reuse pkt-line for this?\n\nPeff suggested that, too, in $gmane/299902. However, this would make the\nprotocol a bit more complicated and it wouldn't buy us anything for Git\nlarge file processing filters (my main motivation for this patch) as these \nfilters can't leverage streaming anyways.\n\n\n>> +static void stop_protocol_filter(struct cmd2process *entry) {\n>> +\tif (!entry)\n>> +\t\treturn;\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tclose(entry->process.in);\n>> +\tclose(entry->process.out);\n>> +\tsigchain_pop(SIGPIPE);\n>> +\tfinish_command(&entry->process);\n>> +\tchild_process_clear(&entry->process);\n>> +\thashmap_remove(&cmd_process_map, entry, NULL);\n>> +\tfree(entry);\n>> +}\n>> +\n>> +static struct cmd2process *start_protocol_filter(const char *cmd)\n>> +{\n>> +\tint ret = 1;\n>> +\tstruct cmd2process *entry = NULL;\n>> +\tstruct child_process *process = NULL;\n> \n> These are unconditionally set below, so initializing to NULL\n> may hide future bugs.\n\nOK. I thought it is generally a good thing to initialize a pointer with \nNULL. Can you explain to me how this might hide future bugs?\nI will remove the initialization.\n\n\n>> +\tstruct strbuf nbuf = STRBUF_INIT;\n>> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n>> +\tconst char *argv[] = { NULL, NULL };\n>> +\tconst char *header = \"git-filter-protocol\\nversion\";\n> \n> \tstatic const char header[] = \"git-filter-protocol\\nversion\";\n> \n> ...might be smaller by avoiding the extra pointer\n> (but compilers ought to be able to optimize it)\n\nAgreed!\n\n\n>> +\tentry = xmalloc(sizeof(*entry));\n>> +\thashmap_entry_init(entry, strhash(cmd));\n>> +\tentry->cmd = cmd;\n>> +\tprocess = &entry->process;\n> \n> \n> \n>> +\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n> \n> starts_with() is probably more readable, here.\n\nOK, will fix.\n\n\n>> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n>> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n>> +\t\t\t\t\t\tconst char *filter_type)\n>> +{\n>> +\tint ret = 1;\n>> +\tstruct cmd2process *entry = NULL;\n>> +\tstruct child_process *process = NULL;\n> \n> I would leave process initialized, here, since it should\n> always be set below:\n\nOK, will fix.\n\n\n>> +\tstruct stat fileStat;\n>> +\tstruct strbuf nbuf = STRBUF_INIT;\n>> +\tsize_t nbuf_len;\n>> +\tchar *strtol_end;\n>> +\tchar c;\n>> +\n>> +\tif (!cmd || !*cmd)\n>> +\t\treturn 0;\n>> +\n>> +\tif (!dst)\n>> +\t\treturn 1;\n>> +\n>> +\tif (!cmd_process_map_init) {\n>> +\t\tcmd_process_map_init = 1;\n>> +\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n>> +\t} else {\n>> +\t\tentry = find_protocol_filter_entry(cmd);\n>> +\t}\n>> +\n>> +\tif (!entry){\n>> +\t\tentry = start_protocol_filter(cmd);\n>> +\t\tif (!entry) {\n>> +\t\t\tstop_protocol_filter(entry);\n> \n> stop_protocol_filter is a no-op, here, since entry is NULL\n\nOops - a result of my own refactoring :-) Thank you!\n\n\n>> +\t\t\treturn 0;\n>> +\t\t}\n>> +\t}\n>> +\tprocess = &entry->process;\n>> +\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tswitch (entry->protocol) {\n>> +\t\tcase 1:\n>> +\t\t\tif (fd >= 0 && !src) {\n>> +\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n>> +\t\t\t\tlen = fileStat.st_size;\n> \n> There's a truncation bug when sizeof(size_t) < sizeof(off_t)\n\nOK. What would you suggest to do in that case? Should we just let the\nfilter fail? Is there anything else we could do?\n\n\n> (and mixedCase is inconsistent with our style)\n\nOK, will fix.\n\n\n>> + my $filelen = ;\n>> + chomp $filelen;\n>> + print $debug \" $filelen\";\n>> +\n>> + $filelen = int($filelen);\n> \n> Calling int() here is unnecessary and may hide bugs if you\n> forget to check $debug. Perhaps a regexp check is safer:\n> \n> \t$filelen =~ /\\A\\d+\\z/ or die \"bad filelen: $filelen\\n\";\n\nOK, will fix!\n\n\nThanks for your review,\nLars\n\n\n"},{"id":"292074","messageId":"579521A1.4070501@gmail.com","threadId":"42904","inReplyTo":"0FB8A921-3465-4081-A36D-D2F19A8ED63D@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-24T20:14:25Z","receivedAt":"2016-07-24T20:14:45Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-24 o 20:36, Lars Schneider pisze:\n> On 23 Jul 2016, at 02:11, Jakub Narębski wrote:\n>> W dniu 2016-07-22 o 17:49, larsxschneider@gmail.com pisze:\n>>> From: Lars Schneider \n>>\n>> Nb. this line is only needed if you want author name and/or date\n>> different from the email sender, or if you have sender line misconfigured\n>> (e.g. lacking the human readable name).\n> \n> I use \"git format-patch\" to generate these emails:\n> \n> git format-patch --cover-letter --subject-prefix=\"PATCH ...\" -M $BASE -o $OUTPUT\n> \n> How would I disable this line? (I already checked the man page to no avail).\n\nIf you are using `git send-email` or equivalent, I think it is\nstripped automatically if it is not needed (in you case it was,\nbecause Sender was lacking human readable name... at least I think\nit was because of what my email reader inserted as reply line).\nIf you are using an ordinary email client, you need to remove it\nyourself, if needed.\n\n> Plus, what does \"Nb\" stand for? :-)\n\nNb. (or N.b.) stands for \"nota bene\", which I meant do denote\nas a note on a certain side aspect; I'll switch to \"note\", or\n\"BTW\" / \"by the way\".\n \n>>> Git's clean/smudge mechanism invokes an external filter process for every\n>>> single blob that is affected by a filter. If Git filters a lot of blobs\n>>> then the startup time of the external filter processes can become a\n>>> significant part of the overall Git execution time.\n>>\n>> Do I understand it correctly (from the commit description) that with\n>> this new option you start one filter process for the whole life of\n>> the Git command (e.g. `git add .`), which may perform more than one\n>> cleanup, but do not create a daemon or daemon-like process which\n>> would live for several commands invocations?\n> \n> Correct!\n\nIt would be nice to make it more obvious. \n\n>>> This patch adds the filter..useProtocol option which, if enabled,\n>>> keeps the external filter process running and processes all blobs with\n>>> the following protocol over stdin/stdout.\n>>\n>> I agree with Junio that the name \"useProtocol\" is bad, and not quite\n>> right. Perhaps \"persistent\" would be better? Also, what is the value\n>> of `filter..useProtocol`: boolean? or a script name?\n\nAs you see I was not sure if `useProtocol` was boolean or a script name,\nwhich means that it should be stated more explicitly. Of course this\nwould end to not matter if the way new protocol is used were changed.\n\n> I agree that the name is not ideal. \"UseProtocol\" as it is would be a boolean. \n> I thought about \"persistent\" but this name wouldn't convey the scope of the \n> persistency (\"persistent for one Git operation\" vs. \"persistent for many Git \n> operations\"). What do you think about the protocol as int version idea\n> described in $gmane/300155 ?\n\nYou mean the `protocol` as a config variable name (fully name being\n`filter..protocol`), being integer-valued, isn't it? Wouldn't\n`protocolVersion` be a more explicit?\n\n>> I also agree that we might wat to be able to keep clean and smudge\n>> filters separate, but be able to run a single program if they are\n>> both the same. I think there is a special case for filter unset,\n>> and/or filter being \"cat\" -- we would want to keep that.\n> \n> Since 1a8630d there is a more efficient way to unset a filter ;-)\n> Can you think of other cases where the separation would be useful?\n\nI can't think of any, but it doesn't mean that it does not exist.\nIt also does not mean that you need to consider situation that may\nnot happen. Covering one-way filters, like \"indent\" filter for `clean`,\nshould be enough... they do work with your proposal, don't they?\n\n>> My proposal is to use `filter..persistent` as an addition\n>> to 'clean' and 'smudge' variables, with the following possible\n>> values:\n>>\n>> * none (the default)\n>> * clean\n>> * smudge\n>> * both\n> \n> That could work. However, I am not convinced, yet, that separate\n> filters are an actual use case.\n\nYAGNI (You Ain't Gonna Need It), right.\n\n>> I assume that either Git would have to start multiple filter\n>> commands for multi-threaded operation, or the protocol would have\n>> to be extended to make persistent filter fork itself.\n> \n> I think it would be better to have Git start multiple filter commands\n> to keep the protocol as simple and error free as possible.\n\nRight. Also, I am not sure if exec+fork would be much faster than\nfork+exec (where fork is n-way fork, and n is number of threads\nthat Git command invoking filter is using).\n\n>> BTW. what would happen in your original proposal if the user had\n>> *both* filter..useProtocol and filter..smudge\n>> (and/or filter..clean) set?\n> \n> That wouldn't be an issue as \"useProtocol\" is just a boolean that\n> tells Git how to talk to \"filter..smudge\" and \"filter..clean\".\n> I need to make this more clear in the documentation.\n> \n> \n>>> 1. Git starts the filter on first usage and expects a welcome message\n>>> with protocol version number:\n>>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>>> \tGit <-- Filter: \"version 1\"\n>>\n>> I was wondering how Git would know that filter executable was started,\n>> but then I realized it was once-per-command invocation, not a daemon.\n>>\n>> I agree with Torsten that there should be a terminator after the\n>> version number.\n> \n> I agree, too :)\n\nNote that if we agree about switch to `protocol` / `protocolVersion`\nas a way to specify this protocol, it would probably need to be \"protocol 2\"\n(assuming that \"protocol 1\" is the original implementation, with one fork\nper affected file).\n\n>> Also, for future extendability this should be probably followed by\n>> possibly empty list of script capabilities, that is:\n>>\n>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>> \tGit <-- Filter: \"version 1.1\\n\"\n\nNote that \"version 1.1\" would not work with current implementation;\nit accepts only integer version numbers. Which might be a good idea,\nanyway.\n\n>> \tGit <-- Filter: \"capabilities clean smudge\\n\"\n>>\n>> Or we can add capabilities in later version...\n> \n> That is an interesting idea. My initial thought was to make the capabilities\n> of a certain version fix. If we want to add new capabilities then we would \n> bump the version. I wonder what others think about your suggestion!\n\nUsing capabilities (like git-upload-pack / git-receive-pack, that is\nsmart Git transfer protocols do) is probably slightly more difficult on\nthe Git side (assuming no capabilities negotiation), but also much more\nflexible than pure version numbers.\n\nOne possible idea for a capability is support for passing input\nand output of a filter via filesystem, like cleanToFile and smudgeFromFile\nproposal in 'jh/clean-smudge-annex' (in 'pu').\n\nFor example:\n\n \tGit <-- Filter: \"capabilities clean smudge cleanToFile smudgeFromFile\\n\"\n\n>> BTW. why not follow e.g. HTTP protocol example, and use\n>>\n>> \tGit <-- Filter: \"git-filter-protocol/1\\n\"\n> \n> I think my proposal is a bit more explicit as it states \"version\". If\n> you feel strongly about it, I could be convinced otherwise.\n\nNo, I don't feel strongly about this. I think SSH also uses a separate\n\"version\"-like line.\n\n>>> 2. Git sends the command (either \"smudge\" or \"clean\"), the filename, the\n>>> content size in bytes, and the content separated by a newline character:\n>>> \tGit --> Filter: \"smudge\\n\"\n>>\n>> Would it help (for some cases) to pass the name of filter that\n>> is being invoked?\n> \n> Interesting thought! Can you imagine a case where this would be useful?\n\nActually... no, I don't think so. I don't think there is a situation\nwhere we might want to use the same filter commands for different filters\nand have it behave differently depending on the filter name.\n\n>>> \tGit --> Filter: \"testfile.dat\\n\"\n>>\n>> Unfortunately, while sane filenames should not contain newlines[1],\n>> the unfortunate fact is that *filenames can include newlines*, and\n>> you need to be able to handle that[2]. Therefore you need either to\n>> choose a different separator (the only one that can be safely used\n>> is \"\\0\", i.e. the NUL character - but it is not something easy to\n>> handle by shell scripts), or C-quote filenames as needed, or always\n>> C-quote filenames. C-quoting at minimum should include quoting newline\n>> character, and the escape character itself.\n>>\n>> BTW. is it the basename of a file, or a full pathname?\n>>\n>> [1]: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n>> [2]: http://www.dwheeler.com/essays/filenames-in-shell.html\n> \n> Thanks for this explanation. A bash version of the protocol is not\n> trivial (I tried it but ended up using Perl). Therefore I think \"\\0\"\n> would be a good choice?\n\nThat, or use git convention of surrounding C-quoted filenames in\ndouble quotes (which means that if it begins with quote, it is C-quoted).\n\nFor example:\n\n $ git commit -m 'Initial commit'\n [master (root-commit) 266dab0] Initial commit\n 2 files changed, 2 insertions(+)\n create mode 100644 foo\n create mode 100644 \"foo \\\" \\\\ ,\"\n $ ls -1\n foo \" \\ ,\n foo\n\nI'm not sure which solution would be easier for filter writers,\nNUL termination, or C-quoting.\n\n>>> \tGit --> Filter: \"7\\n\"\n>>\n>> That's the content size in bytes written as an ASCII number.\n> \n> Correct.\n\nBut not obvious from the description / documentation. \n\n>>> \tGit --> Filter: \"CONTENT\"\n>>\n>> Can filter ignore the content size, and just read all what it was\n>> sent, that is until eof or something?\n> \n> What would that something be? Since CONTENT is binary it can contain\n> any character (even \"\\0\")...\n \nHere by \"or something\" I meant some other way of detecting that there\nis nothing more to read. But providing the size upfront (or size of\nchunk / packet in the streaming interface, if/when it gets implemented)\nis probably a better idea. Git knows it anyway, cheaply.\n\n>>> 3. The filter is expected to answer with the result content size in\n>>> bytes and the result content separated by a newline character:\n>>> \tGit <-- Filter: \"15\\n\"\n>>> \tGit <-- Filter: \"SMUDGED_CONTENT\"\n>>\n>> I wonder how hard would be to write filters for this protocol...\n> \n> Easy :-) Plus you can look at a Perl (see t/t0021) and a golang implementation\n> already (https://github.com/github/git-lfs/pull/1382).\n\nRight. Any programming language that has a way to specify \"read N bytes\"\nwould work. I think even bash would work, with 'read -N $len -r'... I think.\n\n>>> ---\n>>> Documentation/gitattributes.txt | 41 +++++++-\n>>> convert.c | 210 ++++++++++++++++++++++++++++++++++++++--\n>>> t/t0021-conversion.sh | 170 ++++++++++++++++++++++++++++++++\n>>\n>> Wouldn't it be better to name the test case something more\n>> descriptive, for example\n>>\n>> t/t0021-filter-driver-useProtocol.sh\n>>\n>> The name of test should be adjusted to final name of the feature,\n>> of course.\n> \n> I think the prefix numbers should be unique, no? And t0022 is already taken.\n\nI meant here that the \"conversion\" part of \"t/t0021-conversion.sh\" test\nfilename is not descriptive enough.\n \n>>> t/t0021/rot13.pl | 80 +++++++++++++++\n\nThis is all right, because it is in t0021 context.\n\n>>> 4 files changed, 494 insertions(+), 7 deletions(-)\n>>> create mode 100755 t/t0021/rot13.pl\n>>>\n>>> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n>>> index 8882a3e..7026d62 100644\n>>> --- a/Documentation/gitattributes.txt\n>>> +++ b/Documentation/gitattributes.txt\n>>> @@ -300,7 +300,10 @@ checkout, when the `smudge` command is specified, the command is\n>>> fed the blob object from its standard input, and its standard\n>>> output is used to update the worktree file. Similarly, the\n>>> `clean` command is used to convert the contents of worktree file\n>>> -upon checkin.\n>>> +upon checkin. By default these commands process only a single\n>>> +blob and terminate. If the setting filter..useProtocol is\n>>> +enabled then Git can process all blobs with a single filter command\n>>> +invocation (see filter protocol below).\n>>\n>> This does not tell the precedence between `smudge`, `clean` and\n>> filter..useProtocol, see above. Also, discrepancy in how\n>> config variables are referenced.\n> \n> As mentioned above \"useProtocol\" is a boolean. Therefore precedence shouldn't\n> be a problem.\n\nWhich was not obvious (but might not matter in the end).\n\n> What do you mean by \"discrepancy in how config variables are \n> referenced\"?\n \nWhat I meant here that filter..smudge and filter..clean\nwere referenced as \"`smudge` command\" and \"`clean` command\" in the paragraph\nyou modified.\n\nPerhaps filter..useProtocol is all right (I have not looked further),\nbut it should be formatted as `filter..useProtocol` IMVHO.\n \n[...]\n>>> diff --git a/convert.c b/convert.c\n>>> index 522e2c5..91ce86f 100644\n>>> --- a/convert.c\n>>> +++ b/convert.c\n>>> @@ -481,12 +481,188 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n>>> \treturn ret;\n>>> }\n>>>\n>>> +static int cmd_process_map_init = 0;\n>>> +static struct hashmap cmd_process_map;\n>>> +\n>>> +struct cmd2process {\n>>> +\tstruct hashmap_entry ent; /* must be the first member! */\n>>> +\tconst char *cmd;\n>>> +\tlong protocol;\n>>> +\tstruct child_process process;\n>>> +};\n>> [...]\n>>> +static struct cmd2process *find_protocol_filter_entry(const char *cmd)\n>>> +{\n>>> +\tstruct cmd2process k;\n>>> +\thashmap_entry_init(&k, strhash(cmd));\n>>> +\tk.cmd = cmd;\n>>> +\treturn hashmap_get(&cmd_process_map, &k, NULL);\n>>\n>> Should we use global variable cmd_process_map, or pass it as parameter?\n>> The same question apply for other procedures and functions.\n>>\n>> Note that I am not saying that it is a bad thing to use global\n>> variable here.\n> \n> Passing it would be nicer as this would make at least a few functions \"pure\".\n> I will change that!\n\nYou can always provide convenience functions that use global variable.\nThat's what Git code does with the_index, if I remember it correctly.\n\n[...]\n>>> +\n>>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>>> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n>>> +\tsigchain_pop(SIGPIPE);\n>>> +\n>>> +\tstrbuf_stripspace(&nbuf, 0);\n>>> +\tstring_list_split_in_place(&split, nbuf.buf, ' ', 2);\n>>> +\tret &= split.nr > 1;\n>>> +\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n>>> +\tif (ret) {\n>>> +\t\tentry->protocol = strtol(split.items[1].string, NULL, 10);\n>>\n>> This does not handle at least some errors in version number parsing,\n>> for example junk after version number. Don't we have some helper\n>> functions for this?\n> \n> I am not sure. I haven't found one.\n\nHmmm... I remember there were some patches about this, but I don't know\nif they were accepted. We have strtol_i() in git-compat-util.h. \n\nAnd you can always check where the parsing ended (by not passing NULL,\nof course).\n\n[...]\n>>> +\t\tswitch (entry->protocol) {\n>>> +\t\t\tcase 1:\n>>> +\t\t\t\tbreak;\n>>> +\t\t\tdefault:\n>>> +\t\t\t\tret = 0;\n>>> +\t\t\t\terror(\"unsupported protocol version %s for external persistent filter '%s'\",\n>>> +\t\t\t\t\tnbuf.buf, cmd);\n>>> +\t\t}\n>>> +\t}\n>>> +\tstring_list_clear(&split, 0);\n>>> +\tstrbuf_release(&nbuf);\n>>> +\n>>> +\tif (!ret) {\n>>> +\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n>>> +\t\treturn NULL;\n>>> +\t}\n>>\n>> Do we handle persistent filter command being killed before it finishes?\n>> Or exiting with error? I don't know this Git API...\n> \n> If the \"apply_filter\" function fails then Git would proceed and just not\n> filter the content. If you define the \"required\" flag for the filter then\n> Git would error in that case.\n\nAh, right. \n\n[...]\n>>> diff --git a/t/t0021/rot13.pl b/t/t0021/rot13.pl\n>>\n>> That's bit more than rot13... but it might be O.K. for a filename here.\n> \n> \"rot13-$FEATURE_NAME.pl\" ?\n\nAs I said, rot13.pl is all right; if change, then perhaps to rot13-filter.pl \n\n[...]\n>>> + my $input;\n>>> + {\n>>> + binmode(STDIN);\n>>> + my $bytes_read = 0;\n>>> + $bytes_read = read STDIN, $input, $filelen;\n>>> + if ( $bytes_read != $filelen ) {\n>>> + die \"not enough to read\";\n>>\n>> I know it's only a test script (well, a part of one), but we would probably\n>> want to have more information in the case of a real filter.\n> \n> True. Do you think there is anything to change in the script, though?\n\nNo, I don't think so. It is enough that the test script would crash\nif fed incorrect data from Git. Better error messages would be nice,\nbut are not necessary.\n\n + die \"not enough to read: expected $filelen, got $bytes_read\";\n\nI have noticed that some 'die' have \"\\n\" at the end, and some do\nnot. If I remember correctly it is for supressing error message from\nPerl, with filename and line number, isn't it? Anyway, we probably\nwant to be consistent.\n\n\n>>> + }\n>>> + }\n>>\n>> What happens if $filelen is zero, or negative? Ah, I see that $output\n>> would be undef... which is bad, I think.\n> \n> Is this something we need to consider in the test script?\n\nI think the test script should die if it gets incorrect content length\n(e.g. negative), so that it catches bug on the Git side. It should\nwork for zero-length files, even if we don't test it -- we can in the\nfuture.\n\n[...]\n>>> + print STDOUT \"fail\";\n>>\n>> This is not defined in the protocol description! Unless anything that\n>> does not conform to the specification would work here, but at least it\n>> is a recommended practice to be described in the documentation, don't\n>> you think?\n>>\n>> What would happen in $output_len is 4?\n> \n> Then it would work :D\n> I understand your point. However, this is not a reference implementation.\n> It is a test script that is supposed to trigger bad behavior which we can test. \n> Therefore, I would argue that such a return value is OK. I will document it in \n> the header, though. \n\nWhy print \"fail\", and not die?\n\nThe problem is do the protocol need to have some way of communicating\nerrors from the filter to Git? Perhaps using stderr would be enough\n(but then Git would need to drain it, I think... unless it is not\nredirected), perhaps some command is needed?\n\nFor example, instead of:\n\n\tGit <-- Filter: \"15\\n\"\n\tGit <-- Filter: \"SMUDGED_CONTENT\"\n\nperhaps filter should return\n\n\tGit <-- Filter: \"error\\n\"\n\tGit <-- Filter: \"ONE_LINE_OF_ERROR_DESCRIPTION\\n\"\n\non error? Or if printing expected output length upfront is easier,\nuse a signal (but that is supposedly not that reliable as message\npassing mechanism)?\n\nIt might be the case that some files return errors, but some do not.\n\nBTW. do we test the case where filter fails, or returns wrong output?\n\n> Thanks a lot for your extensive review,\n> Lars--\n\nYou are welcome.\n-- \nJakub Narębski\n\n"},{"id":"292076","messageId":"5795337A.1040701@gmail.com","threadId":"42904","inReplyTo":"579521A1.4070501@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-24T21:30:34Z","receivedAt":"2016-07-24T21:30:51Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-24 o 22:14, Jakub Narębski pisze:\n> W dniu 2016-07-24 o 20:36, Lars Schneider pisze:\n\n>> I agree that the name is not ideal. \"UseProtocol\" as it is would be a boolean. \n>> I thought about \"persistent\" but this name wouldn't convey the scope of the \n>> persistency (\"persistent for one Git operation\" vs. \"persistent for many Git \n>> operations\"). What do you think about the protocol as int version idea\n>> described in $gmane/300155 ?\n>\n> You mean the `protocol` as a config variable name (fully name being\n> `filter..protocol`), being integer-valued, isn't it? Wouldn't\n> `protocolVersion` be a more explicit?\n\nJust throwing out further ideas:\n\nPerhaps make `persistent` string-valued variable, with the only value\nsupported for now, namely \"per-process\" / \"operation\"?\n\nPerhaps require for `pidfile` to be present for it to be daemon,\nthat is persist for possibly many Git operations. Or allow \"daemon\"\nor \"server\" value for `persistent`, then?\n\n-- \nJakub Narębski\n\n"},{"id":"292077","messageId":"194ea810-76ff-f32c-0f8a-57e8e60b65f5@ramsayjones.plus.com","threadId":"42904","inReplyTo":"1A0C148F-C7C3-4FAF-BAEE-58B11A2324FF@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2016-07-24T22:36:21Z","receivedAt":"2016-07-24T22:37:04Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 24/07/16 18:16, Lars Schneider wrote:\n> \n> On 23 Jul 2016, at 01:19, Ramsay Jones wrote:\n> \n>> On 22/07/16 16:49, larsxschneider@gmail.com wrote:\n>>> From: Lars Schneider \n>>>\n>>> Git's clean/smudge mechanism invokes an external filter process for every\n>>> single blob that is affected by a filter. If Git filters a lot of blobs\n>>> then the startup time of the external filter processes can become a\n>>> significant part of the overall Git execution time.\n>>>\n>>> This patch adds the filter..useProtocol option which, if enabled,\n>>> keeps the external filter process running and processes all blobs with\n>>> the following protocol over stdin/stdout.\n>>>\n>>> 1. Git starts the filter on first usage and expects a welcome message\n>>> with protocol version number:\n>>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>>> \tGit <-- Filter: \"version 1\"\n>>\n>> Hmm, I was a bit surprised to see a 'filter' talk first (but so long as the\n>> interaction is fully defined, I guess it doesn't matter).\n>>\n>> [If you wanted to check for a version, you could add a \"version\" command\n>> instead, just like \"clean\" and \"smudge\".]\n> \n> It was a conscious decision to have the `filter` talk first. My reasoning was:\n> \n> (1) I want a reliable way to distinguish the existing filter protocol (\"single-shot \n> invocation\") from the new one (\"long running\"). I don't think there would be a\n> situation where the existing protocol would talk first. Therefore the users would\n> not accidentally mix them with a possibly half working, undetermined, outcome.\n\nIf an 'single-shot' filter were incorrectly configured, instead of a new one, then\nthe interaction could last a little while - since it would result in deadlock! ;-)\n\n[If Git talks first instead, configuring a 'single-shot' filter _may_ still result\nin a deadlock - depending on pipe size, etc.]\n\n> \n> (2) In the future we could extend the pipe protocol (see $gmane/297994, it's very\n> interesting). A filter could check Git's version and then pick the most appropriate\n> filter protocol on startup.\n> \n> \n>> [...]\n>>> +static struct cmd2process *start_protocol_filter(const char *cmd)\n>>> +{\n>>> +\tint ret = 1;\n>>> +\tstruct cmd2process *entry = NULL;\n>>> +\tstruct child_process *process = NULL;\n>>> +\tstruct strbuf nbuf = STRBUF_INIT;\n>>> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n>>> +\tconst char *argv[] = { NULL, NULL };\n>>> +\tconst char *header = \"git-filter-protocol\\nversion\";\n>>> +\n>>> +\tentry = xmalloc(sizeof(*entry));\n>>> +\thashmap_entry_init(entry, strhash(cmd));\n>>> +\tentry->cmd = cmd;\n>>> +\tprocess = &entry->process;\n>>> +\n>>> +\tchild_process_init(process);\n>>> +\targv[0] = cmd;\n>>> +\tprocess->argv = argv;\n>>> +\tprocess->use_shell = 1;\n>>> +\tprocess->in = -1;\n>>> +\tprocess->out = -1;\n>>> +\n>>> +\tif (start_command(process)) {\n>>> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n>>> +\t\treturn NULL;\n>>> +\t}\n>>> +\tstrbuf_reset(&nbuf);\n>>> +\n>>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>>> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n>>\n>> Hmm, how much will be read into nbuf by this single call?\n>> Since strbuf_read_once() makes a single call to xread(), with\n>> a len argument that will probably be 8192, you can not really\n>> tell how much it will read, in general. (xread() does not\n>> guarantee how many bytes it will read.)\n>>\n>> In particular, it could be less than strlen(header).\n> \n> As mentioned to Torsten in $gmane/300156, I will add a newline\n> and then read until I find the second newline. That should solve\n> the problem, right?\n> \n> (You wrote in $gmane/300119 that I should ignore your email but\n> I think you have a valid point here ;-)\n\nHeh, as I said, it was late and I was trying to do several things\nat once. (I am updating 3 installations of Linux Mint 17.3 to Linux\nMint 18 - I decided to do a complete re-install, since I needed to\nchange partition sizes anyway. I have only just got email back up ...)\n\nI stopped commenting on the patch early but, after sending the first\nemail, I decided to scan the rest of your patch before going to bed\nand noticed something which would invalidate my comments ...\n\n> \n> \n>>> [...]\n>>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>>> +\tswitch (entry->protocol) {\n>>> +\t\tcase 1:\n>>> +\t\t\tif (fd >= 0 && !src) {\n>>> +\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n>>> +\t\t\t\tlen = fileStat.st_size;\n>>> +\t\t\t}\n>>> +\t\t\tstrbuf_reset(&nbuf);\n>>> +\t\t\tstrbuf_addf(&nbuf, \"%s\\n%s\\n%zu\\n\", filter_type, path, len);\n>>> +\t\t\tret &= write_str_in_full(process->in, nbuf.buf) > 1;\n>>\n>> why not write_in_full(process->in, nbuf.buf, nbuf.len) ?\n> OK, this would save a \"strlen\" call. Do you think such a function could be of general\n> use? If yes, then I would add:\n> \n> static inline ssize_t write_strbuf_in_full(int fd, struct strbuf *str)\n> {\n> \treturn write_in_full(fd, str->buf, str->len);\n> }\n\n[I don't have strong feelings either way (but I suspect it's not worth it).]\n\n> \n> \n>>> +\t\t\tif (len > 0) {\n>>> +\t\t\t\tif (src)\n>>> +\t\t\t\t\tret &= write_in_full(process->in, src, len) == len;\n>>> +\t\t\t\telse if (fd >= 0)\n>>> +\t\t\t\t\tret &= copy_fd(fd, process->in) == 0;\n>>> +\t\t\t\telse\n>>> +\t\t\t\t\tret &= 0;\n>>> +\t\t\t}\n>>> +\n>>> +\t\t\tstrbuf_reset(&nbuf);\n>>> +\t\t\twhile (xread(process->out, &c, 1) == 1 && c != '\\n')\n>>> +\t\t\t\tstrbuf_addchars(&nbuf, c, 1);\n>>> +\t\t\tnbuf_len = (size_t)strtol(nbuf.buf, &strtol_end, 10);\n>>> +\t\t\tret &= (strtol_end != nbuf.buf && errno != ERANGE);\n>>> +\t\t\tstrbuf_reset(&nbuf);\n>>> +\t\t\tif (nbuf_len > 0)\n>>> +\t\t\t\tret &= strbuf_read_once(&nbuf, process->out, nbuf_len) == nbuf_len;\n>>\n>> Again, how many bytes will be read?\n>> Note, that in the default configuration, a _maximum_ of\n>> MAX_IO_SIZE (8MB or SSIZE_MAX, whichever is smaller) bytes\n>> will be read.\n\n... In particular, your 2GB test case should not have worked, so\nI assumed that I had missed a loop somewhere ...\n\n> Would something like this be more appropriate?\n> \n> strbuf_reset(&nbuf);\n> if (nbuf_len > 0) {\n> strbuf_grow(&nbuf, nbuf_len);\n> ret &= read_in_full(process->out, nbuf.buf, nbuf_len) == nbuf_len;\n> }\n\n... and this looks better. [Note: this comment would apply equally to the\nversion message.]\n\n[Hmm, now can I remember which packages I need to install ...]\n\nATB,\nRamsay Jones\n"},{"id":"292078","messageId":"57954DA3.9090006@gmail.com","threadId":"42904","inReplyTo":"194ea810-76ff-f32c-0f8a-57e8e60b65f5@ramsayjones.plus.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-24T23:22:11Z","receivedAt":"2016-07-24T23:22:35Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-25 o 00:36, Ramsay Jones pisze:\n> On 24/07/16 18:16, Lars Schneider wrote:\n>> On 23 Jul 2016, at 01:19, Ramsay Jones wrote:\n>>> On 22/07/16 16:49, larsxschneider@gmail.com wrote:\n[...]\n>>>> This patch adds the filter..useProtocol option which, if enabled,\n>>>> keeps the external filter process running and processes all blobs with\n>>>> the following protocol over stdin/stdout.\n>>>>\n>>>> 1. Git starts the filter on first usage and expects a welcome message\n>>>> with protocol version number:\n>>>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>>>> \tGit <-- Filter: \"version 1\"\n>>>\n>>> Hmm, I was a bit surprised to see a 'filter' talk first (but so long as the\n>>> interaction is fully defined, I guess it doesn't matter).\n>>\n>> It was a conscious decision to have the `filter` talk first. My reasoning was:\n>>\n>> (1) I want a reliable way to distinguish the existing filter protocol (\"single-shot \n>> invocation\") from the new one (\"long running\"). I don't think there would be a\n>> situation where the existing protocol would talk first. Therefore the users would\n>> not accidentally mix them with a possibly half working, undetermined, outcome.\n> \n> If an 'single-shot' filter were incorrectly configured, instead of a new one, then\n> the interaction could last a little while - since it would result in deadlock! ;-)\n> \n> [If Git talks first instead, configuring a 'single-shot' filter _may_ still result\n> in a deadlock - depending on pipe size, etc.]\n\nWould it be possible to do an equivalent of sending empty file to the filter?\nIf it is misconfigured old-style script, it would exit after possibly empty\noutput; if not, we would start new-style interaction.\n \nThis should be, if we agree that detecting misconfigured filters is a good\nthing, tested.\n\n>>\n>> (2) In the future we could extend the pipe protocol (see $gmane/297994, it's very\n>> interesting). A filter could check Git's version and then pick the most appropriate\n>> filter protocol on startup.\n\n"},{"id":"292082","messageId":"20160725072745.GB11634@starla","threadId":"42904","inReplyTo":"121F8453-3A12-47AE-AE10-021EADC4D995@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-25T07:27:45Z","receivedAt":"2016-07-25T07:28:06Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Lars Schneider wrote:\n> On 23 Jul 2016, at 10:14, Eric Wong wrote:\n> > larsxschneider@gmail.com wrote:\n> >> +static struct cmd2process *start_protocol_filter(const char *cmd)\n> >> +{\n> >> +\tint ret = 1;\n> >> +\tstruct cmd2process *entry = NULL;\n> >> +\tstruct child_process *process = NULL;\n> > \n> > These are unconditionally set below, so initializing to NULL\n> > may hide future bugs.\n> \n> OK. I thought it is generally a good thing to initialize a pointer with \n> NULL. Can you explain to me how this might hide future bugs?\n> I will remove the initialization.\n\nCompilers complain about uninitialized variables. Blindly\nsetting them to NULL can allow them to be dereferenced;\ntriggering segfaults; especially if it's passed to a different\ncompilation unit the compiler can't see.\n\n> >> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n> >> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n> >> +\t\t\t\t\t\tconst char *filter_type)\n> >> +{\n\n\n\n> >> +\t\t\tif (fd >= 0 && !src) {\n> >> +\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n> >> +\t\t\t\tlen = fileStat.st_size;\n> > \n> > There's a truncation bug when sizeof(size_t) < sizeof(off_t)\n> \n> OK. What would you suggest to do in that case? Should we just let the\n> filter fail? Is there anything else we could do?\n\nAnything which refers to something on disk (or will eventually\nstored there, such as blobs) should evolve towards off_t rather\nthan size_t. We just discovered a bunch of 32-bit truncation\nbugs the other week:\n\nhttps://public-inbox.org/git/1466807902.28869.8.camel@gmail.com/\n\nIf the protocol/ABI is frozen, it should probably fail;\nand a 64-bit-off_t version for 32-bit systems should be defined.\n"},{"id":"292097","messageId":"CACsJy8D_21pY5FRs4VCpDUuUJ56OZT8DZ1ZKMiwWKeeSMiLSdA@mail.gmail.com","threadId":"42904","inReplyTo":"121F8453-3A12-47AE-AE10-021EADC4D995@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-25T15:48:55Z","receivedAt":"2016-07-25T15:49:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jul 24, 2016 at 9:11 PM, Lars Schneider\n wrote:\n>\n> On 23 Jul 2016, at 10:14, Eric Wong wrote:\n>\n>> larsxschneider@gmail.com wrote:\n>>> Please note that the protocol filters do not support stream processing\n>>> with this implemenatation because the filter needs to know the length of\n>>> the result in advance. A protocol version 2 could address this in a\n>>> future patch.\n>>\n>> Would it be prudent to reuse pkt-line for this?\n>\n> Peff suggested that, too, in $gmane/299902.\n\nAnd I was about to suggest the same too, until I saw his patch then\nstopped. Having a common way to split a byte stream to a packet stream\ncould be a good thing.\n\n> However, this would make the protocol a bit more complicated\n\nFor high level scripting languages, pkt-line is dead simple. If your\nscripts are in sh then it could get a bit ugly, but I'm thinking of a\nsmall utility to make shell scripting pkt-line easier anyway (it goes\nback to the idea of rewriting index-helper as a script, which I might\ndo).\n\n> and it wouldn't buy us anything for Git\n> large file processing filters (my main motivation for this patch) as these\n> filters can't leverage streaming anyways.\n\nThis is a good point. How are you planning to do it? Unless streaming\nis done entirely in kernel (sendfie() and friends, which is not all\npositive), I think you can still stream and wrap/unwrap pkt-line just\nfine.\n-- \nDuy\n"},{"id":"292133","messageId":"1EF550A7-F6D4-4BFE-A4A7-D375268215DE@gmail.com","threadId":"42904","inReplyTo":"579521A1.4070501@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-25T20:09:10Z","receivedAt":"2016-07-25T20:09:25Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 24 Jul 2016, at 22:14, Jakub Narębski wrote:\n\n> W dniu 2016-07-24 o 20:36, Lars Schneider pisze:\n>> On 23 Jul 2016, at 02:11, Jakub Narębski wrote:\n>>> W dniu 2016-07-22 o 17:49, larsxschneider@gmail.com pisze:\n>>>> From: Lars Schneider \n>>> \n>>> Nb. this line is only needed if you want author name and/or date\n>>> different from the email sender, or if you have sender line misconfigured\n>>> (e.g. lacking the human readable name).\n>> \n>> I use \"git format-patch\" to generate these emails:\n>> \n>> git format-patch --cover-letter --subject-prefix=\"PATCH ...\" -M $BASE -o $OUTPUT\n>> \n>> How would I disable this line? (I already checked the man page to no avail).\n> \n> If you are using `git send-email` or equivalent, I think it is\n> stripped automatically if it is not needed (in you case it was,\n> because Sender was lacking human readable name... at least I think\n> it was because of what my email reader inserted as reply line).\n> If you are using an ordinary email client, you need to remove it\n> yourself, if needed.\n\nWeird. I am sending the patches with this command:\n\ngit send-email mystuff/* --to=git@vger.kernel.org --cc=...\n\nMaybe I need to try the \"--suppress-from\" switch?!\n\n\n>> Plus, what does \"Nb\" stand for? :-)\n> \n> Nb. (or N.b.) stands for \"nota bene\", which I meant do denote\n> as a note on a certain side aspect; I'll switch to \"note\", or\n> \"BTW\" / \"by the way\".\n\nOK, thanks for the explanation :-)\n\n\n>>>> Git's clean/smudge mechanism invokes an external filter process for every\n>>>> single blob that is affected by a filter. If Git filters a lot of blobs\n>>>> then the startup time of the external filter processes can become a\n>>>> significant part of the overall Git execution time.\n>>> \n>>> Do I understand it correctly (from the commit description) that with\n>>> this new option you start one filter process for the whole life of\n>>> the Git command (e.g. `git add .`), which may perform more than one\n>>> cleanup, but do not create a daemon or daemon-like process which\n>>> would live for several commands invocations?\n>> \n>> Correct!\n> \n> It would be nice to make it more obvious.\n\nOK, I will try in v2.\n\n> \n>>>> This patch adds the filter..useProtocol option which, if enabled,\n>>>> keeps the external filter process running and processes all blobs with\n>>>> the following protocol over stdin/stdout.\n>>> \n>>> I agree with Junio that the name \"useProtocol\" is bad, and not quite\n>>> right. Perhaps \"persistent\" would be better? Also, what is the value\n>>> of `filter..useProtocol`: boolean? or a script name?\n> \n> As you see I was not sure if `useProtocol` was boolean or a script name,\n> which means that it should be stated more explicitly. Of course this\n> would end to not matter if the way new protocol is used were changed.\n> \n>> I agree that the name is not ideal. \"UseProtocol\" as it is would be a boolean. \n>> I thought about \"persistent\" but this name wouldn't convey the scope of the \n>> persistency (\"persistent for one Git operation\" vs. \"persistent for many Git \n>> operations\"). What do you think about the protocol as int version idea\n>> described in $gmane/300155 ?\n> \n> You mean the `protocol` as a config variable name (fully name being\n> `filter..protocol`), being integer-valued, isn't it? Wouldn't\n> `protocolVersion` be a more explicit?\n\nYes, but based on your other feedback I plan to use this variable differently\nanyways.\n\n\n>>> I also agree that we might wat to be able to keep clean and smudge\n>>> filters separate, but be able to run a single program if they are\n>>> both the same. I think there is a special case for filter unset,\n>>> and/or filter being \"cat\" -- we would want to keep that.\n>> \n>> Since 1a8630d there is a more efficient way to unset a filter ;-)\n>> Can you think of other cases where the separation would be useful?\n> \n> I can't think of any, but it doesn't mean that it does not exist.\n> It also does not mean that you need to consider situation that may\n> not happen. Covering one-way filters, like \"indent\" filter for `clean`,\n> should be enough... they do work with your proposal, don't they?\n\nThis should work right now but it would be a bit inefficient (the filter\nwould just pass the data unchanged through the smudge command). I plan to\nadd a \"capabilities\" flag to the protocol. Then you can define only\nthe \"clean\" capability and nothing or the current filter mechanism \nwould happen for smudge (I will make a test case to demonstrate that\nbehavior in v2).\n\n\n>>> My proposal is to use `filter..persistent` as an addition\n>>> to 'clean' and 'smudge' variables, with the following possible\n>>> values:\n>>> \n>>> * none (the default)\n>>> * clean\n>>> * smudge\n>>> * both\n>> \n>> That could work. However, I am not convinced, yet, that separate\n>> filters are an actual use case.\n> \n> YAGNI (You Ain't Gonna Need It), right.\n\nThat will work in v2.\n\n\n>>> I assume that either Git would have to start multiple filter\n>>> commands for multi-threaded operation, or the protocol would have\n>>> to be extended to make persistent filter fork itself.\n>> \n>> I think it would be better to have Git start multiple filter commands\n>> to keep the protocol as simple and error free as possible.\n> \n> Right. Also, I am not sure if exec+fork would be much faster than\n> fork+exec (where fork is n-way fork, and n is number of threads\n> that Git command invoking filter is using).\n> \n>>> BTW. what would happen in your original proposal if the user had\n>>> *both* filter..useProtocol and filter..smudge\n>>> (and/or filter..clean) set?\n>> \n>> That wouldn't be an issue as \"useProtocol\" is just a boolean that\n>> tells Git how to talk to \"filter..smudge\" and \"filter..clean\".\n>> I need to make this more clear in the documentation.\n>> \n>> \n>>>> 1. Git starts the filter on first usage and expects a welcome message\n>>>> with protocol version number:\n>>>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>>>> \tGit <-- Filter: \"version 1\"\n>>> \n>>> I was wondering how Git would know that filter executable was started,\n>>> but then I realized it was once-per-command invocation, not a daemon.\n>>> \n>>> I agree with Torsten that there should be a terminator after the\n>>> version number.\n>> \n>> I agree, too :)\n> \n> Note that if we agree about switch to `protocol` / `protocolVersion`\n> as a way to specify this protocol, it would probably need to be \"protocol 2\"\n> (assuming that \"protocol 1\" is the original implementation, with one fork\n> per affected file).\n\nAgreed.\n\n\n>>> Also, for future extendability this should be probably followed by\n>>> possibly empty list of script capabilities, that is:\n>>> \n>>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>>> \tGit <-- Filter: \"version 1.1\\n\"\n> \n> Note that \"version 1.1\" would not work with current implementation;\n> it accepts only integer version numbers. Which might be a good idea,\n> anyway.\n\nAgreed.\n\n\n>>> \tGit <-- Filter: \"capabilities clean smudge\\n\"\n>>> \n>>> Or we can add capabilities in later version...\n>> \n>> That is an interesting idea. My initial thought was to make the capabilities\n>> of a certain version fix. If we want to add new capabilities then we would \n>> bump the version. I wonder what others think about your suggestion!\n> \n> Using capabilities (like git-upload-pack / git-receive-pack, that is\n> smart Git transfer protocols do) is probably slightly more difficult on\n> the Git side (assuming no capabilities negotiation), but also much more\n> flexible than pure version numbers.\n> \n> One possible idea for a capability is support for passing input\n> and output of a filter via filesystem, like cleanToFile and smudgeFromFile\n> proposal in 'jh/clean-smudge-annex' (in 'pu').\n> \n> For example:\n> \n> \tGit <-- Filter: \"capabilities clean smudge cleanToFile smudgeFromFile\\n\"\n\nYes, I like that very much. As stated above, I will add that in v2.\n\n\n>>> BTW. why not follow e.g. HTTP protocol example, and use\n>>> \n>>> \tGit <-- Filter: \"git-filter-protocol/1\\n\"\n>> \n>> I think my proposal is a bit more explicit as it states \"version\". If\n>> you feel strongly about it, I could be convinced otherwise.\n> \n> No, I don't feel strongly about this. I think SSH also uses a separate\n> \"version\"-like line.\n> \n>>>> 2. Git sends the command (either \"smudge\" or \"clean\"), the filename, the\n>>>> content size in bytes, and the content separated by a newline character:\n>>>> \tGit --> Filter: \"smudge\\n\"\n>>> \n>>> Would it help (for some cases) to pass the name of filter that\n>>> is being invoked?\n>> \n>> Interesting thought! Can you imagine a case where this would be useful?\n> \n> Actually... no, I don't think so. I don't think there is a situation\n> where we might want to use the same filter commands for different filters\n> and have it behave differently depending on the filter name.\n\nOK\n\n\n>>>> \tGit --> Filter: \"testfile.dat\\n\"\n>>> \n>>> Unfortunately, while sane filenames should not contain newlines[1],\n>>> the unfortunate fact is that *filenames can include newlines*, and\n>>> you need to be able to handle that[2]. Therefore you need either to\n>>> choose a different separator (the only one that can be safely used\n>>> is \"\\0\", i.e. the NUL character - but it is not something easy to\n>>> handle by shell scripts), or C-quote filenames as needed, or always\n>>> C-quote filenames. C-quoting at minimum should include quoting newline\n>>> character, and the escape character itself.\n>>> \n>>> BTW. is it the basename of a file, or a full pathname?\n>>> \n>>> [1]: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n>>> [2]: http://www.dwheeler.com/essays/filenames-in-shell.html\n>> \n>> Thanks for this explanation. A bash version of the protocol is not\n>> trivial (I tried it but ended up using Perl). Therefore I think \"\\0\"\n>> would be a good choice?\n> \n> That, or use git convention of surrounding C-quoted filenames in\n> double quotes (which means that if it begins with quote, it is C-quoted).\n> \n> For example:\n> \n> $ git commit -m 'Initial commit'\n> [master (root-commit) 266dab0] Initial commit\n> 2 files changed, 2 insertions(+)\n> create mode 100644 foo\n> create mode 100644 \"foo \\\" \\\\ ,\"\n> $ ls -1\n> foo \" \\ ,\n> foo\n> \n> I'm not sure which solution would be easier for filter writers,\n> NUL termination, or C-quoting.\n\nUnless someone has a convincing argument for one solution or the other\nI will go with the \\0 termination as it seems easier.\n\n\n>>>> \tGit --> Filter: \"7\\n\"\n>>> \n>>> That's the content size in bytes written as an ASCII number.\n>> \n>> Correct.\n> \n> But not obvious from the description / documentation.\n\nI will improve that in v2. Should I add the info that it is base10 or\nwould you consider that a given?\n\n\n>>>> \tGit --> Filter: \"CONTENT\"\n>>> \n>>> Can filter ignore the content size, and just read all what it was\n>>> sent, that is until eof or something?\n>> \n>> What would that something be? Since CONTENT is binary it can contain\n>> any character (even \"\\0\")...\n> \n> Here by \"or something\" I meant some other way of detecting that there\n> is nothing more to read. But providing the size upfront (or size of\n> chunk / packet in the streaming interface, if/when it gets implemented)\n> is probably a better idea. Git knows it anyway, cheaply.\n> \n>>>> 3. The filter is expected to answer with the result content size in\n>>>> bytes and the result content separated by a newline character:\n>>>> \tGit <-- Filter: \"15\\n\"\n>>>> \tGit <-- Filter: \"SMUDGED_CONTENT\"\n>>> \n>>> I wonder how hard would be to write filters for this protocol...\n>> \n>> Easy :-) Plus you can look at a Perl (see t/t0021) and a golang implementation\n>> already (https://github.com/github/git-lfs/pull/1382).\n> \n> Right. Any programming language that has a way to specify \"read N bytes\"\n> would work. I think even bash would work, with 'read -N $len -r'... I think.\n> \n>>>> ---\n>>>> Documentation/gitattributes.txt | 41 +++++++-\n>>>> convert.c | 210 ++++++++++++++++++++++++++++++++++++++--\n>>>> t/t0021-conversion.sh | 170 ++++++++++++++++++++++++++++++++\n>>> \n>>> Wouldn't it be better to name the test case something more\n>>> descriptive, for example\n>>> \n>>> t/t0021-filter-driver-useProtocol.sh\n>>> \n>>> The name of test should be adjusted to final name of the feature,\n>>> of course.\n>> \n>> I think the prefix numbers should be unique, no? And t0022 is already taken.\n> \n> I meant here that the \"conversion\" part of \"t/t0021-conversion.sh\" test\n> filename is not descriptive enough.\n\nAh, I see. You suggest to rename the test case? Would that be OK with the Git\ncommunity?\n\n\n>>>> t/t0021/rot13.pl | 80 +++++++++++++++\n> \n> This is all right, because it is in t0021 context.\n> \n>>>> 4 files changed, 494 insertions(+), 7 deletions(-)\n>>>> create mode 100755 t/t0021/rot13.pl\n>>>> \n>>>> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n>>>> index 8882a3e..7026d62 100644\n>>>> --- a/Documentation/gitattributes.txt\n>>>> +++ b/Documentation/gitattributes.txt\n>>>> @@ -300,7 +300,10 @@ checkout, when the `smudge` command is specified, the command is\n>>>> fed the blob object from its standard input, and its standard\n>>>> output is used to update the worktree file. Similarly, the\n>>>> `clean` command is used to convert the contents of worktree file\n>>>> -upon checkin.\n>>>> +upon checkin. By default these commands process only a single\n>>>> +blob and terminate. If the setting filter..useProtocol is\n>>>> +enabled then Git can process all blobs with a single filter command\n>>>> +invocation (see filter protocol below).\n>>> \n>>> This does not tell the precedence between `smudge`, `clean` and\n>>> filter..useProtocol, see above. Also, discrepancy in how\n>>> config variables are referenced.\n>> \n>> As mentioned above \"useProtocol\" is a boolean. Therefore precedence shouldn't\n>> be a problem.\n> \n> Which was not obvious (but might not matter in the end).\n> \n>> What do you mean by \"discrepancy in how config variables are \n>> referenced\"?\n> \n> What I meant here that filter..smudge and filter..clean\n> were referenced as \"`smudge` command\" and \"`clean` command\" in the paragraph\n> you modified.\n> \n> Perhaps filter..useProtocol is all right (I have not looked further),\n> but it should be formatted as `filter..useProtocol` IMVHO.\n\nInitially I thought so, too. But \"filter..required\", which is already\nmentioned in gitattributes.txt, does not use this style. Should I change that, too,\nor use the existing style?\n\n\n> [...]\n>>>> diff --git a/convert.c b/convert.c\n>>>> index 522e2c5..91ce86f 100644\n>>>> --- a/convert.c\n>>>> +++ b/convert.c\n>>>> @@ -481,12 +481,188 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n>>>> \treturn ret;\n>>>> }\n>>>> \n>>>> +static int cmd_process_map_init = 0;\n>>>> +static struct hashmap cmd_process_map;\n>>>> +\n>>>> +struct cmd2process {\n>>>> +\tstruct hashmap_entry ent; /* must be the first member! */\n>>>> +\tconst char *cmd;\n>>>> +\tlong protocol;\n>>>> +\tstruct child_process process;\n>>>> +};\n>>> [...]\n>>>> +static struct cmd2process *find_protocol_filter_entry(const char *cmd)\n>>>> +{\n>>>> +\tstruct cmd2process k;\n>>>> +\thashmap_entry_init(&k, strhash(cmd));\n>>>> +\tk.cmd = cmd;\n>>>> +\treturn hashmap_get(&cmd_process_map, &k, NULL);\n>>> \n>>> Should we use global variable cmd_process_map, or pass it as parameter?\n>>> The same question apply for other procedures and functions.\n>>> \n>>> Note that I am not saying that it is a bad thing to use global\n>>> variable here.\n>> \n>> Passing it would be nicer as this would make at least a few functions \"pure\".\n>> I will change that!\n> \n> You can always provide convenience functions that use global variable.\n> That's what Git code does with the_index, if I remember it correctly.\n> \n> [...]\n>>>> +\n>>>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>>>> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n>>>> +\tsigchain_pop(SIGPIPE);\n>>>> +\n>>>> +\tstrbuf_stripspace(&nbuf, 0);\n>>>> +\tstring_list_split_in_place(&split, nbuf.buf, ' ', 2);\n>>>> +\tret &= split.nr > 1;\n>>>> +\tret &= strncmp(header, split.items[0].string, strlen(header)) == 0;\n>>>> +\tif (ret) {\n>>>> +\t\tentry->protocol = strtol(split.items[1].string, NULL, 10);\n>>> \n>>> This does not handle at least some errors in version number parsing,\n>>> for example junk after version number. Don't we have some helper\n>>> functions for this?\n>> \n>> I am not sure. I haven't found one.\n> \n> Hmmm... I remember there were some patches about this, but I don't know\n> if they were accepted. We have strtol_i() in git-compat-util.h. \n> \n> And you can always check where the parsing ended (by not passing NULL,\n> of course).\n\nOK, try to make this nicer in v2.\n\n\n> [...]\n>>>> +\t\tswitch (entry->protocol) {\n>>>> +\t\t\tcase 1:\n>>>> +\t\t\t\tbreak;\n>>>> +\t\t\tdefault:\n>>>> +\t\t\t\tret = 0;\n>>>> +\t\t\t\terror(\"unsupported protocol version %s for external persistent filter '%s'\",\n>>>> +\t\t\t\t\tnbuf.buf, cmd);\n>>>> +\t\t}\n>>>> +\t}\n>>>> +\tstring_list_clear(&split, 0);\n>>>> +\tstrbuf_release(&nbuf);\n>>>> +\n>>>> +\tif (!ret) {\n>>>> +\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n>>>> +\t\treturn NULL;\n>>>> +\t}\n>>> \n>>> Do we handle persistent filter command being killed before it finishes?\n>>> Or exiting with error? I don't know this Git API...\n>> \n>> If the \"apply_filter\" function fails then Git would proceed and just not\n>> filter the content. If you define the \"required\" flag for the filter then\n>> Git would error in that case.\n> \n> Ah, right. \n> \n> [...]\n>>>> diff --git a/t/t0021/rot13.pl b/t/t0021/rot13.pl\n>>> \n>>> That's bit more than rot13... but it might be O.K. for a filename here.\n>> \n>> \"rot13-$FEATURE_NAME.pl\" ?\n> \n> As I said, rot13.pl is all right; if change, then perhaps to rot13-filter.pl \n\nOK.\n\n\n> [...]\n>>>> + my $input;\n>>>> + {\n>>>> + binmode(STDIN);\n>>>> + my $bytes_read = 0;\n>>>> + $bytes_read = read STDIN, $input, $filelen;\n>>>> + if ( $bytes_read != $filelen ) {\n>>>> + die \"not enough to read\";\n>>> \n>>> I know it's only a test script (well, a part of one), but we would probably\n>>> want to have more information in the case of a real filter.\n>> \n>> True. Do you think there is anything to change in the script, though?\n> \n> No, I don't think so. It is enough that the test script would crash\n> if fed incorrect data from Git. Better error messages would be nice,\n> but are not necessary.\n> \n> + die \"not enough to read: expected $filelen, got $bytes_read\";\n> \n> I have noticed that some 'die' have \"\\n\" at the end, and some do\n> not. If I remember correctly it is for supressing error message from\n> Perl, with filename and line number, isn't it? Anyway, we probably\n> want to be consistent.\n\nOK.\n\n\n>>>> + }\n>>>> + }\n>>> \n>>> What happens if $filelen is zero, or negative? Ah, I see that $output\n>>> would be undef... which is bad, I think.\n>> \n>> Is this something we need to consider in the test script?\n> \n> I think the test script should die if it gets incorrect content length\n> (e.g. negative), so that it catches bug on the Git side. It should\n> work for zero-length files, even if we don't test it -- we can in the\n> future.\n\nOK.\n\n\n> [...]\n>>>> + print STDOUT \"fail\";\n>>> \n>>> This is not defined in the protocol description! Unless anything that\n>>> does not conform to the specification would work here, but at least it\n>>> is a recommended practice to be described in the documentation, don't\n>>> you think?\n>>> \n>>> What would happen in $output_len is 4?\n>> \n>> Then it would work :D\n>> I understand your point. However, this is not a reference implementation.\n>> It is a test script that is supposed to trigger bad behavior which we can test. \n>> Therefore, I would argue that such a return value is OK. I will document it in \n>> the header, though. \n> \n> Why print \"fail\", and not die?\n\nAgree, \"die\" is better.\n\n\n> The problem is do the protocol need to have some way of communicating\n> errors from the filter to Git? Perhaps using stderr would be enough\n> (but then Git would need to drain it, I think... unless it is not\n> redirected), perhaps some command is needed?\n> \n> For example, instead of:\n> \n> \tGit <-- Filter: \"15\\n\"\n> \tGit <-- Filter: \"SMUDGED_CONTENT\"\n> \n> perhaps filter should return\n> \n> \tGit <-- Filter: \"error\\n\"\n> \tGit <-- Filter: \"ONE_LINE_OF_ERROR_DESCRIPTION\\n\"\n> \n> on error? Or if printing expected output length upfront is easier,\n> use a signal (but that is supposedly not that reliable as message\n> passing mechanism)?\n> \n> It might be the case that some files return errors, but some do not.\n\nI would prefer it if the filter just dies in case of trouble and that\nway communicates to Git that something went wrong. Everything else\njust complicates the protocol.\n\n\n> BTW. do we test the case where filter fails, or returns wrong output?\n\nYes, I added a failure test in v2.\n\nThanks a lot,\nLars"},{"id":"292135","messageId":"34F8B256-A129-4238-AE1A-CD8C1DE8C7DA@gmail.com","threadId":"42904","inReplyTo":"5795337A.1040701@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-25T20:16:14Z","receivedAt":"2016-07-25T20:16:37Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 24 Jul 2016, at 23:30, Jakub Narębski wrote:\n\n> W dniu 2016-07-24 o 22:14, Jakub Narębski pisze:\n>> W dniu 2016-07-24 o 20:36, Lars Schneider pisze:\n> \n>>> I agree that the name is not ideal. \"UseProtocol\" as it is would be a boolean. \n>>> I thought about \"persistent\" but this name wouldn't convey the scope of the \n>>> persistency (\"persistent for one Git operation\" vs. \"persistent for many Git \n>>> operations\"). What do you think about the protocol as int version idea\n>>> described in $gmane/300155 ?\n>> \n>> You mean the `protocol` as a config variable name (fully name being\n>> `filter..protocol`), being integer-valued, isn't it? Wouldn't\n>> `protocolVersion` be a more explicit?\n> \n> Just throwing out further ideas:\n> \n> Perhaps make `persistent` string-valued variable, with the only value\n> supported for now, namely \"per-process\" / \"operation\"?\n> \n> Perhaps require for `pidfile` to be present for it to be daemon,\n> that is persist for possibly many Git operations. Or allow \"daemon\"\n> or \"server\" value for `persistent`, then?\n\nI like the direction of this idea. What if we use a string-valued \n\"filter..protocol\" with the following options:\n\n\"simple\" / \"invocation-per-file\" / << empty >> --> current clean/smudge behavior\n\"invocation-per-process\" --> new, proposed behavior\n\nIf necessary this could be enhanced in the future to support even a \"daemon\"\nmode (with a pidfile config).\n\nThanks,\nLars\n\n"},{"id":"292137","messageId":"940904FE-93EB-45E9-B3F2-54C07BBF7E54@gmail.com","threadId":"42904","inReplyTo":"194ea810-76ff-f32c-0f8a-57e8e60b65f5@ramsayjones.plus.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-25T20:24:44Z","receivedAt":"2016-07-25T20:24:57Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 25 Jul 2016, at 00:36, Ramsay Jones wrote:\n\n> On 24/07/16 18:16, Lars Schneider wrote:\n>> \n>> On 23 Jul 2016, at 01:19, Ramsay Jones wrote:\n>> \n>>> On 22/07/16 16:49, larsxschneider@gmail.com wrote:\n>>>> From: Lars Schneider \n>>>> \n>>>> Git's clean/smudge mechanism invokes an external filter process for every\n>>>> single blob that is affected by a filter. If Git filters a lot of blobs\n>>>> then the startup time of the external filter processes can become a\n>>>> significant part of the overall Git execution time.\n>>>> \n>>>> This patch adds the filter..useProtocol option which, if enabled,\n>>>> keeps the external filter process running and processes all blobs with\n>>>> the following protocol over stdin/stdout.\n>>>> \n>>>> 1. Git starts the filter on first usage and expects a welcome message\n>>>> with protocol version number:\n>>>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>>>> \tGit <-- Filter: \"version 1\"\n>>> \n>>> Hmm, I was a bit surprised to see a 'filter' talk first (but so long as the\n>>> interaction is fully defined, I guess it doesn't matter).\n>>> \n>>> [If you wanted to check for a version, you could add a \"version\" command\n>>> instead, just like \"clean\" and \"smudge\".]\n>> \n>> It was a conscious decision to have the `filter` talk first. My reasoning was:\n>> \n>> (1) I want a reliable way to distinguish the existing filter protocol (\"single-shot \n>> invocation\") from the new one (\"long running\"). I don't think there would be a\n>> situation where the existing protocol would talk first. Therefore the users would\n>> not accidentally mix them with a possibly half working, undetermined, outcome.\n> \n> If an 'single-shot' filter were incorrectly configured, instead of a new one, then\n> the interaction could last a little while - since it would result in deadlock! ;-)\n> \n> [If Git talks first instead, configuring a 'single-shot' filter _may_ still result\n> in a deadlock - depending on pipe size, etc.]\n\nDo you think this is an issue that needs to be addressed in the first version?\nIf yes, I would probably look into \"select\" to specify a timeout for the filter.\nHowever, wouldn't the current \"single-shot\" clean/smudge filter block in the \nsame way if they don't write anything?\n\n\n>> (2) In the future we could extend the pipe protocol (see $gmane/297994, it's very\n>> interesting). A filter could check Git's version and then pick the most appropriate\n>> filter protocol on startup.\n>> \n>> \n>>> [...]\n>>>> +static struct cmd2process *start_protocol_filter(const char *cmd)\n>>>> +{\n>>>> +\tint ret = 1;\n>>>> +\tstruct cmd2process *entry = NULL;\n>>>> +\tstruct child_process *process = NULL;\n>>>> +\tstruct strbuf nbuf = STRBUF_INIT;\n>>>> +\tstruct string_list split = STRING_LIST_INIT_NODUP;\n>>>> +\tconst char *argv[] = { NULL, NULL };\n>>>> +\tconst char *header = \"git-filter-protocol\\nversion\";\n>>>> +\n>>>> +\tentry = xmalloc(sizeof(*entry));\n>>>> +\thashmap_entry_init(entry, strhash(cmd));\n>>>> +\tentry->cmd = cmd;\n>>>> +\tprocess = &entry->process;\n>>>> +\n>>>> +\tchild_process_init(process);\n>>>> +\targv[0] = cmd;\n>>>> +\tprocess->argv = argv;\n>>>> +\tprocess->use_shell = 1;\n>>>> +\tprocess->in = -1;\n>>>> +\tprocess->out = -1;\n>>>> +\n>>>> +\tif (start_command(process)) {\n>>>> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n>>>> +\t\treturn NULL;\n>>>> +\t}\n>>>> +\tstrbuf_reset(&nbuf);\n>>>> +\n>>>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>>>> +\tret &= strbuf_read_once(&nbuf, process->out, 0) > 0;\n>>> \n>>> Hmm, how much will be read into nbuf by this single call?\n>>> Since strbuf_read_once() makes a single call to xread(), with\n>>> a len argument that will probably be 8192, you can not really\n>>> tell how much it will read, in general. (xread() does not\n>>> guarantee how many bytes it will read.)\n>>> \n>>> In particular, it could be less than strlen(header).\n>> \n>> As mentioned to Torsten in $gmane/300156, I will add a newline\n>> and then read until I find the second newline. That should solve\n>> the problem, right?\n>> \n>> (You wrote in $gmane/300119 that I should ignore your email but\n>> I think you have a valid point here ;-)\n> \n> Heh, as I said, it was late and I was trying to do several things\n> at once. (I am updating 3 installations of Linux Mint 17.3 to Linux\n> Mint 18 - I decided to do a complete re-install, since I needed to\n> change partition sizes anyway. I have only just got email back up ...)\n> \n> I stopped commenting on the patch early but, after sending the first\n> email, I decided to scan the rest of your patch before going to bed\n> and noticed something which would invalidate my comments ...\n> \n>> \n>> \n>>>> [...]\n>>>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>>>> +\tswitch (entry->protocol) {\n>>>> +\t\tcase 1:\n>>>> +\t\t\tif (fd >= 0 && !src) {\n>>>> +\t\t\t\tret &= fstat(fd, &fileStat) != -1;\n>>>> +\t\t\t\tlen = fileStat.st_size;\n>>>> +\t\t\t}\n>>>> +\t\t\tstrbuf_reset(&nbuf);\n>>>> +\t\t\tstrbuf_addf(&nbuf, \"%s\\n%s\\n%zu\\n\", filter_type, path, len);\n>>>> +\t\t\tret &= write_str_in_full(process->in, nbuf.buf) > 1;\n>>> \n>>> why not write_in_full(process->in, nbuf.buf, nbuf.len) ?\n>> OK, this would save a \"strlen\" call. Do you think such a function could be of general\n>> use? If yes, then I would add:\n>> \n>> static inline ssize_t write_strbuf_in_full(int fd, struct strbuf *str)\n>> {\n>> \treturn write_in_full(fd, str->buf, str->len);\n>> }\n> \n> [I don't have strong feelings either way (but I suspect it's not worth it).]\n\nOK\n\n\n>>>> +\t\t\tif (len > 0) {\n>>>> +\t\t\t\tif (src)\n>>>> +\t\t\t\t\tret &= write_in_full(process->in, src, len) == len;\n>>>> +\t\t\t\telse if (fd >= 0)\n>>>> +\t\t\t\t\tret &= copy_fd(fd, process->in) == 0;\n>>>> +\t\t\t\telse\n>>>> +\t\t\t\t\tret &= 0;\n>>>> +\t\t\t}\n>>>> +\n>>>> +\t\t\tstrbuf_reset(&nbuf);\n>>>> +\t\t\twhile (xread(process->out, &c, 1) == 1 && c != '\\n')\n>>>> +\t\t\t\tstrbuf_addchars(&nbuf, c, 1);\n>>>> +\t\t\tnbuf_len = (size_t)strtol(nbuf.buf, &strtol_end, 10);\n>>>> +\t\t\tret &= (strtol_end != nbuf.buf && errno != ERANGE);\n>>>> +\t\t\tstrbuf_reset(&nbuf);\n>>>> +\t\t\tif (nbuf_len > 0)\n>>>> +\t\t\t\tret &= strbuf_read_once(&nbuf, process->out, nbuf_len) == nbuf_len;\n>>> \n>>> Again, how many bytes will be read?\n>>> Note, that in the default configuration, a _maximum_ of\n>>> MAX_IO_SIZE (8MB or SSIZE_MAX, whichever is smaller) bytes\n>>> will be read.\n> \n> ... In particular, your 2GB test case should not have worked, so\n> I assumed that I had missed a loop somewhere ...\n\nThanks a lot for this comment. The 2GB test case was bogus... v2\nwill have a much improved version :-)\n\n\n>> Would something like this be more appropriate?\n>> \n>> strbuf_reset(&nbuf);\n>> if (nbuf_len > 0) {\n>> strbuf_grow(&nbuf, nbuf_len);\n>> ret &= read_in_full(process->out, nbuf.buf, nbuf_len) == nbuf_len;\n>> }\n> \n> ... and this looks better. [Note: this comment would apply equally to the\n> version message.]\n\nAnd it works better with large files, too :D\n\n\n> [Hmm, now can I remember which packages I need to install ...]\n\n:-)\n\n\nThanks,\nLars"},{"id":"292139","messageId":"7D0AD199-F077-4EB2-83A5-58EB603973CA@gmail.com","threadId":"42904","inReplyTo":"57954DA3.9090006@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-25T20:32:39Z","receivedAt":"2016-07-25T20:32:48Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 25 Jul 2016, at 01:22, Jakub Narębski wrote:\n\n> W dniu 2016-07-25 o 00:36, Ramsay Jones pisze:\n>> On 24/07/16 18:16, Lars Schneider wrote:\n>>> On 23 Jul 2016, at 01:19, Ramsay Jones wrote:\n>>>> On 22/07/16 16:49, larsxschneider@gmail.com wrote:\n> [...]\n>>>>> This patch adds the filter..useProtocol option which, if enabled,\n>>>>> keeps the external filter process running and processes all blobs with\n>>>>> the following protocol over stdin/stdout.\n>>>>> \n>>>>> 1. Git starts the filter on first usage and expects a welcome message\n>>>>> with protocol version number:\n>>>>> \tGit <-- Filter: \"git-filter-protocol\\n\"\n>>>>> \tGit <-- Filter: \"version 1\"\n>>>> \n>>>> Hmm, I was a bit surprised to see a 'filter' talk first (but so long as the\n>>>> interaction is fully defined, I guess it doesn't matter).\n>>> \n>>> It was a conscious decision to have the `filter` talk first. My reasoning was:\n>>> \n>>> (1) I want a reliable way to distinguish the existing filter protocol (\"single-shot \n>>> invocation\") from the new one (\"long running\"). I don't think there would be a\n>>> situation where the existing protocol would talk first. Therefore the users would\n>>> not accidentally mix them with a possibly half working, undetermined, outcome.\n>> \n>> If an 'single-shot' filter were incorrectly configured, instead of a new one, then\n>> the interaction could last a little while - since it would result in deadlock! ;-)\n>> \n>> [If Git talks first instead, configuring a 'single-shot' filter _may_ still result\n>> in a deadlock - depending on pipe size, etc.]\n> \n> Would it be possible to do an equivalent of sending empty file to the filter?\n> If it is misconfigured old-style script, it would exit after possibly empty\n> output; if not, we would start new-style interaction.\n\nI think we would need to close the pipe to communicate \"end\" to the filter, no?\nI would prefer to define the protocol explicitly as this is clearly easier.\n\n\n> \n> This should be, if we agree that detecting misconfigured filters is a good\n> thing, tested.\n> \n>>> \n>>> (2) In the future we could extend the pipe protocol (see $gmane/297994, it's very\n>>> interesting). A filter could check Git's version and then pick the most appropriate\n>>> filter protocol on startup.\n\nThanks,\nLars"},{"id":"292190","messageId":"57974240.5030003@gmail.com","threadId":"42904","inReplyTo":"7D0AD199-F077-4EB2-83A5-58EB603973CA@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-26T10:58:08Z","receivedAt":"2016-07-26T10:58:27Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-25 o 22:32, Lars Schneider pisze: \n> On 25 Jul 2016, at 01:22, Jakub Narębski wrote:\n>> W dniu 2016-07-25 o 00:36, Ramsay Jones pisze:\n>>> On 24/07/16 18:16, Lars Schneider wrote:\n\n>>>> It was a conscious decision to have the `filter` talk first.\n>>>> My reasoning was:\n>>>> \n>>>> (1) I want a reliable way to distinguish the existing filter \n>>>> protocol (\"single-shot invocation\") from the new one (\"long \n>>>> running\"). I don't think there would be a situation where the \n>>>> existing protocol would talk first. Therefore the users would \n>>>> not accidentally mix them with a possibly half working, \n>>>> undetermined, outcome.\n>>> \n>>> If an 'single-shot' filter were incorrectly configured, instead \n>>> of a new one, then the interaction could last a little while - \n>>> since it would result in deadlock! ;-)\n>>> \n>>> [If Git talks first instead, configuring a 'single-shot' filter \n>>> _may_ still result in a deadlock - depending on pipe size, etc.]\n>> \n>> Would it be possible to do an equivalent of sending empty file to \n>> the filter? If it is misconfigured old-style script, it would exit \n>> after possibly empty output; if not, we would start new-style \n>> interaction.\n> \n> I think we would need to close the pipe to communicate \"end\" to the \n> filter, no? I would prefer to define the protocol explicitly as this \n> is clearly easier.\n\nWell, we could always close stdin of a script, check if it quits,\nthen reopen. Or close stdin, and send commands via file descriptor 4.\nOr send SIGPIPE. But I don't know if it is a good idea.\n\n> On 25 Jul 2016, at 00:36, Ramsay Jones wrote:\n\n>> If an 'single-shot' filter were incorrectly configured, instead of\n>> a new one, then the interaction could last a little while - since\n>> it would result in deadlock! ;-)\n>> \n>> [If Git talks first instead, configuring a 'single-shot' filter\n>> _may_ still result in a deadlock - depending on pipe size, etc.]\n> \n> Do you think this is an issue that needs to be addressed in the first\n> version? If yes, I would probably look into \"select\" to specify a\n> timeout for the filter.\n\nThis might be a better idea. Additionally, it would make it possible\nto detect buggy v2 filter scripts.\n\n> However, wouldn't the current \"single-shot\"\n> clean/smudge filter block in the same way if they don't write\n> anything?\n\nHmmm... so deadlocking (waiting for user to press ^C) might be\nan acceptable solution. It would be good to tell him or her why\nthere was a deadlock (catch SIGINT), that Git was waiting for\nspecific command in a specific filter driver, for a specific file.\n\n\nOn the other hand v2 protocol has an additional problem: users\nswitching to v2, while using old one-shot filters (that worked\ncorrectly). So in my opinion you need to ensure two things:\n\n(1) name things in such way that it is easy to see that you\nneed to write filter script specifically for the v2 protocol,\n\n(2) if possible, do not hang but warn the user if he or she\nwants to use v1 filter (per-file) with v2 protocol (per-command),\nor at least help diagnose the issue.\n\n>> This should be tested, if we agree that detecting misconfigured \n>> filters is a good thing.\n\n[clarified]\n\n-- \nJakub Narębski\n"},{"id":"292195","messageId":"57975662.9050707@gmail.com","threadId":"42904","inReplyTo":"34F8B256-A129-4238-AE1A-CD8C1DE8C7DA@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-26T12:24:02Z","receivedAt":"2016-07-26T12:24:21Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-25 o 22:16, Lars Schneider pisze:\n> \n> On 24 Jul 2016, at 23:30, Jakub Narębski wrote:\n> \n>> W dniu 2016-07-24 o 22:14, Jakub Narębski pisze:\n>>> W dniu 2016-07-24 o 20:36, Lars Schneider pisze:\n>>\n>>>> I agree that the name is not ideal. \"UseProtocol\" as it is would be a boolean. \n>>>> I thought about \"persistent\" but this name wouldn't convey the scope of the \n>>>> persistency (\"persistent for one Git operation\" vs. \"persistent for many Git \n>>>> operations\"). What do you think about the protocol as int version idea\n>>>> described in $gmane/300155 ?\n>>>\n>>> You mean the `protocol` as a config variable name (fully name being\n>>> `filter..protocol`), being integer-valued, isn't it? Wouldn't\n>>> `protocolVersion` be a more explicit?\n>>\n>> Just throwing out further ideas:\n>>\n>> Perhaps make `persistent` string-valued variable, with the only value\n>> supported for now, namely \"per-process\" / \"operation\"?\n>>\n>> Perhaps require for `pidfile` to be present for it to be daemon,\n>> that is persist for possibly many Git operations. Or allow \"daemon\"\n>> or \"server\" value for `persistent`, then?\n> \n> I like the direction of this idea. What if we use a string-valued \n> \"filter..protocol\" with the following options:\n> \n> \"simple\" / \"invocation-per-file\" / << empty >> --> current clean/smudge behavior\n> \"invocation-per-process\" --> new, proposed behavior\n> \n> If necessary this could be enhanced in the future to support even a \"daemon\"\n> mode (with a pidfile config).\n\nThough, after thinking about it, this solution has the problem\nthat people might think that they can use their old per-file\nfilters, just flipping the `filter..protocol`.\n\nI dunno.\n-- \nJakub Narębski\n\n"},{"id":"292209","messageId":"57977118.1020006@gmail.com","threadId":"42904","inReplyTo":"1EF550A7-F6D4-4BFE-A4A7-D375268215DE@gmail.com","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-26T14:18:00Z","receivedAt":"2016-07-26T14:18:36Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-25 o 22:09, Lars Schneider pisze:\n> On 24 Jul 2016, at 22:14, Jakub Narębski wrote:\n>> W dniu 2016-07-24 o 20:36, Lars Schneider pisze:\n>>> On 23 Jul 2016, at 02:11, Jakub Narębski wrote:\n>>>> W dniu 2016-07-22 o 17:49, larsxschneider@gmail.com pisze:\n>>>>> From: Lars Schneider \n>>>>\n>>>> Nb. this line is only needed if you want author name and/or date\n>>>> different from the email sender, or if you have sender line misconfigured\n>>>> (e.g. lacking the human readable name).\n>>>\n>>> I use \"git format-patch\" to generate these emails:\n>>>\n>>> git format-patch --cover-letter --subject-prefix=\"PATCH ...\" -M $BASE -o $OUTPUT\n>>>\n>>> How would I disable this line? (I already checked the man page to no avail).\n>>\n>> If you are using `git send-email` or equivalent, I think it is\n>> stripped automatically if it is not needed (in you case it was,\n>> because Sender was lacking human readable name... at least I think\n>> it was because of what my email reader inserted as reply line).\n>> If you are using an ordinary email client, you need to remove it\n>> yourself, if needed.\n> \n> Weird. I am sending the patches with this command:\n> \n> git send-email mystuff/* --to=git@vger.kernel.org --cc=...\n> \n> Maybe I need to try the \"--suppress-from\" switch?!\n\nNo, the \"From:\" line is needed, but only because your Sender seems\nto be lacking human-readable name (for some strange reason).\n\nBut let's stop this here.\n \n\n[...]\n>>>> I also agree that we might want to be able to keep clean and smudge\n>>>> filters separate, but be able to run a single program if they are\n>>>> both the same. I think there is a special case for filter unset,\n>>>> and/or filter being \"cat\" -- we would want to keep that.\n>>>\n>>> Since 1a8630d there is a more efficient way to unset a filter ;-)\n>>> Can you think of other cases where the separation would be useful?\n>>\n>> I can't think of any, but it doesn't mean that it does not exist.\n>> It also does not mean that you need to consider situation that may\n>> not happen. Covering one-way filters, like \"indent\" filter for `clean`,\n>> should be enough... they do work with your proposal, don't they?\n> \n> This should work right now but it would be a bit inefficient (the filter\n> would just pass the data unchanged through the smudge command). I plan to\n> add a \"capabilities\" flag to the protocol. Then you can define only\n> the \"clean\" capability and nothing or the current filter mechanism \n> would happen for smudge (I will make a test case to demonstrate that\n> behavior in v2).\n\nIsn't no-op filter (value not set, value set to empty string, \"cat\")\ncaught earlier in a common code? We would certainly want to keep\none-way filter configuration mechanism without many changes.\n\nAlso, this should be of course tested.\n \n>>>> \tGit <-- Filter: \"capabilities clean smudge\\n\"\n>>>>\n>>>> Or we can add capabilities in later version...\n>>>\n>>> That is an interesting idea. My initial thought was to make the capabilities\n>>> of a certain version fix. If we want to add new capabilities then we would \n>>> bump the version. I wonder what others think about your suggestion!\n>>\n>> Using capabilities (like git-upload-pack / git-receive-pack, that is\n>> smart Git transfer protocols do) is probably slightly more difficult on\n>> the Git side (assuming no capabilities negotiation), but also much more\n>> flexible than pure version numbers.\n>>\n>> One possible idea for a capability is support for passing input\n>> and output of a filter via filesystem, like cleanToFile and smudgeFromFile\n>> proposal in 'jh/clean-smudge-annex' (in 'pu').\n>>\n>> For example:\n>>\n>> \tGit <-- Filter: \"capabilities clean smudge cleanToFile smudgeFromFile\\n\"\n> \n> Yes, I like that very much. As stated above, I will add that in v2.\n\nI guess that you would add the idea of capabilities (though this\ncould be left for v3), not support for \"cleanToFile\" / \"smudgeFromFile\"\ncapabilities, and accompanying extension to the protocol, isn't it?\n \n \n>>>>> \tGit --> Filter: \"testfile.dat\\n\"\n>>>>\n>>>> Unfortunately, while sane filenames should not contain newlines[1],\n>>>> the unfortunate fact is that *filenames can include newlines*, and\n>>>> you need to be able to handle that[2]. Therefore you need either to\n>>>> choose a different separator (the only one that can be safely used\n>>>> is \"\\0\", i.e. the NUL character - but it is not something easy to\n>>>> handle by shell scripts), or C-quote filenames as needed, or always\n>>>> C-quote filenames. C-quoting at minimum should include quoting newline\n>>>> character, and the escape character itself.\n>>>>\n>>>> BTW. is it the basename of a file, or a full pathname?\n>>>>\n>>>> [1]: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n>>>> [2]: http://www.dwheeler.com/essays/filenames-in-shell.html\n>>>\n>>> Thanks for this explanation. A bash version of the protocol is not\n>>> trivial (I tried it but ended up using Perl). Therefore I think \"\\0\"\n>>> would be a good choice?\n>>\n>> That, or use git convention of surrounding C-quoted filenames in\n>> double quotes (which means that if it begins with quote, it is C-quoted).\n>>\n>> For example:\n>>\n>> $ git commit -m 'Initial commit'\n>> [master (root-commit) 266dab0] Initial commit\n>> 2 files changed, 2 insertions(+)\n>> create mode 100644 foo\n>> create mode 100644 \"foo \\\" \\\\ ,\"\n>> $ ls -1\n>> foo \" \\ ,\n>> foo\n>>\n>> I'm not sure which solution would be easier for filter writers,\n>> NUL termination, or C-quoting.\n> \n> Unless someone has a convincing argument for one solution or the other\n> I will go with the \\0 termination as it seems easier.\n\nWell, I think both C-quoting when necessary, and NUL-terminating\nis easy from the Git side, but I guess that handling NUL-terminated\nfilenames is easier from scripts.\n \n>>>>> \tGit --> Filter: \"7\\n\"\n>>>>\n>>>> That's the content size in bytes written as an ASCII number.\n>>>\n>>> Correct.\n>>\n>> But not obvious from the description / documentation.\n> \n> I will improve that in v2. Should I add the info that it is base10 or\n> would you consider that a given?\n\nNo, this is obvious, and used thorough Git code / formats. \n\n>>>>> ---\n>>>>> Documentation/gitattributes.txt | 41 +++++++-\n>>>>> convert.c | 210 ++++++++++++++++++++++++++++++++++++++--\n>>>>> t/t0021-conversion.sh | 170 ++++++++++++++++++++++++++++++++\n>>>>\n>>>> Wouldn't it be better to name the test case something more\n>>>> descriptive, for example\n>>>>\n>>>> t/t0021-filter-driver-useProtocol.sh\n>>>>\n>>>> The name of test should be adjusted to final name of the feature,\n>>>> of course.\n>>>\n>>> I think the prefix numbers should be unique, no? And t0022 is already taken.\n>>\n>> I meant here that the \"conversion\" part of \"t/t0021-conversion.sh\" test\n>> filename is not descriptive enough.\n> \n> Ah, I see. You suggest to rename the test case? Would that be OK with the Git\n> community?\n\nAh, I'm sorry. I was in mistaken assumption that it is a new test,\nnot an extension to an existing test case. In this case the filename\nchange can come as a separate patch, anyway.\n \n \n>>> What do you mean by \"discrepancy in how config variables are \n>>> referenced\"?\n>>\n>> What I meant here that filter..smudge and filter..clean\n>> were referenced as \"`smudge` command\" and \"`clean` command\" in the paragraph\n>> you modified.\n>>\n>> Perhaps filter..useProtocol is all right (I have not looked further),\n>> but it should be formatted as `filter..useProtocol` IMVHO.\n> \n> Initially I thought so, too. But \"filter..required\", which is already\n> mentioned in gitattributes.txt, does not use this style. Should I change that, too,\n> or use the existing style?\n\nAll right, if there is a precedent for using this style, it would\nbe all right.\n \n \n>> The problem is do the protocol need to have some way of communicating\n>> errors from the filter to Git? Perhaps using stderr would be enough\n>> (but then Git would need to drain it, I think... unless it is not\n>> redirected), perhaps some command is needed?\n>>\n>> For example, instead of:\n>>\n>> \tGit <-- Filter: \"15\\n\"\n>> \tGit <-- Filter: \"SMUDGED_CONTENT\"\n>>\n>> perhaps filter should return\n>>\n>> \tGit <-- Filter: \"error\\n\"\n>> \tGit <-- Filter: \"ONE_LINE_OF_ERROR_DESCRIPTION\\n\"\n>>\n>> on error? Or if printing expected output length upfront is easier,\n>> use a signal (but that is supposedly not that reliable as message\n>> passing mechanism)?\n>>\n>> It might be the case that some files return errors, but some do not.\n> \n> I would prefer it if the filter just dies in case of trouble and that\n> way communicates to Git that something went wrong. Everything else\n> just complicates the protocol.\n\nAll right, this makes protocol simpler. One thing we might want\nto do is to ensure that stderr from the filter driver (where error\nmessages should be sent IMVHO, and which of course needs to be\ndocumented) goes to the user, possibly with prefixing (like for\nerrors from the remote hooks).\n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"292213","messageId":"323759193.1122705.1469546329160.JavaMail.zimbra@ensimag.grenoble-inp.fr","threadId":"42904","inReplyTo":"20160722154900.19477-3-larsxschneider@gmail.com","subject":"Re: [PATCH v1 2/3] convert: modernize tests","fromName":"Remi Galan Alfonso","fromEmail":"remi.galan-alfonso@ensimag.grenoble-inp.fr","sentAt":"2016-07-26T15:18:49Z","receivedAt":"2016-07-26T15:09:55Z","isPatch":true,"sender":{"key":"remi.galan-alfonso@ensimag.grenoble-inp.fr","avatar":"https://avatars.githubusercontent.com/u/12509162?v=4"},"body":"Hi Lars,\n\nSorry, minor nit that I noticed a couple of days ago but didn't\ncomment on the moment and forgot until now.\n\nLars Schneider wrote:\n> Use `test_config` to set the config, check that files are empty with\n> `test_must_be_empty`, compare files with `test_cmp`, and remove spaces\n> after \">\".\n\nConsidering how close it is to your patch, you might also want to\nremove spaces after '<'.\n\nThere is only one occurrence in this file and it's in a line you are\nalready modifying.\n\nSee below:\n\n> test_expect_success check '\n> \n> - cmp test.o test &&\n> - cmp test.o test.t &&\n> + test_cmp test.o test &&\n> + test_cmp test.o test.t &&\n> \n> # ident should be stripped in the repository\n> git diff --raw --exit-code :test :test.i &&\n> @@ -47,10 +47,10 @@ test_expect_success check '\n> embedded=$(sed -ne \"$script\" test.i) &&\n> test \"z$id\" = \"z$embedded\" &&\n> \n> - git cat-file blob :test.t > test.r &&\n> + git cat-file blob :test.t >test.r &&\n> \n> - ./rot13.sh < test.o > test.t &&\n> - cmp test.r test.t\n> + ./rot13.sh < test.o >test.t &&\n\nHere.\n\n> + test_cmp test.r test.t\n> '\n\nThanks,\nRémi\n"},{"id":"292254","messageId":"20160726200028.GA21194@sigill.intra.peff.net","threadId":"42904","inReplyTo":"20160723072721.GA20875@starla","subject":"Re: [PATCH v1 3/3] convert: add filter..useProtocol option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-26T20:00:28Z","receivedAt":"2016-07-26T20:02:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jul 23, 2016 at 07:27:21AM +0000, Eric Wong wrote:\n\n> Jakub Narębski wrote:\n> > W dniu 2016-07-22 o 17:49, larsxschneider@gmail.com pisze:\n> > > +use strict;\n> > > +use warnings;\n> > > +use autodie;\n> > \n> > autodie?\n> \n> \"set -e\" for Perl (man autodie)\n> \n> It's been a part of Perl for ages, but I've never used it\n> myself, either; I suppose it's fine for tests...\n\nautodie has been around for a long time, but it only became part of the\nperl core in v5.10.1 (according to Module::CoreList). I think the code\nin perl/ requires only 5.8, but whenever we unconditionally use perl\nwithout respect to NO_PERL (like in the test scripts), we usually shoot\nfor even antique versions of perl like 5.005.\n\nSo by those rules, we should avoid \"autodie\" here, though I wouldn't be\nsurprised if it takes a while for people to complain in practice (most\nmodern systems will have a recent enough perl, but it seems we go\nthrough cycles where every few years somebody posts a bunch of patches\nfor ancient versions of IRIX or some other platform, cleaning up all of\nthese sorts of portability problems).\n\n-Peff\n"},{"id":"292255","messageId":"20160726201105.GB21194@sigill.intra.peff.net","threadId":"42904","inReplyTo":"7DABD0C7-8838-4FEC-B6E5-36F82E521312@gmail.com","subject":"Re: [PATCH v1 0/3] Git filter protocol","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-26T20:11:05Z","receivedAt":"2016-07-26T20:11:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jul 24, 2016 at 01:24:29PM +0200, Lars Schneider wrote:\n\n> What if we would keep the config option \"protocol\" and make it an \"int\"? \n> Undefined or version \"1\" would describe the existing clean/smudge \n> protocol via command line and pipe. Version \"2\" would be the new protocol?\n\nFWIW, that is what I expected when I saw the word \"protocol\".\n\nIt's possible that we might never need a \"v3\" protocol specified here,\nbecause your v2 protocol should be able to auto-upgrade. That is, if we\nstart a filter and it says \"hi, I am speaking protocol 3\", then Git\nknows to speak the requested version from there on (or will barf if it\ndoesn't understand the version).\n\nSo you'd only need to say \"filter.foo.protocol=v3\" if there was some\nprotocol change that broke the initial conversation.\n\nThat does mean it is the filter which sets the maximum protocol level,\nnot git. So a filter which can speak v3 or v2 (to work with older\nversions of git) does not know which to use. That could be solved by\nspecifying\n\n [filter \"foo\"]\n smudge = my-filter --version=3\n\nor something.\n\nI'm not sure it's worth thinking too hard about what-ifs here. We should\ndo the simplest thing that will work and avoid painting ourselves into a\ncorner for future upgrades.\n\n> > * The way the serialized access to these long-running processes\n> > work in 3/3 would make it harder or impossible to later\n> > parallelize conversion? I am imagining a far future where we\n> > would run \"git checkout .\" using (say) two threads, one\n> > responsible for active_cache[0..active_nr/2] and the other\n> > responsible for the remainder.\n> I hope this future is not too far away :-) \n> However, I don't think that would be a problem as we could start the\n> long-running process once for each checkout thread, no?\n\nThat's reasonable if we have a worker-thread model (which seems likely,\nas that's what we use elsewhere in git), and if the main cost you want\nto amortize is just process startup (so you pay the cost once per\nworker, which is a constant factor and not too bad).\n\nIt's not a good model if the long-running process wants to amortize\nother shared costs. For example, persistent https connections. Or even\nuser-interactive authentication steps, where you really would prefer to\ndo them once. The filter can implement its own ad-hoc sharing of\nresources, but doing that portably is complicated.\n\nOf course having an async protocol between git and the filter is also\ncomplicated. Perhaps that's something that could wait for a v3 if\nsomebody really wants it.\n\n-Peff\n"},{"id":"292260","messageId":"CAPc5daXj9vafDDRFmXHCOoMoaHQ_YJgzCF2z9f9frCZceZuJ-A@mail.gmail.com","threadId":"42904","inReplyTo":"323759193.1122705.1469546329160.JavaMail.zimbra@ensimag.grenoble-inp.fr","subject":"Re: [PATCH v1 2/3] convert: modernize tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-26T20:40:51Z","receivedAt":"2016-07-26T20:42:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Tue, Jul 26, 2016 at 8:18 AM, Remi Galan Alfonso\n wrote:\n> Considering how close it is to your patch, you might also want to\n> remove spaces after '<'.\n>\n> There is only one occurrence in this file and it's in a line you are\n> already modifying.\n\nGood eyes.\n"},{"id":"292286","messageId":"20160727000605.49982-1-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160722154900.19477-1-larsxschneider@gmail.com","subject":"[PATCH v2 0/5] Git filter protocol","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T00:06:00Z","receivedAt":"2016-07-27T00:06:18Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nHi,\n\nthanks a lot for the extensive reviews. I tried to address all mentioned\nconcerns and summarized them below. The most prominent changes since v1 are\nthe following:\n* pipe communication uses a packet format (pkt-line) based protocol\n* a long running filter application is defined with \"filter..process\"\n* Git offers a number of filter capabilties that a filter can request\n (right now only \"smudge\" and \"clean\" - in the future maybe \"cleanFromFile\",\n \"smudgeToFile\", and/or \"stream\")\n\nCheers,\nLars\n\n\n## Torsten:\n* add \"\\n\" line terminator after version in init sequence\n* prepare big file for EXPENSIVE tests once\n* set \"#!/usr/bin/perl\" as shebang for rot13.pl to mimic other Perl test scripts\n* add test_have_prereq PERL to t0021\n\n## Ramsay:\n* use write_in_full(process->in, nbuf.buf, nbuf.len) to avoid unneccsary strlen call\n* use read_in_full to read data that exceeds MAX_IO_SIZE properly\n* fix test case to check for large file filtering\n\n## Jakub:\n* use standard input/standard output instead of stdin/stdout\n* replace global variable \"cmd_process_map\" with a function parameter where possible\n* avoid \"strbuf_reset\" after STRBUF_INIT\n* align test_config_global\n* rename rot13.pl to rot13-filter.pl\n* make Perl style consistent\n* describe hard coded filenames in test filter header\n* improve docs\n* add filter capabilities field (enables cleanToFile, smudgeFromFile, and/or stream later)\n* explain that content size in bytes is encoded in ASCII\n* consistent line ending for die call in Perl (without \"\\n\")\n* make rot13 test filter die in case of failure (instead of returning \"fail\")\n\n## Eric:\n* flush explicitly in Perl test filter\n* do not initialize variables to NULL if they are set unconditionally\n* fix no-op stop_protocol_filter\n* use off_t instead of size_t\n* improve test filter int parsing ($filelen =~ /\\A\\d+\\z/ or die \"bad filelen: $filelen\")\n\n## Peff:\n* use pkt-line protocol\n* do not use Perl autodie\n\n## Remi:\n* remove spaces after '<'\n\n\n\nLars Schneider (5):\n convert: quote filter names in error messages\n convert: modernize tests\n pkt-line: extract and use `set_packet_header` function\n convert: generate large test files only once\n convert: add filter..process option\n\n Documentation/gitattributes.txt | 54 +++++++-\n convert.c | 281 +++++++++++++++++++++++++++++++++++++---\n pkt-line.c | 15 ++-\n pkt-line.h | 1 +\n t/t0021-conversion.sh | 272 ++++++++++++++++++++++++++++++++------\n t/t0021/rot13-filter.pl | 146 +++++++++++++++++++++\n 6 files changed, 704 insertions(+), 65 deletions(-)\n create mode 100755 t/t0021/rot13-filter.pl\n\n--\n2.9.0\n\n"},{"id":"292287","messageId":"20160727000605.49982-3-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160727000605.49982-1-larsxschneider@gmail.com","subject":"[PATCH v2 2/5] convert: modernize tests","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T00:06:02Z","receivedAt":"2016-07-27T00:06:28Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nUse `test_config` to set the config, check that files are empty with\n`test_must_be_empty`, compare files with `test_cmp`, and remove spaces\nafter \">\".\n\nSigned-off-by: Lars Schneider \n---\n t/t0021-conversion.sh | 62 +++++++++++++++++++++++++--------------------------\n 1 file changed, 31 insertions(+), 31 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 7bac2bc..7b45136 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -13,8 +13,8 @@ EOF\n chmod +x rot13.sh\n \n test_expect_success setup '\n-\tgit config filter.rot13.smudge ./rot13.sh &&\n-\tgit config filter.rot13.clean ./rot13.sh &&\n+\ttest_config filter.rot13.smudge ./rot13.sh &&\n+\ttest_config filter.rot13.clean ./rot13.sh &&\n \n \t{\n \t echo \"*.t filter=rot13\"\n@@ -38,8 +38,8 @@ script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n \n test_expect_success check '\n \n-\tcmp test.o test &&\n-\tcmp test.o test.t &&\n+\ttest_cmp test.o test &&\n+\ttest_cmp test.o test.t &&\n \n \t# ident should be stripped in the repository\n \tgit diff --raw --exit-code :test :test.i &&\n@@ -47,10 +47,10 @@ test_expect_success check '\n \tembedded=$(sed -ne \"$script\" test.i) &&\n \ttest \"z$id\" = \"z$embedded\" &&\n \n-\tgit cat-file blob :test.t > test.r &&\n+\tgit cat-file blob :test.t >test.r &&\n \n-\t./rot13.sh < test.o > test.t &&\n-\tcmp test.r test.t\n+\t./rot13.sh test.t &&\n+\ttest_cmp test.r test.t\n '\n \n # If an expanded ident ever gets into the repository, we want to make sure that\n@@ -130,7 +130,7 @@ test_expect_success 'filter shell-escaped filenames' '\n \n \t# delete the files and check them out again, using a smudge filter\n \t# that will count the args and echo the command-line back to us\n-\tgit config filter.argc.smudge \"sh ./argc.sh %f\" &&\n+\ttest_config filter.argc.smudge \"sh ./argc.sh %f\" &&\n \trm \"$normal\" \"$special\" &&\n \tgit checkout -- \"$normal\" \"$special\" &&\n \n@@ -141,7 +141,7 @@ test_expect_success 'filter shell-escaped filenames' '\n \ttest_cmp expect \"$special\" &&\n \n \t# do the same thing, but with more args in the filter expression\n-\tgit config filter.argc.smudge \"sh ./argc.sh %f --my-extra-arg\" &&\n+\ttest_config filter.argc.smudge \"sh ./argc.sh %f --my-extra-arg\" &&\n \trm \"$normal\" \"$special\" &&\n \tgit checkout -- \"$normal\" \"$special\" &&\n \n@@ -154,9 +154,9 @@ test_expect_success 'filter shell-escaped filenames' '\n '\n \n test_expect_success 'required filter should filter data' '\n-\tgit config filter.required.smudge ./rot13.sh &&\n-\tgit config filter.required.clean ./rot13.sh &&\n-\tgit config filter.required.required true &&\n+\ttest_config filter.required.smudge ./rot13.sh &&\n+\ttest_config filter.required.clean ./rot13.sh &&\n+\ttest_config filter.required.required true &&\n \n \techo \"*.r filter=required\" >.gitattributes &&\n \n@@ -165,17 +165,17 @@ test_expect_success 'required filter should filter data' '\n \n \trm -f test.r &&\n \tgit checkout -- test.r &&\n-\tcmp test.o test.r &&\n+\ttest_cmp test.o test.r &&\n \n \t./rot13.sh expected &&\n \tgit cat-file blob :test.r >actual &&\n-\tcmp expected actual\n+\ttest_cmp expected actual\n '\n \n test_expect_success 'required filter smudge failure' '\n-\tgit config filter.failsmudge.smudge false &&\n-\tgit config filter.failsmudge.clean cat &&\n-\tgit config filter.failsmudge.required true &&\n+\ttest_config filter.failsmudge.smudge false &&\n+\ttest_config filter.failsmudge.clean cat &&\n+\ttest_config filter.failsmudge.required true &&\n \n \techo \"*.fs filter=failsmudge\" >.gitattributes &&\n \n@@ -186,9 +186,9 @@ test_expect_success 'required filter smudge failure' '\n '\n \n test_expect_success 'required filter clean failure' '\n-\tgit config filter.failclean.smudge cat &&\n-\tgit config filter.failclean.clean false &&\n-\tgit config filter.failclean.required true &&\n+\ttest_config filter.failclean.smudge cat &&\n+\ttest_config filter.failclean.clean false &&\n+\ttest_config filter.failclean.required true &&\n \n \techo \"*.fc filter=failclean\" >.gitattributes &&\n \n@@ -197,8 +197,8 @@ test_expect_success 'required filter clean failure' '\n '\n \n test_expect_success 'filtering large input to small output should use little memory' '\n-\tgit config filter.devnull.clean \"cat >/dev/null\" &&\n-\tgit config filter.devnull.required true &&\n+\ttest_config filter.devnull.clean \"cat >/dev/null\" &&\n+\ttest_config filter.devnull.required true &&\n \tfor i in $(test_seq 1 30); do printf \"%1048576d\" 1; done >30MB &&\n \techo \"30MB filter=devnull\" >.gitattributes &&\n \tGIT_MMAP_LIMIT=1m GIT_ALLOC_LIMIT=1m git add 30MB\n@@ -207,7 +207,7 @@ test_expect_success 'filtering large input to small output should use little mem\n test_expect_success 'filter that does not read is fine' '\n \ttest-genrandom foo $((128 * 1024 + 1)) >big &&\n \techo \"big filter=epipe\" >.gitattributes &&\n-\tgit config filter.epipe.clean \"echo xyzzy\" &&\n+\ttest_config filter.epipe.clean \"echo xyzzy\" &&\n \tgit add big &&\n \tgit cat-file blob :big >actual &&\n \techo xyzzy >expect &&\n@@ -215,20 +215,20 @@ test_expect_success 'filter that does not read is fine' '\n '\n \n test_expect_success EXPENSIVE 'filter large file' '\n-\tgit config filter.largefile.smudge cat &&\n-\tgit config filter.largefile.clean cat &&\n+\ttest_config filter.largefile.smudge cat &&\n+\ttest_config filter.largefile.clean cat &&\n \tfor i in $(test_seq 1 2048); do printf \"%1048576d\" 1; done >2GB &&\n \techo \"2GB filter=largefile\" >.gitattributes &&\n \tgit add 2GB 2>err &&\n-\t! test -s err &&\n+\ttest_must_be_empty err &&\n \trm -f 2GB &&\n \tgit checkout -- 2GB 2>err &&\n-\t! test -s err\n+\ttest_must_be_empty err\n '\n \n test_expect_success \"filter: clean empty file\" '\n-\tgit config filter.in-repo-header.clean \"echo cleaned && cat\" &&\n-\tgit config filter.in-repo-header.smudge \"sed 1d\" &&\n+\ttest_config filter.in-repo-header.clean \"echo cleaned && cat\" &&\n+\ttest_config filter.in-repo-header.smudge \"sed 1d\" &&\n \n \techo \"empty-in-worktree filter=in-repo-header\" >>.gitattributes &&\n \t>empty-in-worktree &&\n@@ -240,8 +240,8 @@ test_expect_success \"filter: clean empty file\" '\n '\n \n test_expect_success \"filter: smudge empty file\" '\n-\tgit config filter.empty-in-repo.clean \"cat >/dev/null\" &&\n-\tgit config filter.empty-in-repo.smudge \"echo smudged && cat\" &&\n+\ttest_config filter.empty-in-repo.clean \"cat >/dev/null\" &&\n+\ttest_config filter.empty-in-repo.smudge \"echo smudged && cat\" &&\n \n \techo \"empty-in-repo filter=empty-in-repo\" >>.gitattributes &&\n \techo dead data walking >empty-in-repo &&\n-- \n2.9.0\n\n"},{"id":"292288","messageId":"20160727000605.49982-4-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160727000605.49982-1-larsxschneider@gmail.com","subject":"[PATCH v2 3/5] pkt-line: extract and use `set_packet_header` function","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T00:06:03Z","receivedAt":"2016-07-27T00:06:31Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\n`set_packet_header` converts an integer to a 4 byte hex string. Make\nthis function publicly available so that other parts of Git can easily\ngenerate a pkt-line.\n\nSigned-off-by: Lars Schneider \n---\n pkt-line.c | 15 ++++++++++-----\n pkt-line.h | 1 +\n 2 files changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 62fdb37..6820224 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -98,9 +98,17 @@ void packet_buf_flush(struct strbuf *buf)\n }\n \n #define hex(a) (hexchar[(a) & 15])\n-static void format_packet(struct strbuf *out, const char *fmt, va_list args)\n+void set_packet_header(char *buf, const int size)\n {\n \tstatic char hexchar[] = \"0123456789abcdef\";\n+\tbuf[0] = hex(size >> 12);\n+\tbuf[1] = hex(size >> 8);\n+\tbuf[2] = hex(size >> 4);\n+\tbuf[3] = hex(size);\n+}\n+\n+static void format_packet(struct strbuf *out, const char *fmt, va_list args)\n+{\n \tsize_t orig_len, n;\n \n \torig_len = out->len;\n@@ -111,10 +119,7 @@ static void format_packet(struct strbuf *out, const char *fmt, va_list args)\n \tif (n > LARGE_PACKET_MAX)\n \t\tdie(\"protocol error: impossibly long line\");\n \n-\tout->buf[orig_len + 0] = hex(n >> 12);\n-\tout->buf[orig_len + 1] = hex(n >> 8);\n-\tout->buf[orig_len + 2] = hex(n >> 4);\n-\tout->buf[orig_len + 3] = hex(n);\n+\tset_packet_header(&out->buf[orig_len], n);\n \tpacket_trace(out->buf + orig_len + 4, n - 4, 1);\n }\n \ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3cb9d91..925c6d3 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -23,6 +23,7 @@ void packet_flush(int fd);\n void packet_write(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+void set_packet_header(char *buf, const int size);\n \n /*\n * Read a packetized line into the buffer, which must be at least size bytes\n-- \n2.9.0\n\n"},{"id":"292289","messageId":"20160727000605.49982-5-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160727000605.49982-1-larsxschneider@gmail.com","subject":"[PATCH v2 4/5] convert: generate large test files only once","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T00:06:04Z","receivedAt":"2016-07-27T00:06:33Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nGenerate a more interesting large test file with random characters in\nbetween and reuse this test file in multiple tests. Run tests formerly\nmarked as EXPENSIVE every time but with a smaller test file.\n\nSigned-off-by: Lars Schneider \n---\n t/t0021-conversion.sh | 35 +++++++++++++++++++++++++----------\n 1 file changed, 25 insertions(+), 10 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 7b45136..b9911a4 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -4,6 +4,13 @@ test_description='blob conversion via gitattributes'\n \n . ./test-lib.sh\n \n+if test_have_prereq EXPENSIVE\n+then\n+\tT0021_LARGE_FILE_SIZE=2048\n+else\n+\tT0021_LARGE_FILE_SIZE=30\n+fi\n+\n cat <rot13.sh\n #!$SHELL_PATH\n tr \\\n@@ -31,7 +38,15 @@ test_expect_success setup '\n \tcat test >test.i &&\n \tgit add test test.t test.i &&\n \trm -f test test.t test.i &&\n-\tgit checkout -- test test.t test.i\n+\tgit checkout -- test test.t test.i &&\n+\n+\tmkdir -p generated-test-data &&\n+\tfor i in $(test_seq 1 $T0021_LARGE_FILE_SIZE)\n+\tdo\n+\t\t# Generate 1MB of empty data and 100 bytes of random characters\n+\t\tprintf \"%1048576d\" 1\n+\t\tprintf \"$(LC_ALL=C tr -dc \"A-Za-z0-9\" >8)) count=1 2>/dev/null)\"\n+\tdone >generated-test-data/large.file\n '\n \n script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n@@ -199,9 +214,9 @@ test_expect_success 'required filter clean failure' '\n test_expect_success 'filtering large input to small output should use little memory' '\n \ttest_config filter.devnull.clean \"cat >/dev/null\" &&\n \ttest_config filter.devnull.required true &&\n-\tfor i in $(test_seq 1 30); do printf \"%1048576d\" 1; done >30MB &&\n-\techo \"30MB filter=devnull\" >.gitattributes &&\n-\tGIT_MMAP_LIMIT=1m GIT_ALLOC_LIMIT=1m git add 30MB\n+\tcp generated-test-data/large.file large.file &&\n+\techo \"large.file filter=devnull\" >.gitattributes &&\n+\tGIT_MMAP_LIMIT=1m GIT_ALLOC_LIMIT=1m git add large.file\n '\n \n test_expect_success 'filter that does not read is fine' '\n@@ -214,15 +229,15 @@ test_expect_success 'filter that does not read is fine' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success EXPENSIVE 'filter large file' '\n+test_expect_success 'filter large file' '\n \ttest_config filter.largefile.smudge cat &&\n \ttest_config filter.largefile.clean cat &&\n-\tfor i in $(test_seq 1 2048); do printf \"%1048576d\" 1; done >2GB &&\n-\techo \"2GB filter=largefile\" >.gitattributes &&\n-\tgit add 2GB 2>err &&\n+\techo \"large.file filter=largefile\" >.gitattributes &&\n+\tcp generated-test-data/large.file large.file &&\n+\tgit add large.file 2>err &&\n \ttest_must_be_empty err &&\n-\trm -f 2GB &&\n-\tgit checkout -- 2GB 2>err &&\n+\trm -f large.file &&\n+\tgit checkout -- large.file 2>err &&\n \ttest_must_be_empty err\n '\n \n-- \n2.9.0\n\n"},{"id":"292290","messageId":"20160727000605.49982-6-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160727000605.49982-1-larsxschneider@gmail.com","subject":"[PATCH v2 5/5] convert: add filter..process option","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T00:06:05Z","receivedAt":"2016-07-27T00:06:36Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nGit's clean/smudge mechanism invokes an external filter process for every\nsingle blob that is affected by a filter. If Git filters a lot of blobs\nthen the startup time of the external filter processes can become a\nsignificant part of the overall Git execution time.\n\nThis patch adds the filter..process string option which, if used,\nkeeps the external filter process running and processes all blobs with\nthe following packet format (pkt-line) based protocol over standard input\nand standard output.\n\nGit starts the filter on first usage and expects a welcome\nmessage, protocol version number, and filter capabilities\nseperated by spaces:\n------------------------\npacket: git< git-filter-protocol\npacket: git< version 2\npacket: git< clean smudge\n------------------------\nSupported filter capabilities are \"clean\" and \"smudge\".\n\nAfterwards Git sends a command (e.g. \"smudge\" or \"clean\" - based\non the supported capabilities), the filename, the content size as\nASCII number in bytes, and the content in packet format with a\nflush packet at the end:\n------------------------\npacket: git> smudge\npacket: git> testfile.dat\npacket: git> 7\npacket: git> CONTENT\npacket: git> 0000\n------------------------\n\nThe filter is expected to respond with the result content size as\nASCII number in bytes and the result content in packet format with\na flush packet at the end:\n------------------------\npacket: git< 57\npacket: git< SMUDGED_CONTENT\npacket: git< 0000\n------------------------\nPlease note: In a future version of Git the capability \"stream\"\nmight be supported. In that case the content size must not be\npart of the filter response.\n\nAfterwards the filter is expected to wait for the next command.\n\nSigned-off-by: Lars Schneider \nHelped-by: Martin-Louis Bright \nSigned-off-by: Lars Schneider \n---\n Documentation/gitattributes.txt | 54 +++++++-\n convert.c | 269 ++++++++++++++++++++++++++++++++++++++--\n t/t0021-conversion.sh | 175 ++++++++++++++++++++++++++\n t/t0021/rot13-filter.pl | 146 ++++++++++++++++++++++\n 4 files changed, 631 insertions(+), 13 deletions(-)\n create mode 100755 t/t0021/rot13-filter.pl\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 8882a3e..8fb40d2 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -300,7 +300,11 @@ checkout, when the `smudge` command is specified, the command is\n fed the blob object from its standard input, and its standard\n output is used to update the worktree file. Similarly, the\n `clean` command is used to convert the contents of worktree file\n-upon checkin.\n+upon checkin. By default these commands process only a single\n+blob and terminate. If a long running filter process (see section\n+below) is used then Git can process all blobs with a single filter\n+invocation for the entire life of a single Git command (e.g.\n+`git add .`).\n \n One use of the content filtering is to massage the content into a shape\n that is more convenient for the platform, filesystem, and the user to use.\n@@ -375,6 +379,54 @@ substitution. For example:\n ------------------------\n \n \n+Long Running Filter Process\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+If the filter command (string value) is defined via\n+filter..process then Git can process all blobs with a\n+single filter invocation for the entire life of a single Git\n+command by talking with the following packet format (pkt-line)\n+based protocol over standard input and standard output.\n+\n+Git starts the filter on first usage and expects a welcome\n+message, protocol version number, and filter capabilities\n+seperated by spaces:\n+------------------------\n+packet: git< git-filter-protocol\n+packet: git< version 2\n+packet: git< clean smudge\n+------------------------\n+Supported filter capabilities are \"clean\" and \"smudge\".\n+\n+Afterwards Git sends a command (e.g. \"smudge\" or \"clean\" - based\n+on the supported capabilities), the filename, the content size as\n+ASCII number in bytes, and the content in packet format with a\n+flush packet at the end:\n+------------------------\n+packet: git> smudge\n+packet: git> testfile.dat\n+packet: git> 7\n+packet: git> CONTENT\n+packet: git> 0000\n+------------------------\n+\n+The filter is expected to respond with the result content size as\n+ASCII number in bytes and the result content in packet format with\n+a flush packet at the end:\n+------------------------\n+packet: git< 57\n+packet: git< SMUDGED_CONTENT\n+packet: git< 0000\n+------------------------\n+Please note: In a future version of Git the capability \"stream\"\n+might be supported. In that case the content size must not be\n+part of the filter response.\n+\n+Afterwards the filter is expected to wait for the next command.\n+A demo implementation can be found in `t/t0021/rot13-filter.pl`\n+located in the Git core repository.\n+\n+\n Interaction between checkin/checkout attributes\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \ndiff --git a/convert.c b/convert.c\nindex 522e2c5..5ff200b 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -3,6 +3,7 @@\n #include \"run-command.h\"\n #include \"quote.h\"\n #include \"sigchain.h\"\n+#include \"pkt-line.h\"\n \n /*\n * convert.c - convert a file when checking it out and checking it in.\n@@ -481,11 +482,232 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \treturn ret;\n }\n \n+static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n+{\n+\toff_t bytes_read;\n+\toff_t total_bytes_read = 0;\n+\tstrbuf_grow(sb, size + 1);\t// we need one extra byte for the packet flush\n+\tdo {\n+\t\tbytes_read = packet_read(\n+\t\t\tfd, NULL, NULL,\n+\t\t\tsb->buf + total_bytes_read, sb->len - total_bytes_read - 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF\n+\t\t);\n+\t\ttotal_bytes_read += bytes_read;\n+\t}\n+\twhile (\n+\t\tbytes_read > 0 && \t\t\t\t\t// the last packet was no flush\n+\t\tsb->len - total_bytes_read - 1 > 0 \t// we still have space left in the buffer\n+\t);\n+\tstrbuf_setlen(sb, total_bytes_read);\n+\treturn total_bytes_read;\n+}\n+\n+static int multi_packet_write(const char *src, size_t len, const int in, const int out)\n+{\n+\tint ret = 1;\n+\tchar header[4];\n+\tchar buffer[8192];\n+\toff_t bytes_to_write;\n+\twhile (ret) {\n+\t\tif (in >= 0) {\n+\t\t\tbytes_to_write = xread(in, buffer, sizeof(buffer));\n+\t\t\tif (bytes_to_write < 0)\n+\t\t\t\tret &= 0;\n+\t\t\tsrc = buffer;\n+\t\t} else {\n+\t\t\tbytes_to_write = len > LARGE_PACKET_MAX - 4 ? LARGE_PACKET_MAX - 4 : len;\n+\t\t\tlen -= bytes_to_write;\n+\t\t}\n+\t\tif (!bytes_to_write)\n+\t\t\tbreak;\n+\t\tset_packet_header(header, bytes_to_write + 4);\n+\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n+\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n+\t}\n+\tret &= write_in_full(out, \"0000\", 4) == 4;\n+\treturn ret;\n+}\n+\n+struct cmd2process {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tconst char *cmd;\n+\tint clean;\n+\tint smudge;\n+\tstruct child_process process;\n+};\n+\n+static int cmd2process_cmp(const struct cmd2process *e1,\n+\t\t\t\t\t\t\tconst struct cmd2process *e2,\n+\t\t\t\t\t\t\tconst void *unused)\n+{\n+\treturn strcmp(e1->cmd, e2->cmd);\n+}\n+\n+static struct cmd2process *find_protocol_filter_entry(struct hashmap *hashmap, const char *cmd)\n+{\n+\tstruct cmd2process k;\n+\thashmap_entry_init(&k, strhash(cmd));\n+\tk.cmd = cmd;\n+\treturn hashmap_get(hashmap, &k, NULL);\n+}\n+\n+static void stop_protocol_filter(struct hashmap *hashmap, struct cmd2process *entry) {\n+\tif (!entry)\n+\t\treturn;\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tclose(entry->process.in);\n+\tclose(entry->process.out);\n+\tsigchain_pop(SIGPIPE);\n+\tfinish_command(&entry->process);\n+\tchild_process_clear(&entry->process);\n+\thashmap_remove(hashmap, entry, NULL);\n+\tfree(entry);\n+}\n+\n+static struct cmd2process *start_protocol_filter(struct hashmap *hashmap, const char *cmd)\n+{\n+\tint ret = 1;\n+\tstruct cmd2process *entry;\n+\tstruct child_process *process;\n+\tconst char *argv[] = { NULL, NULL };\n+\tstruct string_list capabilities = STRING_LIST_INIT_NODUP;\n+\tchar *capabilities_buffer;\n+\tint i;\n+\n+\tentry = xmalloc(sizeof(*entry));\n+\thashmap_entry_init(entry, strhash(cmd));\n+\tentry->cmd = cmd;\n+\tentry->clean = 0;\n+\tentry->smudge = 0;\n+\tprocess = &entry->process;\n+\n+\tchild_process_init(process);\n+\targv[0] = cmd;\n+\tprocess->argv = argv;\n+\tprocess->use_shell = 1;\n+\tprocess->in = -1;\n+\tprocess->out = -1;\n+\n+\tif (start_command(process)) {\n+\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n+\t\tstop_protocol_filter(hashmap, entry);\n+\t\treturn NULL;\n+\t}\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tret &= strcmp(packet_read_line(process->out, NULL), \"git-filter-protocol\") == 0;\n+\tret &= strcmp(packet_read_line(process->out, NULL), \"version 2\") == 0;\n+\tcapabilities_buffer = packet_read_line(process->out, NULL);\n+\tsigchain_pop(SIGPIPE);\n+\n+\tstring_list_split_in_place(&capabilities, capabilities_buffer, ' ', -1);\n+\tfor (i = 0; i < capabilities.nr; i++) {\n+\t\tconst char *requested = capabilities.items[i].string;\n+\t\tif (!strcmp(requested, \"clean\")) {\n+\t\t\tentry->clean = 1;\n+\t\t} else if (!strcmp(requested, \"smudge\")) {\n+\t\t\tentry->smudge = 1;\n+\t\t} else {\n+\t\t\twarning(\n+\t\t\t\t\"filter process '%s' requested unsupported filter capability '%s'\",\n+\t\t\t\tcmd, requested\n+\t\t\t);\n+\t\t}\n+\t}\n+\tstring_list_clear(&capabilities, 0);\n+\n+\tif (!ret) {\n+\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n+\t\tstop_protocol_filter(hashmap, entry);\n+\t\treturn NULL;\n+\t}\n+\n+\thashmap_add(hashmap, entry);\n+\treturn entry;\n+}\n+\n+static int cmd_process_map_init = 0;\n+static struct hashmap cmd_process_map;\n+\n+static int apply_protocol_filter(const char *path, const char *src, size_t len,\n+\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n+\t\t\t\t\t\tconst char *filter_type)\n+{\n+\tint ret = 1;\n+\tstruct cmd2process *entry;\n+\tstruct child_process *process;\n+\tstruct stat file_stat;\n+\tstruct strbuf nbuf = STRBUF_INIT;\n+\toff_t expected_bytes;\n+\tchar *strtol_end;\n+\tchar *strbuf;\n+\n+\tif (!cmd || !*cmd)\n+\t\treturn 0;\n+\n+\tif (!dst)\n+\t\treturn 1;\n+\n+\tif (!cmd_process_map_init) {\n+\t\tcmd_process_map_init = 1;\n+\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n+\t\tentry = NULL;\n+\t} else {\n+\t\tentry = find_protocol_filter_entry(&cmd_process_map, cmd);\n+\t}\n+\n+\tif (!entry) {\n+\t\tentry = start_protocol_filter(&cmd_process_map, cmd);\n+\t\tif (!entry) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\tprocess = &entry->process;\n+\n+\tif (!(!strcmp(filter_type, \"clean\") && entry->clean) &&\n+\t\t!(!strcmp(filter_type, \"smudge\") && entry->smudge)) {\n+\t\treturn 0;\n+\t}\n+\n+\tif (fd >= 0 && !src) {\n+\t\tret &= fstat(fd, &file_stat) != -1;\n+\t\tlen = file_stat.st_size;\n+\t}\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\tpacket_write(process->in, \"%s\\n\", filter_type);\n+\tpacket_write(process->in, \"%s\\n\", path);\n+\tpacket_write(process->in, \"%zu\\n\", len);\n+\tret &= multi_packet_write(src, len, fd, process->in);\n+\n+\tstrbuf = packet_read_line(process->out, NULL);\n+\texpected_bytes = (off_t)strtol(strbuf, &strtol_end, 10);\n+\tret &= (strtol_end != strbuf && errno != ERANGE);\n+\n+\tif (expected_bytes > 0) {\n+\t\tret &= multi_packet_read(&nbuf, process->out, expected_bytes) == expected_bytes;\n+\t}\n+\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (ret) {\n+\t\tstrbuf_swap(dst, &nbuf);\n+\t} else {\n+\t\t// Something went wrong with the protocol filter. Force shutdown!\n+\t\tstop_protocol_filter(&cmd_process_map, entry);\n+\t}\n+\tstrbuf_release(&nbuf);\n+\treturn ret;\n+}\n+\n static struct convert_driver {\n \tconst char *name;\n \tstruct convert_driver *next;\n \tconst char *smudge;\n \tconst char *clean;\n+\tconst char *process;\n \tint required;\n } *user_convert, **user_convert_tail;\n \n@@ -526,6 +748,10 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(\"clean\", key))\n \t\treturn git_config_string(&drv->clean, var, value);\n \n+\tif (!strcmp(\"process\", key)) {\n+\t\treturn git_config_string(&drv->process, var, value);\n+\t}\n+\n \tif (!strcmp(\"required\", key)) {\n \t\tdrv->required = git_config_bool(var, value);\n \t\treturn 0;\n@@ -823,7 +1049,10 @@ int would_convert_to_git_filter_fd(const char *path)\n \tif (!ca.drv->required)\n \t\treturn 0;\n \n-\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n+\tif (!ca.drv->clean && ca.drv->process)\n+\t\treturn apply_protocol_filter(path, NULL, 0, -1, NULL, ca.drv->process, \"clean\");\n+\telse\n+\t\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n }\n \n const char *get_convert_attr_ascii(const char *path)\n@@ -856,17 +1085,22 @@ int convert_to_git(const char *path, const char *src, size_t len,\n struct strbuf *dst, enum safe_crlf checksafe)\n {\n \tint ret = 0;\n-\tconst char *filter = NULL;\n+\tconst char *clean_filter = NULL;\n+\tconst char *process_filter = NULL;\n \tint required = 0;\n \tstruct conv_attrs ca;\n \n \tconvert_attrs(&ca, path);\n \tif (ca.drv) {\n-\t\tfilter = ca.drv->clean;\n+\t\tclean_filter = ca.drv->clean;\n+\t\tprocess_filter = ca.drv->process;\n \t\trequired = ca.drv->required;\n \t}\n \n-\tret |= apply_filter(path, src, len, -1, dst, filter);\n+\tif (!clean_filter && process_filter)\n+\t\tret |= apply_protocol_filter(path, src, len, -1, dst, process_filter, \"clean\");\n+\telse\n+\t\tret |= apply_filter(path, src, len, -1, dst, clean_filter);\n \tif (!ret && required)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n@@ -885,13 +1119,19 @@ int convert_to_git(const char *path, const char *src, size_t len,\n void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n \t\t\t enum safe_crlf checksafe)\n {\n+\tint ret = 0;\n \tstruct conv_attrs ca;\n \tconvert_attrs(&ca, path);\n \n \tassert(ca.drv);\n-\tassert(ca.drv->clean);\n+\tassert(ca.drv->clean || ca.drv->process);\n \n-\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))\n+\tif (!ca.drv->clean && ca.drv->process)\n+\t\tret = apply_protocol_filter(path, NULL, 0, fd, dst, ca.drv->process, \"clean\");\n+\telse\n+\t\tret = apply_filter(path, NULL, 0, fd, dst, ca.drv->clean);\n+\n+\tif (!ret)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n@@ -902,14 +1142,16 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t\t int normalizing)\n {\n-\tint ret = 0, ret_filter = 0;\n-\tconst char *filter = NULL;\n+\tint ret = 0, ret_filter;\n+\tconst char *smudge_filter = NULL;\n+\tconst char *process_filter = NULL;\n \tint required = 0;\n \tstruct conv_attrs ca;\n \n \tconvert_attrs(&ca, path);\n \tif (ca.drv) {\n-\t\tfilter = ca.drv->smudge;\n+\t\tprocess_filter = ca.drv->process;\n+\t\tsmudge_filter = ca.drv->smudge;\n \t\trequired = ca.drv->required;\n \t}\n \n@@ -922,7 +1164,7 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t * CRLF conversion can be skipped if normalizing, unless there\n \t * is a smudge filter. The filter might expect CRLFs.\n \t */\n-\tif (filter || !normalizing) {\n+\tif (smudge_filter || process_filter || !normalizing) {\n \t\tret |= crlf_to_worktree(path, src, len, dst, ca.crlf_action);\n \t\tif (ret) {\n \t\t\tsrc = dst->buf;\n@@ -930,7 +1172,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n-\tret_filter = apply_filter(path, src, len, -1, dst, filter);\n+\tif (!smudge_filter && process_filter)\n+\t\tret_filter = apply_protocol_filter(path, src, len, -1, dst, process_filter, \"smudge\");\n+\telse\n+\t\tret_filter = apply_filter(path, src, len, -1, dst, smudge_filter);\n \tif (!ret_filter && required)\n \t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n \n@@ -1383,7 +1628,7 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n \tstruct stream_filter *filter = NULL;\n \n \tconvert_attrs(&ca, path);\n-\tif (ca.drv && (ca.drv->smudge || ca.drv->clean))\n+\tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n \t\treturn NULL;\n \n \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex b9911a4..c4793ed 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -4,6 +4,11 @@ test_description='blob conversion via gitattributes'\n \n . ./test-lib.sh\n \n+if ! test_have_prereq PERL; then\n+\tskip_all='skipping perl interface tests, perl not available'\n+\ttest_done\n+fi\n+\n if test_have_prereq EXPENSIVE\n then\n \tT0021_LARGE_FILE_SIZE=2048\n@@ -283,4 +288,174 @@ test_expect_success 'disable filter with empty override' '\n \ttest_must_be_empty err\n '\n \n+test_expect_success 'required protocol filter should filter data' '\n+\ttest_config_global filter.protocol.process \\\"$TEST_DIRECTORY/t0021/rot13-filter.pl\\\" &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\tgit branch empty &&\n+\n+\t\tcat ../test.o >test.r &&\n+\t\techo \"test22\" >test2.r &&\n+\t\techo \"test333\" >test3.r &&\n+\n+\t\trm -f output.log &&\n+\t\tgit add . &&\n+\t\tsort output.log | uniq -c | sed \"s/^[ ]*//\" >uniq_output.log &&\n+\t\tcat >expected_add.log <<-\\EOF &&\n+\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 [OK]\n+\t\t\t1 IN: clean test2.r 7 [OK] -- OUT: 7 [OK]\n+\t\t\t1 IN: clean test3.r 8 [OK] -- OUT: 8 [OK]\n+\t\t\t1 start\n+\t\t\t1 wrote filter header\n+\t\tEOF\n+\t\ttest_cmp expected_add.log uniq_output.log &&\n+\n+\t\t>output.log &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\tsort output.log | uniq -c | sed \"s/^[ ]*//\" |\n+\t\t\tsed \"s/^\\([0-9]\\) IN: clean/x IN: clean/\" >uniq_output.log &&\n+\t\tcat >expected_commit.log <<-\\EOF &&\n+\t\t\tx IN: clean test.r 57 [OK] -- OUT: 57 [OK]\n+\t\t\tx IN: clean test2.r 7 [OK] -- OUT: 7 [OK]\n+\t\t\tx IN: clean test3.r 8 [OK] -- OUT: 8 [OK]\n+\t\t\t1 start\n+\t\t\t1 wrote filter header\n+\t\tEOF\n+\t\ttest_cmp expected_commit.log uniq_output.log &&\n+\n+\t\t>output.log &&\n+\t\trm -f test?.r &&\n+\t\tgit checkout . &&\n+\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n+\t\tcat >expected_checkout.log <<-\\EOF &&\n+\t\t\tstart\n+\t\t\twrote filter header\n+\t\t\tIN: smudge test2.r 7 [OK] -- OUT: 7 [OK]\n+\t\t\tIN: smudge test3.r 8 [OK] -- OUT: 8 [OK]\n+\t\tEOF\n+\t\ttest_cmp expected_checkout.log smudge_output.log &&\n+\n+\t\tgit checkout empty &&\n+\n+\t\t>output.log &&\n+\t\tgit checkout master &&\n+\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n+\t\tcat >expected_checkout_master.log <<-\\EOF &&\n+\t\t\tstart\n+\t\t\twrote filter header\n+\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 [OK]\n+\t\t\tIN: smudge test2.r 7 [OK] -- OUT: 7 [OK]\n+\t\t\tIN: smudge test3.r 8 [OK] -- OUT: 8 [OK]\n+\t\tEOF\n+\t\ttest_cmp expected_checkout_master.log smudge_output.log\n+\t)\n+'\n+\n+test_expect_success 'protocol filter large file' '\n+\ttest_config_global filter.protocol.process \\\"$TEST_DIRECTORY/t0021/rot13-filter.pl\\\" &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.file filter=protocol\" >.gitattributes &&\n+\t\tcp ../generated-test-data/large.file large.file &&\n+\t\tcp large.file large.original &&\n+\t\t./../rot13.sh large.rot13 &&\n+\n+\t\tgit add large.file .gitattributes &&\n+\t\tgit commit . -m \"test commit\" &&\n+\n+\t\trm -f large.file &&\n+\t\tgit checkout -- large.file &&\n+\t\tgit cat-file blob :large.file >actual &&\n+\t\ttest_cmp large.rot13 actual\n+\t)\n+'\n+\n+test_expect_success 'required protocol filter should fail with clean' '\n+\ttest_config_global filter.protocol.process \\\"$TEST_DIRECTORY/t0021/rot13-filter.pl\\\" &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcat ../test.o >test.r &&\n+\t\techo \"this is going to fail\" >clean-write-fail.r &&\n+\t\techo \"test333\" >test3.r &&\n+\n+\t\t# Note: There are three clean paths in convert.c we just test one here.\n+\t\ttest_must_fail git add .\n+\t)\n+'\n+\n+test_expect_success 'protocol filter should restart after failure' '\n+\ttest_config_global filter.protocol.process \\\"$TEST_DIRECTORY/t0021/rot13-filter.pl\\\" &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcat ../test.o >test.r &&\n+\t\techo \"1234567\" >test2.o &&\n+\t\tcat test2.o >test2.r &&\n+\t\techo \"this is going to fail\" >smudge-write-fail.o &&\n+\t\tcat smudge-write-fail.o >smudge-write-fail.r &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\trm -f *.r &&\n+\n+\t\tprintf \"\" >output.log &&\n+\t\tgit checkout . &&\n+\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n+\t\tcat >expected_checkout_master.log <<-\\EOF &&\n+\t\t\tstart\n+\t\t\twrote filter header\n+\t\t\tIN: smudge smudge-write-fail.r 22 [OK] -- OUT: 22 [FAIL]\n+\t\t\tstart\n+\t\t\twrote filter header\n+\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 [OK]\n+\t\t\tIN: smudge test2.r 8 [OK] -- OUT: 8 [OK]\n+\t\tEOF\n+\t\ttest_cmp expected_checkout_master.log smudge_output.log &&\n+\n+\t\ttest_cmp ../test.o test.r &&\n+\t\t./../rot13.sh <../test.o >expected &&\n+\t\tgit cat-file blob :test.r >actual &&\n+\t\ttest_cmp expected actual\n+\n+\t\ttest_cmp test2.o test2.r &&\n+\t\t./../rot13.sh expected &&\n+\t\tgit cat-file blob :test2.r >actual &&\n+\t\ttest_cmp expected actual\n+\n+\t\ttest_cmp test2.o test2.r &&\n+\t\t./../rot13.sh expected &&\n+\t\tgit cat-file blob :test2.r >actual &&\n+\t\ttest_cmp expected actual\n+\n+\t\t! test_cmp smudge-write-fail.o smudge-write-fail.r && # Smudge failed!\n+\t\t./../rot13.sh expected &&\n+\t\tgit cat-file blob :smudge-write-fail.r >actual &&\n+\t\ttest_cmp expected actual\t\t\t\t\t\t\t # Clean worked!\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nnew file mode 100755\nindex 0000000..7176836\n--- /dev/null\n+++ b/t/t0021/rot13-filter.pl\n@@ -0,0 +1,146 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git filter protocol version 2\n+# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n+#\n+# This implementation supports two special test cases:\n+# (1) If data with the filename \"clean-write-fail.r\" is processed with\n+# a \"clean\" operation then the write operation will die.\n+# (2) If data with the filename \"smudge-write-fail.r\" is processed with\n+# a \"smudge\" operation then the write operation will die.\n+#\n+\n+use strict;\n+use warnings;\n+\n+my $MAX_PACKET_CONTENT_SIZE = 65516;\n+\n+sub rot13 {\n+ my ($str) = @_;\n+ $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n+ return $str;\n+}\n+\n+sub packet_read {\n+ my $buffer;\n+ my $bytes_read = read STDIN, $buffer, 4;\n+ if ( $bytes_read == 0 ) {\n+ return;\n+ }\n+ elsif ( $bytes_read != 4 ) {\n+ die \"invalid packet size '$bytes_read' field\";\n+ }\n+ my $pkt_size = hex($buffer);\n+ if ( $pkt_size == 0 ) {\n+ return ( 1, \"\" );\n+ }\n+ elsif ( $pkt_size > 4 ) {\n+ my $content_size = $pkt_size - 4;\n+ $bytes_read = read STDIN, $buffer, $content_size;\n+ if ( $bytes_read != $content_size ) {\n+ die \"invalid packet\";\n+ }\n+ return ( 0, $buffer );\n+ }\n+ else {\n+ die \"invalid packet size\";\n+ }\n+}\n+\n+sub packet_write {\n+ my ($packet) = @_;\n+ print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n+ print STDOUT $packet;\n+ STDOUT->flush();\n+}\n+\n+sub packet_flush {\n+ print STDOUT sprintf( \"%04x\", 0 );\n+ STDOUT->flush();\n+}\n+\n+open my $debug, \">>\", \"output.log\";\n+print $debug \"start\\n\";\n+$debug->flush();\n+\n+packet_write(\"git-filter-protocol\\n\");\n+packet_write(\"version 2\\n\");\n+packet_write(\"clean smudge\\n\");\n+print $debug \"wrote filter header\\n\";\n+$debug->flush();\n+\n+while (1) {\n+ my $command = packet_read();\n+ unless ( defined($command) ) {\n+ exit();\n+ }\n+ chomp $command;\n+ print $debug \"IN: $command\";\n+ $debug->flush();\n+ my $filename = packet_read();\n+ chomp $filename;\n+ print $debug \" $filename\";\n+ $debug->flush();\n+ my $filelen = packet_read();\n+ chomp $filelen;\n+ print $debug \" $filelen\";\n+ $debug->flush();\n+\n+ $filelen =~ /\\A\\d+\\z/ or die \"bad filelen: $filelen\";\n+ my $output;\n+\n+ if ( $filelen > 0 ) {\n+ my $input = \"\";\n+ {\n+ binmode(STDIN);\n+ my $buffer;\n+ my $done = 0;\n+ while ( !$done ) {\n+ ( $done, $buffer ) = packet_read();\n+ $input .= $buffer;\n+ }\n+ print $debug \" [OK] -- \";\n+ $debug->flush();\n+ }\n+\n+ if ( $command eq \"clean\" ) {\n+ $output = rot13($input);\n+ }\n+ elsif ( $command eq \"smudge\" ) {\n+ $output = rot13($input);\n+ }\n+ else {\n+ die \"bad command\";\n+ }\n+ }\n+\n+ my $output_len = length($output);\n+ packet_write(\"$output_len\\n\");\n+ print $debug \"OUT: $output_len \";\n+ $debug->flush();\n+ if ( $output_len > 0 ) {\n+ if ( ( $command eq \"clean\" and $filename eq \"clean-write-fail.r\" )\n+ or\n+ ( $command eq \"smudge\" and $filename eq \"smudge-write-fail.r\" ) )\n+ {\n+ print $debug \" [FAIL]\\n\";\n+ $debug->flush();\n+ die \"write error\";\n+ }\n+ else {\n+ while ( length($output) > 0 ) {\n+ my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n+ packet_write($packet);\n+ if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n+ $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n+ }\n+ else {\n+ $output = \"\";\n+ }\n+ }\n+ packet_flush();\n+ print $debug \"[OK]\\n\";\n+ $debug->flush();\n+ }\n+ }\n+}\n-- \n2.9.0\n\n"},{"id":"292291","messageId":"20160727000605.49982-2-larsxschneider@gmail.com","threadId":"42904","inReplyTo":"20160727000605.49982-1-larsxschneider@gmail.com","subject":"[PATCH v2 1/5] convert: quote filter names in error messages","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T00:06:01Z","receivedAt":"2016-07-27T00:07:58Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nGit filter with spaces (e.g. `filter.sh foo`) are hard to read in\nerror messages. Quote them to improve the readability.\n\nSigned-off-by: Lars Schneider \n---\n convert.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex b1614bf..522e2c5 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -397,7 +397,7 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \tchild_process.out = out;\n \n \tif (start_command(&child_process))\n-\t\treturn error(\"cannot fork to run external filter %s\", params->cmd);\n+\t\treturn error(\"cannot fork to run external filter '%s'\", params->cmd);\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n@@ -415,13 +415,13 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \tif (close(child_process.in))\n \t\twrite_err = 1;\n \tif (write_err)\n-\t\terror(\"cannot feed the input to external filter %s\", params->cmd);\n+\t\terror(\"cannot feed the input to external filter '%s'\", params->cmd);\n \n \tsigchain_pop(SIGPIPE);\n \n \tstatus = finish_command(&child_process);\n \tif (status)\n-\t\terror(\"external filter %s failed %d\", params->cmd, status);\n+\t\terror(\"external filter '%s' failed %d\", params->cmd, status);\n \n \tstrbuf_release(&cmd);\n \treturn (write_err || status);\n@@ -462,15 +462,15 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \t\treturn 0;\t/* error was already reported */\n \n \tif (strbuf_read(&nbuf, async.out, len) < 0) {\n-\t\terror(\"read from external filter %s failed\", cmd);\n+\t\terror(\"read from external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \tif (close(async.out)) {\n-\t\terror(\"read from external filter %s failed\", cmd);\n+\t\terror(\"read from external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \tif (finish_async(&async)) {\n-\t\terror(\"external filter %s failed\", cmd);\n+\t\terror(\"external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \n-- \n2.9.0\n\n"},{"id":"292292","messageId":"xmqqinvsx77w.fsf@gitster.mtv.corp.google.com","threadId":"42904","inReplyTo":"20160727000605.49982-4-larsxschneider@gmail.com","subject":"Re: [PATCH v2 3/5] pkt-line: extract and use `set_packet_header` function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-27T00:20:03Z","receivedAt":"2016-07-27T00:20:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"larsxschneider@gmail.com writes:\n\n> From: Lars Schneider \n>\n> `set_packet_header` converts an integer to a 4 byte hex string. Make\n> this function publicly available so that other parts of Git can easily\n> generate a pkt-line.\n\nI think that having to do this is a strong sign that the design of\nthis series is going in a wrong direction.\n\nIf you need a helper function that writes a pkt-line format that\nbehaves differently from what is already available (for example,\npacket_write()), it would be much better to design that new function\nso that it would be generally useful and add that to pkt-line.[ch],\ninstead of creating random helper functions that use write(2)\ndirectly, bypassing pkt-line API, to write stuff.\n\nIn other words, do not _mimick_ pkt-line; enhance pkt-line as\nnecessary and use it.\n\n"},{"id":"292294","messageId":"20160727013251.GA12159@sigill.intra.peff.net","threadId":"42904","inReplyTo":"20160727000605.49982-6-larsxschneider@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-27T01:32:51Z","receivedAt":"2016-07-27T01:33:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 27, 2016 at 02:06:05AM +0200, larsxschneider@gmail.com wrote:\n\n> +static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n> +{\n> +\toff_t bytes_read;\n> +\toff_t total_bytes_read = 0;\n\nI haven't looked carefully at the whole patch yet, but there seems to be\nsome type issues here. off_t is a good type for storing the whole size\nof a file (which may be larger than the amount of memory we can\nallocate). But size_t is the right size for an in-memory object.\n\nThis function takes a size_t size, which makes sense if it is meant to\nread everything into a strbuf.\n\nSo I think our total_bytes_read would probably want to be a size_t here,\ntoo, because it cannot possibly grow larger than that (and that is\nenforced by the loop below). Otherwise you get weirdness like \"sb->buf +\ntotal_bytes_ref\" possibly overflowing memory.\n\n> +\tstrbuf_grow(sb, size + 1);\t// we need one extra byte for the packet flush\n\nWhat happens if size is the maximum for size_t here (i.e., 4GB-1 on a\n32-bit system)?\n\n> +\tdo {\n> +\t\tbytes_read = packet_read(\n> +\t\t\tfd, NULL, NULL,\n> +\t\t\tsb->buf + total_bytes_read, sb->len - total_bytes_read - 1,\n> +\t\t\tPACKET_READ_GENTLE_ON_EOF\n> +\t\t);\n\npacket_read() actually returns an int, and may return \"-1\" on EOF (and\nint is fine because we know that we are constrained to 16-bit values\nby the pkt-line definition). You read it into an \"off_t\". I _think_ that\nis OK, because I believe POSIX says off_t must be signed. But probably\n\"int\" is the more correct type here.\n\n> +\t\ttotal_bytes_read += bytes_read;\n\nIf you do get \"-1\", I think you need to detect it here before adjusting\ntotal_bytes_read.\n\n> +\twhile (\n> +\t\tbytes_read > 0 && \t\t\t\t\t// the last packet was no flush\n> +\t\tsb->len - total_bytes_read - 1 > 0 \t// we still have space left in the buffer\n> +\t);\n\nAnd I'm not sure if you need to distinguish between \"0\" and \"-1\" when\nchecking byte_read here.\n\n> +\tstrbuf_setlen(sb, total_bytes_read);\n\nPassing an off_t to something expecting a size_t, which can involve\ntruncation (though I think in practice you really are limited to\nsize_t).\n\n> +static int multi_packet_write(const char *src, size_t len, const int in, const int out)\n> +{\n> +\tint ret = 1;\n> +\tchar header[4];\n> +\tchar buffer[8192];\n> +\toff_t bytes_to_write;\n> +\twhile (ret) {\n> +\t\tif (in >= 0) {\n> +\t\t\tbytes_to_write = xread(in, buffer, sizeof(buffer));\n\nLikewise here, xread() is returning ssize_t. Again, OK if we can assume\noff_t is signed, but it probably makes sense to use the correct type (we\nalso know it cannot be larger than 8K, of course).\n\nWhy 8K? The pkt-line format naturally restricts us to just under 64K, so\nwhy not take advantage of that and minimize the framing overhead for\nlarge data?\n\n> +\t\t\tif (bytes_to_write < 0)\n> +\t\t\t\tret &= 0;\n\nI think \"&= 0\" is unusual for our codebase? Would just writing \"= 0\" be\nmore clear?\n\nWe do sometimes do \"ret |= something()\" but that is in cases where\n\"ret\" is zero for success, and non-zero (usually -1) otherwise. Perhaps\nyour function's error-reporting is inverted from our usual style?\n\n> +\t\tset_packet_header(header, bytes_to_write + 4);\n> +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n> +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n> +\t}\n\nIf you look at format_packet(), it pulls a slight trick: we have a\nbuffer 4 bytes larger than we need, format into \"buf + 4\", and then\nwrite the final size at the beginning. That lets us write() it all in\none go.\n\nAt first I thought this function was simply reinventing packet_write(),\nbut I guess you are trying to avoid the extra copy of the data (once\ninto the buffer from xread, and then again via format_packet just to add\nthe extra bytes at the beginning).\n\nI agree with what Junio said elsewhere, that there may be a way to make\nthe pkt-line code handle this zero-copy situation better. Perhaps\nsomething like:\n\n struct pktline {\n\t/* first 4 bytes are reserved for length header */\n\tchar buf[LARGE_PACKET_MAX];\n };\n #define PKTLINE_DATA_START(pkt) ((pkt)->buf + 4)\n #define PKTLINE_DATA_LEN (LARGE_PACKET_MAX - 4)\n\n ...\n struct pktline pkt;\n ssize_t len = xread(fd, PKTLINE_DATA_START(&pkt), PKTLINE_DATA_LEN);\n packet_send(&pkt, len);\n\nThen packet_send() knows that the first 4 bytes are reserved for it. I\nsuspect that the strbuf used by format_packet() could get away with\nusing such a \"struct pktline\" too, though in practice I doubt there's\nany real efficiency to be gained (we generally reuse the same strbuf\nover and over, so it will grow once to 64K and get reused).\n\n> +\tret &= write_in_full(out, \"0000\", 4) == 4;\n\npacket_flush() ?\n\nI know the packet functions are keen on write_or_die() versus\nwrite_in_full(). That is perhaps something that should be fixed.\n\nThis was just supposed to be a short note about off_t before eating\ndinner (oops), so I didn't read past here.\n\n-Peff\n"},{"id":"292295","messageId":"c3d7466b-cd3f-9f62-bc82-9f77b4f3d1a4@web.de","threadId":"42904","inReplyTo":"20160727000605.49982-5-larsxschneider@gmail.com","subject":"Re: [PATCH v2 4/5] convert: generate large test files only once","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-07-27T02:35:32Z","receivedAt":"2016-07-27T02:38:17Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\nOn 07/27/2016 02:06 AM, larsxschneider@gmail.com wrote:\n> From: Lars Schneider \n>\n> Generate a more interesting large test file with random characters in\n> between and reuse this test file in multiple tests. Run tests formerly\n> marked as EXPENSIVE every time but with a smaller test file.\n>\n> Signed-off-by: Lars Schneider \n> ---\n> t/t0021-conversion.sh | 35 +++++++++++++++++++++++++----------\n> 1 file changed, 25 insertions(+), 10 deletions(-)\n>\n> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n> index 7b45136..b9911a4 100755\n> --- a/t/t0021-conversion.sh\n> +++ b/t/t0021-conversion.sh\n> @@ -4,6 +4,13 @@ test_description='blob conversion via gitattributes'\n>\n> . ./test-lib.sh\n>\n> +if test_have_prereq EXPENSIVE\n> +then\n> +\tT0021_LARGE_FILE_SIZE=2048\n> +else\n> +\tT0021_LARGE_FILE_SIZE=30\n> +fi\n> +\n> cat <rot13.sh\n> #!$SHELL_PATH\n> tr \\\n> @@ -31,7 +38,15 @@ test_expect_success setup '\n> \tcat test >test.i &&\n> \tgit add test test.t test.i &&\n> \trm -f test test.t test.i &&\n> -\tgit checkout -- test test.t test.i\n> +\tgit checkout -- test test.t test.i &&\n> +\n> +\tmkdir -p generated-test-data &&\n> +\tfor i in $(test_seq 1 $T0021_LARGE_FILE_SIZE)\n> +\tdo\n> +\t\t# Generate 1MB of empty data and 100 bytes of random characters\n> +\t\tprintf \"%1048576d\" 1\n> +\t\tprintf \"$(LC_ALL=C tr -dc \"A-Za-z0-9\" >8)) count=1 2>/dev/null)\"\nI'm not sure how portable /dev/urandom is.\nThe other thing, that \"really random\" numbers are an overkill, and\nit may be easier to use pre-defined numbers,\n\nThe rest of 1..4 looks good, I will look at 5/5 later.\n\n> +\tdone >generated-test-data/large.file\n> '\n>\n> script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n> @@ -199,9 +214,9 @@ test_expect_success 'required filter clean failure' '\n> test_expect_success 'filtering large input to small output should use little memory' '\n> \ttest_config filter.devnull.clean \"cat >/dev/null\" &&\n> \ttest_config filter.devnull.required true &&\n> -\tfor i in $(test_seq 1 30); do printf \"%1048576d\" 1; done >30MB &&\n> -\techo \"30MB filter=devnull\" >.gitattributes &&\n> -\tGIT_MMAP_LIMIT=1m GIT_ALLOC_LIMIT=1m git add 30MB\n> +\tcp generated-test-data/large.file large.file &&\n> +\techo \"large.file filter=devnull\" >.gitattributes &&\n> +\tGIT_MMAP_LIMIT=1m GIT_ALLOC_LIMIT=1m git add large.file\n> '\n>\n> test_expect_success 'filter that does not read is fine' '\n> @@ -214,15 +229,15 @@ test_expect_success 'filter that does not read is fine' '\n> \ttest_cmp expect actual\n> '\n>\n> -test_expect_success EXPENSIVE 'filter large file' '\n> +test_expect_success 'filter large file' '\n> \ttest_config filter.largefile.smudge cat &&\n> \ttest_config filter.largefile.clean cat &&\n> -\tfor i in $(test_seq 1 2048); do printf \"%1048576d\" 1; done >2GB &&\n> -\techo \"2GB filter=largefile\" >.gitattributes &&\n> -\tgit add 2GB 2>err &&\n> +\techo \"large.file filter=largefile\" >.gitattributes &&\n> +\tcp generated-test-data/large.file large.file &&\n> +\tgit add large.file 2>err &&\n> \ttest_must_be_empty err &&\n> -\trm -f 2GB &&\n> -\tgit checkout -- 2GB 2>err &&\n> +\trm -f large.file &&\n> +\tgit checkout -- large.file 2>err &&\n> \ttest_must_be_empty err\n> '\n>\n>\n"},{"id":"292303","messageId":"A6E129CE-26F9-456A-B9C7-A502245DD19E@gmail.com","threadId":"42904","inReplyTo":"xmqqinvsx77w.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 3/5] pkt-line: extract and use `set_packet_header` function","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T09:13:26Z","receivedAt":"2016-07-27T09:13:34Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Jul 2016, at 02:20, Junio C Hamano wrote:\n> \n> larsxschneider@gmail.com writes:\n> \n>> From: Lars Schneider \n>> \n>> `set_packet_header` converts an integer to a 4 byte hex string. Make\n>> this function publicly available so that other parts of Git can easily\n>> generate a pkt-line.\n> \n> I think that having to do this is a strong sign that the design of\n> this series is going in a wrong direction.\n\nThanks for the feedback. Do you think using \"pkt-line\" is a move into\nthe wrong direction in general or do you think only my usage of \n\"pkt-line\" is not ideal?\n\n\n> If you need a helper function that writes a pkt-line format that\n> behaves differently from what is already available (for example,\n> packet_write()), it would be much better to design that new function\n> so that it would be generally useful and add that to pkt-line.[ch],\n> instead of creating random helper functions that use write(2)\n> directly, bypassing pkt-line API, to write stuff.\n> \n> In other words, do not _mimick_ pkt-line; enhance pkt-line as\n> necessary and use it.\n\nOK, I understand your argument. If we agree on the \"pkt-line\" usage\nthen I will address this issue.\n\nThanks,\nLars\n"},{"id":"292305","messageId":"20160727094102.GA31374@starla","threadId":"42904","inReplyTo":"20160727000605.49982-6-larsxschneider@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-27T09:41:02Z","receivedAt":"2016-07-27T09:41:09Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"larsxschneider@gmail.com wrote:\n> +static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n\nI'm no expert in C, but this might be const-correctness taken\ntoo far. I think basing this on the read(2) prototype is less\nsurprising:\n\n static ssize_t multi_packet_read(int fd, struct strbuf *sb, size_t size)\n\nAlso what Jeff said about off_t vs size_t, but my previous\nemails may have confused you w.r.t. off_t usage...\n\n> +static int multi_packet_write(const char *src, size_t len, const int in, const int out)\n\nSame comment about over const ints above.\nlen can probably be off_t based on what is below; but you need\nto process the loop in ssize_t-friendly chunks.\n\n> +{\n> +\tint ret = 1;\n> +\tchar header[4];\n> +\tchar buffer[8192];\n> +\toff_t bytes_to_write;\n\nWhat Jeff said, this should be ssize_t to match read(2) and xread\n\n> +\twhile (ret) {\n> +\t\tif (in >= 0) {\n> +\t\t\tbytes_to_write = xread(in, buffer, sizeof(buffer));\n> +\t\t\tif (bytes_to_write < 0)\n> +\t\t\t\tret &= 0;\n> +\t\t\tsrc = buffer;\n> +\t\t} else {\n> +\t\t\tbytes_to_write = len > LARGE_PACKET_MAX - 4 ? LARGE_PACKET_MAX - 4 : len;\n> +\t\t\tlen -= bytes_to_write;\n> +\t\t}\n> +\t\tif (!bytes_to_write)\n> +\t\t\tbreak;\n\nThe whole ret &= .. style error handling is hard-to-follow and\nhere, a source of bugs. I think the expected convention on\nhitting errors is:\n\n\t1) stop whatever you're doing\n\t2) cleanup\n\t3) propagate the error to callers\n\n\"goto\" is an acceptable way of accomplishing this.\n\nFor example, byte_to_write may still be negative at this point\n(and interpreted as a really big number when cast to unsigned\nsize_t) and src/buffer could be stack garbage.\n\n> +\t\tset_packet_header(header, bytes_to_write + 4);\n> +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n> +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n> +\t}\n> +\tret &= write_in_full(out, \"0000\", 4) == 4;\n> +\treturn ret;\n> +}\n> +\n\n> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n> +\t\t\t\t\t\tconst char *filter_type)\n> +{\n\n\n\n> +\tif (fd >= 0 && !src) {\n> +\t\tret &= fstat(fd, &file_stat) != -1;\n> +\t\tlen = file_stat.st_size;\n\nSame truncation bug I noticed earlier; what I originally meant\nis the `len' arg probably ought to be off_t, here, not size_t.\n32-bit x86 Linux systems have 32-bit size_t (unsigned), but\nlarge file support means off_t is 64-bits (signed).\n\nAlso, is it worth continuing this function if fstat fails?\n\n> +\t}\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\n> +\tpacket_write(process->in, \"%s\\n\", filter_type);\n> +\tpacket_write(process->in, \"%s\\n\", path);\n> +\tpacket_write(process->in, \"%zu\\n\", len);\n\nI'm not sure if \"%zu\" is portable since we don't do C99 (yet?)\nFor 64-bit signed off_t, you can probably do:\n\n\tpacket_write(process->in, \"%\"PRIuMAX\"\\n\", (uintmax_t)len);\n\nSince we don't have PRIiMAX or intmax_t, here, and a negative\nlen would be a bug (probably from failed fstat) anyways.\n\n> +\tret &= multi_packet_write(src, len, fd, process->in);\n\nmulti_packet_write will probably fail if fstat failed above...\n\n> +\tstrbuf = packet_read_line(process->out, NULL);\n\nAnd this may just block or timeout if multi_packet_write failed.\n\n\nNaptime, I may look at the rest another day.\n"},{"id":"292313","messageId":"20160727133214.GA14928@sigill.intra.peff.net","threadId":"42904","inReplyTo":"c3d7466b-cd3f-9f62-bc82-9f77b4f3d1a4@web.de","subject":"Re: [PATCH v2 4/5] convert: generate large test files only once","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-27T13:32:14Z","receivedAt":"2016-07-27T13:32:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 27, 2016 at 04:35:32AM +0200, Torsten Bögershausen wrote:\n\n> > +\tmkdir -p generated-test-data &&\n> > +\tfor i in $(test_seq 1 $T0021_LARGE_FILE_SIZE)\n> > +\tdo\n> > +\t\t# Generate 1MB of empty data and 100 bytes of random characters\n> > +\t\tprintf \"%1048576d\" 1\n> > +\t\tprintf \"$(LC_ALL=C tr -dc \"A-Za-z0-9\" >8)) count=1 2>/dev/null)\"\n> I'm not sure how portable /dev/urandom is.\n> The other thing, that \"really random\" numbers are an overkill, and\n> it may be easier to use pre-defined numbers,\n\nRight, there are a few reasons not to use /dev/urandom:\n\n - it's not portable\n\n - if we have to generate a lot of numbers, it drains the system's\n entropy pool, which is an unfriendly thing to do (and may also be\n slow)\n\n - it makes our tests random! This sounds like a good thing, but it\n means that if some input happens to cause failure, you are unlikely\n to be able to reproduce it.\n\nInstead, use test-genrandom, which is an LCG that starts at a seed. So\nyou get a large amount of random-ish quickly and portably, and you get\nthe same data each time.\n\n-Peff\n"},{"id":"292327","messageId":"xmqq60rrxct3.fsf@gitster.mtv.corp.google.com","threadId":"42904","inReplyTo":"A6E129CE-26F9-456A-B9C7-A502245DD19E@gmail.com","subject":"Re: [PATCH v2 3/5] pkt-line: extract and use `set_packet_header` function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-27T16:31:36Z","receivedAt":"2016-07-27T16:31:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n>> On 27 Jul 2016, at 02:20, Junio C Hamano wrote:\n>> \n>> larsxschneider@gmail.com writes:\n>> \n>>> From: Lars Schneider \n>>> \n>>> `set_packet_header` converts an integer to a 4 byte hex string. Make\n>>> this function publicly available so that other parts of Git can easily\n>>> generate a pkt-line.\n>> \n>> I think that having to do this is a strong sign that the design of\n>> this series is going in a wrong direction.\n>\n> Thanks for the feedback. Do you think using \"pkt-line\" is a move into\n> the wrong direction in general or do you think only my usage of \n> \"pkt-line\" is not ideal?\n\nI only meant this:\n\n If you try to produce packet-line data without using helper\n functions in pkt-line.[ch] that are designed to do so\n (presumably because the current set of helpers lack some\n capability you want to use), I am not enthused.\n\nAnd I did not see a utility of a public set-packet-header helper\nunless you are hand-rolling a function that produces packet-line\ndata outside pkt-line.[ch], hence the comment we are discussing here\nwas made before actually seeing how this new helper is used.\n\nAs to the use of packet-line as the data format, I do not have a\nstrong opinion either way.\n"},{"id":"292330","messageId":"1D133FC4-032F-434A-882C-5931AA285CD9@gmail.com","threadId":"42904","inReplyTo":"20160727133214.GA14928@sigill.intra.peff.net","subject":"Re: [PATCH v2 4/5] convert: generate large test files only once","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T16:50:40Z","receivedAt":"2016-07-27T16:50:57Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Jul 2016, at 15:32, Jeff King wrote:\n> \n> On Wed, Jul 27, 2016 at 04:35:32AM +0200, Torsten Bögershausen wrote:\n> \n>>> +\tmkdir -p generated-test-data &&\n>>> +\tfor i in $(test_seq 1 $T0021_LARGE_FILE_SIZE)\n>>> +\tdo\n>>> +\t\t# Generate 1MB of empty data and 100 bytes of random characters\n>>> +\t\tprintf \"%1048576d\" 1\n>>> +\t\tprintf \"$(LC_ALL=C tr -dc \"A-Za-z0-9\" >8)) count=1 2>/dev/null)\"\n>> I'm not sure how portable /dev/urandom is.\n>> The other thing, that \"really random\" numbers are an overkill, and\n>> it may be easier to use pre-defined numbers,\n> \n> Right, there are a few reasons not to use /dev/urandom:\n> \n> - it's not portable\n> \n> - if we have to generate a lot of numbers, it drains the system's\n> entropy pool, which is an unfriendly thing to do (and may also be\n> slow)\n> \n> - it makes our tests random! This sounds like a good thing, but it\n> means that if some input happens to cause failure, you are unlikely\n> to be able to reproduce it.\n> \n> Instead, use test-genrandom, which is an LCG that starts at a seed. So\n> you get a large amount of random-ish quickly and portably, and you get\n> the same data each time.\n\nThank you! That's exactly what I need here :-)\n\n- Lars"},{"id":"292335","messageId":"5FE50D2C-5D97-4523-9BE2-88745B3F83EA@gmail.com","threadId":"42904","inReplyTo":"20160727013251.GA12159@sigill.intra.peff.net","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-27T17:31:26Z","receivedAt":"2016-07-27T17:31:34Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Jul 2016, at 03:32, Jeff King wrote:\n> \n> On Wed, Jul 27, 2016 at 02:06:05AM +0200, larsxschneider@gmail.com wrote:\n> \n>> +static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n>> +{\n>> +\toff_t bytes_read;\n>> +\toff_t total_bytes_read = 0;\n> \n> I haven't looked carefully at the whole patch yet, but there seems to be\n> some type issues here. off_t is a good type for storing the whole size\n> of a file (which may be larger than the amount of memory we can\n> allocate). But size_t is the right size for an in-memory object.\n> \n> This function takes a size_t size, which makes sense if it is meant to\n> read everything into a strbuf.\n> \n> So I think our total_bytes_read would probably want to be a size_t here,\n> too, because it cannot possibly grow larger than that (and that is\n> enforced by the loop below). Otherwise you get weirdness like \"sb->buf +\n> total_bytes_ref\" possibly overflowing memory.\n\nOK\n\n\n>> +\tstrbuf_grow(sb, size + 1);\t// we need one extra byte for the packet flush\n> \n> What happens if size is the maximum for size_t here (i.e., 4GB-1 on a\n> 32-bit system)?\n\nWould that be an acceptable solution?\n\nif (size + 1 > SIZE_MAX)\n\treturn die(\"unrepresentable length for filter buffer\");\n\nCan you point me to an example in the Git source how this kind of thing should\nbe handled?\n\n\n>> +\tdo {\n>> +\t\tbytes_read = packet_read(\n>> +\t\t\tfd, NULL, NULL,\n>> +\t\t\tsb->buf + total_bytes_read, sb->len - total_bytes_read - 1,\n>> +\t\t\tPACKET_READ_GENTLE_ON_EOF\n>> +\t\t);\n> \n> packet_read() actually returns an int, and may return \"-1\" on EOF (and\n> int is fine because we know that we are constrained to 16-bit values\n> by the pkt-line definition). You read it into an \"off_t\". I _think_ that\n> is OK, because I believe POSIX says off_t must be signed. But probably\n> \"int\" is the more correct type here.\n\nOK\n\n\n>> +\t\ttotal_bytes_read += bytes_read;\n> \n> If you do get \"-1\", I think you need to detect it here before adjusting\n> total_bytes_read.\n\nCorrect!\n\n\n>> +\twhile (\n>> +\t\tbytes_read > 0 && \t\t\t\t\t// the last packet was no flush\n>> +\t\tsb->len - total_bytes_read - 1 > 0 \t// we still have space left in the buffer\n>> +\t);\n> \n> And I'm not sure if you need to distinguish between \"0\" and \"-1\" when\n> checking byte_read here.\n\nWe want to finish reading in both cases, no?\n\n\n> \n>> +\tstrbuf_setlen(sb, total_bytes_read);\n> \n> Passing an off_t to something expecting a size_t, which can involve\n> truncation (though I think in practice you really are limited to\n> size_t).\n\nOK\n\n\n>> +static int multi_packet_write(const char *src, size_t len, const int in, const int out)\n>> +{\n>> +\tint ret = 1;\n>> +\tchar header[4];\n>> +\tchar buffer[8192];\n>> +\toff_t bytes_to_write;\n>> +\twhile (ret) {\n>> +\t\tif (in >= 0) {\n>> +\t\t\tbytes_to_write = xread(in, buffer, sizeof(buffer));\n> \n> Likewise here, xread() is returning ssize_t. Again, OK if we can assume\n> off_t is signed, but it probably makes sense to use the correct type (we\n> also know it cannot be larger than 8K, of course).\n\nOK\n\n\n> Why 8K? The pkt-line format naturally restricts us to just under 64K, so\n> why not take advantage of that and minimize the framing overhead for\n> large data?\n\nI took inspiration from here for 8K MAX_IO_SIZE:\nhttps://github.com/git/git/blob/master/copy.c#L6\n\nIs this read limit correct? Should I read 8 times to fill a pkt-line?\n\n\n>> +\t\t\tif (bytes_to_write < 0)\n>> +\t\t\t\tret &= 0;\n> \n> I think \"&= 0\" is unusual for our codebase? Would just writing \"= 0\" be\n> more clear?\n\nYes!\n\n\n> We do sometimes do \"ret |= something()\" but that is in cases where\n> \"ret\" is zero for success, and non-zero (usually -1) otherwise. Perhaps\n> your function's error-reporting is inverted from our usual style?\n\nI thought it makes the code easier to read and the filter doesn't care\nat what point the error happens anyways. The filter either succeeds\nor fails. What style would you suggest?\n\n\n>> +\t\tset_packet_header(header, bytes_to_write + 4);\n>> +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n>> +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n>> +\t}\n> \n> If you look at format_packet(), it pulls a slight trick: we have a\n> buffer 4 bytes larger than we need, format into \"buf + 4\", and then\n> write the final size at the beginning. That lets us write() it all in\n> one go.\n> \n> At first I thought this function was simply reinventing packet_write(),\n> but I guess you are trying to avoid the extra copy of the data (once\n> into the buffer from xread, and then again via format_packet just to add\n> the extra bytes at the beginning).\n\nYes, that was my intention.\n\n\n> I agree with what Junio said elsewhere, that there may be a way to make\n> the pkt-line code handle this zero-copy situation better. Perhaps\n> something like:\n> \n> struct pktline {\n> \t/* first 4 bytes are reserved for length header */\n> \tchar buf[LARGE_PACKET_MAX];\n> };\n> #define PKTLINE_DATA_START(pkt) ((pkt)->buf + 4)\n> #define PKTLINE_DATA_LEN (LARGE_PACKET_MAX - 4)\n> \n> ...\n> struct pktline pkt;\n> ssize_t len = xread(fd, PKTLINE_DATA_START(&pkt), PKTLINE_DATA_LEN);\n> packet_send(&pkt, len);\n> \n> Then packet_send() knows that the first 4 bytes are reserved for it. I\n> suspect that the strbuf used by format_packet() could get away with\n> using such a \"struct pktline\" too, though in practice I doubt there's\n> any real efficiency to be gained (we generally reuse the same strbuf\n> over and over, so it will grow once to 64K and get reused).\n\nOK, I will try that.\n\n\n>> +\tret &= write_in_full(out, \"0000\", 4) == 4;\n> \n> packet_flush() ?\n> \n> I know the packet functions are keen on write_or_die() versus\n> write_in_full(). That is perhaps something that should be fixed.\n\nYes, the write_or_die calls were the reason for the manual packet flush. \nI will propose a change for these functions to accommodate non \"required\"\nfilters as it is OK when they fail. \n\n\n> This was just supposed to be a short note about off_t before eating\n> dinner (oops), so I didn't read past here.\n\nThank you :-)\n\n- Lars"},{"id":"292350","messageId":"20160727181148.GC32219@sigill.intra.peff.net","threadId":"42904","inReplyTo":"5FE50D2C-5D97-4523-9BE2-88745B3F83EA@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-27T18:11:48Z","receivedAt":"2016-07-27T18:11:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 27, 2016 at 07:31:26PM +0200, Lars Schneider wrote:\n\n> >> +\tstrbuf_grow(sb, size + 1);\t// we need one extra byte for the packet flush\n> > \n> > What happens if size is the maximum for size_t here (i.e., 4GB-1 on a\n> > 32-bit system)?\n> \n> Would that be an acceptable solution?\n> \n> if (size + 1 > SIZE_MAX)\n> \treturn die(\"unrepresentable length for filter buffer\");\n\nNo, because by definition \"size\" will wrap to 0. :)\n\nYou have to do:\n\n if (size > SIZE_MAX - 1)\n\tdie(\"whoops\");\n\n> Can you point me to an example in the Git source how this kind of thing should\n> be handled?\n\nThe strbuf code itself checks for overflows. So you could do:\n\n strbuf_grow(sb, size);\n ... fill up with size bytes ...\n strbuf_addch(sb, ...); /* extra byte for whatever */\n\nThat does mean _possibly_ making a second allocation just to add the\nextra byte, but in practice it's not likely (unless the input exactly\nmatches the strbuf's growth pattern).\n\nIf you want to do it yourself, I think:\n\n strbuf_grow(sb, st_add(size, 1));\n\nwould work.\n\n> >> +\twhile (\n> >> +\t\tbytes_read > 0 && \t\t\t\t\t// the last packet was no flush\n> >> +\t\tsb->len - total_bytes_read - 1 > 0 \t// we still have space left in the buffer\n> >> +\t);\n> > \n> > And I'm not sure if you need to distinguish between \"0\" and \"-1\" when\n> > checking byte_read here.\n> \n> We want to finish reading in both cases, no?\n\nIf we get \"-1\", that's from an unexpected EOF during the packet_read(),\nbecause you set GENTLE_ON_EOF. So there's nothing left to read, and we\nshould break and return an error.\n\nI guess \"0\" would come from a flush packet? Why would the filter send\nback a flush packet (unless you were using them to signal end-of-input,\nbut then why does the filter have to send back the number of bytes ahead\nof time?).\n\n> > Why 8K? The pkt-line format naturally restricts us to just under 64K, so\n> > why not take advantage of that and minimize the framing overhead for\n> > large data?\n> \n> I took inspiration from here for 8K MAX_IO_SIZE:\n> https://github.com/git/git/blob/master/copy.c#L6\n> \n> Is this read limit correct? Should I read 8 times to fill a pkt-line?\n\nMAX_IO_SIZE is generally 8 _megabytes_, not 8K. The loop in copy.c just\nhaad to pick an arbitrary size for doing its read/write proxying. I\nthink in practice you are not likely to get much benefit from going\nbeyond 8K or so there, just because operating systems tend to do things\nin page-sizes anyway, which are usually 4K.\n\nBut since you are formatting the result into a form that has framing\noverhead, anything up to LARGE_PACKET_MAX will see benefits (though\nadmittedly even 4 bytes per 8K is not much).\n\nI don't think it's worth the complexity of reading 8 times, but just\nusing a buffer size of LARGE_PACKET_MAX-4 would be the most efficient.\n\nI doubt it matters _that much_ in practice, but any time I see a magic\nnumber I have to wonder at the \"why\". At least basing it on\nLARGE_PACKET_MAX has some basis, whereas 8K is largely just made-up. :)\n\n> > We do sometimes do \"ret |= something()\" but that is in cases where\n> > \"ret\" is zero for success, and non-zero (usually -1) otherwise. Perhaps\n> > your function's error-reporting is inverted from our usual style?\n> \n> I thought it makes the code easier to read and the filter doesn't care\n> at what point the error happens anyways. The filter either succeeds\n> or fails. What style would you suggest?\n\nI think that's orthogonal. I just mean that using zero for success puts\nyou in our usual style, and then accumulating errors can be done with\n\"|=\".\n\nI didn't look carefully at whether the accumulating style you're using\nmakes sense or not. But something like:\n\n> >> +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n> >> +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n\ndoes mean that we call the second write() even if the first one failed.\nThat's a waste of time (albeit a minor one), but it also means you could\npotentially cover up the value of \"errno\" from the first one (though in\npractice I'd expect the second one to fail for the same reason).\n\n-Peff\n"},{"id":"292358","messageId":"579906C5.1050809@gmail.com","threadId":"42904","inReplyTo":"20160727000605.49982-1-larsxschneider@gmail.com","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-27T19:08:53Z","receivedAt":"2016-07-27T19:09:25Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-27 o 02:06, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> Hi,\n> \n> thanks a lot for the extensive reviews. I tried to address all mentioned\n> concerns and summarized them below. The most prominent changes since v1 are\n> the following:\n> * Git offers a number of filter capabilities that a filter can request\n> (right now only \"smudge\" and \"clean\" - in the future maybe \"cleanFromFile\",\n> \"smudgeToFile\", and/or \"stream\")\n> * pipe communication uses a packet format (pkt-line) based protocol\n\nI wonder if it would make sense to support both whole-file pipe communication,\nand packet format (pkt-line) pipe communication.\n\nThe problem with whole-file pipe communication (original proposal for\nnew filter protocol is that it needs file size upfront. For some types\nof filters it is not a problem:\n - if a filtered file has the same size as original, like for rot13\n example in the test for the feature\n - if you can calculate the resulting file size from original size,\n like for most if not all encryption formats (that includes GPG,\n uudecode, base64, quoted-printable, hex, etc.); same for decryption,\n and from converting between fixed-width encodings\n - if resulting file size is saved somewhere that is easy to get, like\n for LFS solutions (I think).\n\nFor other filters it is serious problem. For example indent, keyword\nexpansion, rezipping with zero compression (well, maybe not this one,\nbut at least the reverse of it), converting between encodings where\nat least one is variable width (like UTF-8),...\n\nIMHO writing whole-file persistent filters is easier than using pkt-line.\nOn the other hand using pkt-line allow for more detailed progress report.\n\n> * a long running filter application is defined with \"filter..process\"\n\nI hope that won't confuse Git users into trying to use single-shot\nfilters with a new protocol...\n\n> ## Torsten:\n> * add \"\\n\" line terminator after version in init sequence\n> * prepare big file for EXPENSIVE tests once\n> * set \"#!/usr/bin/perl\" as shebang for rot13.pl to mimic other Perl test scripts\n> * add test_have_prereq PERL to t0021\n> \n> ## Ramsay:\n> * use write_in_full(process->in, nbuf.buf, nbuf.len) to avoid unneccesary strlen call\n> * use read_in_full to read data that exceeds MAX_IO_SIZE properly\n> * fix test case to check for large file filtering\n> \n> ## Jakub:\n> * use standard input/standard output instead of stdin/stdout [in description/documentation]\n> * replace global variable \"cmd_process_map\" with a function parameter where possible\n> * avoid \"strbuf_reset\" after STRBUF_INIT\n> * align test_config_global\n> * rename rot13.pl to rot13-filter.pl\n> * make Perl style consistent\n> * describe hard coded filenames in test filter header\n> * improve docs\n> * add filter capabilities field (enables cleanToFile, smudgeFromFile, and/or stream later)\n> * explain that content size in bytes is encoded in ASCII\n> * consistent line ending for die call in Perl (without \"\\n\")\n> * make rot13 test filter die in case of failure (instead of returning \"fail\")\n> \n> ## Eric:\n> * flush explicitly in Perl test filter\n> * do not initialize variables to NULL if they are set unconditionally\n> * fix no-op stop_protocol_filter\n> * use off_t instead of size_t\n> * improve test filter int parsing ($filelen =~ /\\A\\d+\\z/ or die \"bad filelen: $filelen\")\n> \n> ## Peff:\n> * use pkt-line protocol\n> * do not use Perl autodie\n> \n> ## Remi:\n> * remove spaces after '<'\n\n\n"},{"id":"292366","messageId":"57991329.1000709@gmail.com","threadId":"42904","inReplyTo":"20160727000605.49982-2-larsxschneider@gmail.com","subject":"Re: [PATCH v2 1/5] convert: quote filter names in error messages","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-27T20:01:45Z","receivedAt":"2016-07-27T20:02:06Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-27 o 02:06, larsxschneider@gmail.com pisze:\n\n> Git filter with spaces (e.g. `filter.sh foo`) are hard to read in\n> error messages. Quote them to improve the readability.\n\nThis is not something very important, but the above commit message\nfeels a bit clunky to me. The change is easy to understand, though,\nso the commit message style is not that important.\n\nPerhaps \"Git filter driver command with spaces\"?\n\nWell, nevermind; if you have a better idea, good. If not, it is\ngood enough, IMHO.\n-- \nJakub Narębski\n"},{"id":"292383","messageId":"57994436.4080308@gmail.com","threadId":"42904","inReplyTo":"20160727000605.49982-6-larsxschneider@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-27T23:31:02Z","receivedAt":"2016-07-27T23:31:36Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-27 o 02:06, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> Git's clean/smudge mechanism invokes an external filter process for every\n> single blob that is affected by a filter. If Git filters a lot of blobs\n> then the startup time of the external filter processes can become a\n> significant part of the overall Git execution time.\n\nIt is not strictly necessary... but do we have any benchmarks for this,\nor is it just the feeling? That is, in what situations Git may filter\na large number of files (initial checkout? initial add?, switching\nto unrelated branch? getting large files from LFS solution?, and when\nstartup time might become significant part of execution time (MS Windows?\nfast filters?)?\n\n> \n> This patch adds the filter..process string option which, if used,\n\nString option... what are possible values? What happens if you use\nvalue that is not recognized by Git (it is \"if used\", isn't it)? That's\nnot obvious from the commit message (though it might be from the docs).\n\nWhat is missing is the description that it is set to a command, and\nhow it interacts with `clean` and `smudge` options.\n\n> keeps the external filter process running and processes all blobs with\n> the following packet format (pkt-line) based protocol over standard input\n> and standard output.\n> \n> Git starts the filter on first usage and expects a welcome\n> message, protocol version number, and filter capabilities\n> seperated by spaces:\n\ns/seperated/separated/\n\nIs there any handling of misconfigured one-shot filters, or would\nthey still hang the execution of a Git command?\n\n> ------------------------\n> packet: git< git-filter-protocol\n> packet: git< version 2\n> packet: git< clean smudge\n\nWouldn't \"capabilities clean smudge\" be better? Or is it the\n\"clean smudge\" proposal easier to handle?\n\n> ------------------------\n> Supported filter capabilities are \"clean\" and \"smudge\".\n> \n> Afterwards Git sends a command (e.g. \"smudge\" or \"clean\" - based\n> on the supported capabilities), the filename, the content size as\n> ASCII number in bytes, and the content in packet format with a\n> flush packet at the end:\n> ------------------------\n> packet: git> smudge\n> packet: git> testfile.dat\n\nAnd here we don't have any problems with files containing embedded\nnewlines etc. Also Git should not be sending invalid file names.\nThe question remains: is it absolute file path, or basename?\n\n> packet: git> 7\n> packet: git> CONTENT\n\nCan Git send file contents using more than one packet? I think\nit should be stated upfront.\n\n> packet: git> 0000\n> ------------------------\n\nWhy we need to send content size upfront? Well, it is not a problem\nfor Git, but (as I wrote in reply to the cover letter for this\nseries) might be a problem for filter scripts.\n\n> \n> The filter is expected to respond with the result content size as\n> ASCII number in bytes and the result content in packet format with\n> a flush packet at the end:\n> ------------------------\n> packet: git< 57\n\nThis is not neccessary (and might be hard for scripts to do) if\npkt-line protocol is used.\n\nIn short: I think pkt-line is not worth the complication on\nthe Git side and on the filter size, unless it is used for\nstreaming, or at least filter not having to calculate output\nsize upfront.\n\n> packet: git< SMUDGED_CONTENT\n> packet: git< 0000\n> ------------------------\n> Please note: In a future version of Git the capability \"stream\"\n> might be supported. In that case the content size must not be\n> part of the filter response.\n> \n> Afterwards the filter is expected to wait for the next command.\n\nWhen filter is supposed to exit, then?\n\n> \n> Signed-off-by: Lars Schneider \n> Helped-by: Martin-Louis Bright \n> Signed-off-by: Lars Schneider \n> ---\n> Documentation/gitattributes.txt | 54 +++++++-\n> convert.c | 269 ++++++++++++++++++++++++++++++++++++++--\n> t/t0021-conversion.sh | 175 ++++++++++++++++++++++++++\n> t/t0021/rot13-filter.pl | 146 ++++++++++++++++++++++\n> 4 files changed, 631 insertions(+), 13 deletions(-)\n> create mode 100755 t/t0021/rot13-filter.pl\n> \n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index 8882a3e..8fb40d2 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -300,7 +300,11 @@ checkout, when the `smudge` command is specified, the command is\n> fed the blob object from its standard input, and its standard\n> output is used to update the worktree file. Similarly, the\n> `clean` command is used to convert the contents of worktree file\n> -upon checkin.\n> +upon checkin. By default these commands process only a single\n> +blob and terminate. If a long running filter process (see section\n> +below) is used then Git can process all blobs with a single filter\n> +invocation for the entire life of a single Git command (e.g.\n> +`git add .`).\n\nAh, all right, here we give an example.\n\nBut, is \"blob\" term used in this document, or do we use \"file\"\nand \"file contents\" only?\n\n> \n> One use of the content filtering is to massage the content into a shape\n> that is more convenient for the platform, filesystem, and the user to use.\n> @@ -375,6 +379,54 @@ substitution. For example:\n> ------------------------\n> \n> \n> +Long Running Filter Process\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +If the filter command (string value) is defined via\n> +filter..process then Git can process all blobs with a\n> +single filter invocation for the entire life of a single Git\n> +command by talking with the following packet format (pkt-line)\n> +based protocol over standard input and standard output.\n\nAh, so now it is the name of command, and I assume it is\nexclusive with `clean` / `smudge`, or does it only takes\nprecedence based on capabilities of filter (that is if\nfor example \"`process`\" does not include 'clean' capability,\nthen `clean` filter is used, using per-file \"protocol\").\nOr do something different happens (like preference for\nold-style `clean` and `smudge` filters, and `process`\nused if any unset)?\n\nAnyway, Git command would never (I think) run both\n\"clean\" and \"smudge\" filters. But I might be wrong.\n\nYeah, I know this going back and forth seems like \nbike-shedding, but designing good user-facing API\nis very, very important.\n\n> +\n> +Git starts the filter on first usage and expects a welcome\n> +message, protocol version number, and filter capabilities\n> +seperated by spaces:\n> +------------------------\n> +packet: git< git-filter-protocol\n> +packet: git< version 2\n> +packet: git< clean smudge\n> +------------------------\n\nNeither of those is terminated by end of line character,\nthat is, \"\\n\", isn't it?\n\n> diff --git a/convert.c b/convert.c\n> index 522e2c5..5ff200b 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -3,6 +3,7 @@\n> #include \"run-command.h\"\n> #include \"quote.h\"\n> #include \"sigchain.h\"\n> +#include \"pkt-line.h\"\n> \n> /*\n> * convert.c - convert a file when checking it out and checking it in.\n> @@ -481,11 +482,232 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n> \treturn ret;\n> }\n> \n> +static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n\nWhat's the purpose of this function? Is it to gather read whole\ncontents of file into strbuf? Or is it to read at most 'size'\nbytes of file / of pkt-line stream into strbuf?\n\nWe probably don't want to keep the whole file in memory,\nif possible.\n\n> +{\n> +\toff_t bytes_read;\n> +\toff_t total_bytes_read = 0;\n> +\tstrbuf_grow(sb, size + 1);\t// we need one extra byte for the packet flush\n\nWhy we put packet flush into strbuf? Or is it only temporarily,\nand we adjust that at the end... I see that it is.\n\n> +\tdo {\n> +\t\tbytes_read = packet_read(\n> +\t\t\tfd, NULL, NULL,\n> +\t\t\tsb->buf + total_bytes_read, sb->len - total_bytes_read - 1,\n> +\t\t\tPACKET_READ_GENTLE_ON_EOF\n> +\t\t);\n> +\t\ttotal_bytes_read += bytes_read;\n> +\t}\n> +\twhile (\n> +\t\tbytes_read > 0 && \t\t\t\t\t// the last packet was no flush\n> +\t\tsb->len - total_bytes_read - 1 > 0 \t// we still have space left in the buffer\n> +\t);\n> +\tstrbuf_setlen(sb, total_bytes_read);\n> +\treturn total_bytes_read;\n> +}\n> +\n> +static int multi_packet_write(const char *src, size_t len, const int in, const int out)\n\nWhat's the purpose of this function? What are those 'in' and 'out'\nparameters? Those names do not describe them well. If they are\nfile descriptors, add fd_* prefix (or whatever Git code uses).\nEdit: I see that's what existing code uses.\n\nEdit: so we are reading from *src + len or from fd_in, depending on\nwhether fd_in is set to 0 or not? I guess that follows existing\ncode, where it is even worse, because it is hidden...\n\n> +{\n> +\tint ret = 1;\n> +\tchar header[4];\n> +\tchar buffer[8192];\n\nCould those two be in one variable? Also, 'header' or 'pkt_header'?\n\nWhy 8192, and not LARGE_PACKET_MAX - 4?\n\n> +\toff_t bytes_to_write;\n> +\twhile (ret) {\n> +\t\tif (in >= 0) {\n> +\t\t\tbytes_to_write = xread(in, buffer, sizeof(buffer));\n> +\t\t\tif (bytes_to_write < 0)\n> +\t\t\t\tret &= 0;\n> +\t\t\tsrc = buffer;\n> +\t\t} else {\n> +\t\t\tbytes_to_write = len > LARGE_PACKET_MAX - 4 ? LARGE_PACKET_MAX - 4 : len;\n> +\t\t\tlen -= bytes_to_write;\n> +\t\t}\n> +\t\tif (!bytes_to_write)\n> +\t\t\tbreak;\n> +\t\tset_packet_header(header, bytes_to_write + 4);\n> +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n> +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n\nThese three lines are equivalent to write_packet(), or however\nit is named, isn't it?\n\n> +\t}\n> +\tret &= write_in_full(out, \"0000\", 4) == 4;\n\nThis is equivalent to packet_flush(), or however it is named,\nisn't it?\n\n> +\treturn ret;\n> +}\n> +\n> +struct cmd2process {\n> +\tstruct hashmap_entry ent; /* must be the first member! */\n> +\tconst char *cmd;\n> +\tint clean;\n> +\tint smudge;\n\nThese two are 'int' used as 'bool', isn't it?\n\n> +\tstruct child_process process;\n> +};\n[...]\n> +static struct cmd2process *find_protocol_filter_entry(struct hashmap *hashmap, const char *cmd)\n\nWouldn't it be more descriptive to name the first parameter\nto this function 'cmd_hashmap', or something like that, rather\nthan plain 'hashmap' (it might be the same that is used / was\nused for a global variable)?\n\nEdit: or 'cmd_process_map'.\n\n> +{\n> +\tstruct cmd2process k;\n> +\thashmap_entry_init(&k, strhash(cmd));\n> +\tk.cmd = cmd;\n> +\treturn hashmap_get(hashmap, &k, NULL);\n> +}\n\n[...]\n> +static struct cmd2process *start_protocol_filter(struct hashmap *hashmap, const char *cmd)\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry;\n> +\tstruct child_process *process;\n> +\tconst char *argv[] = { NULL, NULL };\n\nCould we initialize it with { cmd, NULL };?\n\nEdit: Ah, I see that you follow filter_buffer_or_fd() example from\nconvert.c, isn't it?\n\n> +\tstruct string_list capabilities = STRING_LIST_INIT_NODUP;\n> +\tchar *capabilities_buffer;\n> +\tint i;\n> +\n> +\tentry = xmalloc(sizeof(*entry));\n> +\thashmap_entry_init(entry, strhash(cmd));\n> +\tentry->cmd = cmd;\n> +\tentry->clean = 0;\n> +\tentry->smudge = 0;\n\nWouldn't\n\n \tentry->clean = entry->smudge = 0;\n\nbe more readable?\n\n> +\tprocess = &entry->process;\n> +\n> +\tchild_process_init(process);\n> +\targv[0] = cmd;\n> +\tprocess->argv = argv;\n> +\tprocess->use_shell = 1;\n> +\tprocess->in = -1;\n\nMaybe\n\n +\tprocess->in = -1;\n\nto align, but perhaps it is not worth it.\n\n> +\tprocess->out = -1;\n> +\n> +\tif (start_command(process)) {\n> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n\nJust a question: is \"cannot fork\" the only reason why start_command()\nmight have failed there?\n\nEdit: Ah, I see that you follow filter_buffer_or_fd() example from\nconvert.c, again.\n\n> +\t\tstop_protocol_filter(hashmap, entry);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tret &= strcmp(packet_read_line(process->out, NULL), \"git-filter-protocol\") == 0;\n> +\tret &= strcmp(packet_read_line(process->out, NULL), \"version 2\") == 0;\n\nSo that's why you need packet_read_line() to return string...\n\n> +\tcapabilities_buffer = packet_read_line(process->out, NULL);\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tstring_list_split_in_place(&capabilities, capabilities_buffer, ' ', -1);\n\nThis does not modify capabilities_buffer, does it?\n\n> +\tfor (i = 0; i < capabilities.nr; i++) {\n> +\t\tconst char *requested = capabilities.items[i].string;\n> +\t\tif (!strcmp(requested, \"clean\")) {\n> +\t\t\tentry->clean = 1;\n> +\t\t} else if (!strcmp(requested, \"smudge\")) {\n> +\t\t\tentry->smudge = 1;\n> +\t\t} else {\n> +\t\t\twarning(\n> +\t\t\t\t\"filter process '%s' requested unsupported filter capability '%s'\",\n> +\t\t\t\tcmd, requested\n> +\t\t\t);\n\nNice. This makes it (somewhat) forward- and backward-compatibile.\n\n> +\t\t}\n> +\t}\n> +\tstring_list_clear(&capabilities, 0);\n> +\n> +\tif (!ret) {\n> +\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n\nDo we need more detailed information about the source of error?\n\n> +\t\tstop_protocol_filter(hashmap, entry);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\thashmap_add(hashmap, entry);\n> +\treturn entry;\n> +}\n> +\n> +static int cmd_process_map_init = 0;\n> +static struct hashmap cmd_process_map;\n> +\n> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n> +\t\t\t\t\t\tconst char *filter_type)\n\nThat is... quite a lot of parameters. But I guess there is precedens.\nBut I think 'fd' belongs to previous line, as it is alternative to\nsrc+len.\n\n> +{\n> +\tint ret = 1;\n> +\tstruct cmd2process *entry;\n> +\tstruct child_process *process;\n> +\tstruct stat file_stat;\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n> +\toff_t expected_bytes;\n> +\tchar *strtol_end;\n> +\tchar *strbuf;\n> +\n> +\tif (!cmd || !*cmd)\n> +\t\treturn 0;\n> +\n> +\tif (!dst)\n> +\t\treturn 1;\n> +\n> +\tif (!cmd_process_map_init) {\n> +\t\tcmd_process_map_init = 1;\n> +\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n> +\t\tentry = NULL;\n\nIs it better than having entry NULL-initialized?\n\n> +\t} else {\n> +\t\tentry = find_protocol_filter_entry(&cmd_process_map, cmd);\n> +\t}\n> +\n> +\tif (!entry) {\n> +\t\tentry = start_protocol_filter(&cmd_process_map, cmd);\n\nHmmm... apply_filter() uses start_async() for some reason. Why\nit does not apply for this new kind of filter?\n\n> +\t\tif (!entry) {\n> +\t\t\treturn 0;\n> +\t\t}\n> +\t}\n> +\tprocess = &entry->process;\n> +\n> +\tif (!(!strcmp(filter_type, \"clean\") && entry->clean) &&\n> +\t\t!(!strcmp(filter_type, \"smudge\") && entry->smudge)) {\n\nWould it be more readable as !(A || B) rather than (!A && !B)?\n\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tif (fd >= 0 && !src) {\n> +\t\tret &= fstat(fd, &file_stat) != -1;\n> +\t\tlen = file_stat.st_size;\n> +\t}\n\nAll right, so we either use src+len, or if we use fd we get\nfile size.\n\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\n> +\tpacket_write(process->in, \"%s\\n\", filter_type);\n> +\tpacket_write(process->in, \"%s\\n\", path);\n> +\tpacket_write(process->in, \"%zu\\n\", len);\n\nSo \"\\n\" is included in protocol?\n\n> +\tret &= multi_packet_write(src, len, fd, process->in);\n\nHow git-receive-pack etc. handle multi-packet write?\n\n> +\n> +\tstrbuf = packet_read_line(process->out, NULL);\n> +\texpected_bytes = (off_t)strtol(strbuf, &strtol_end, 10);\n> +\tret &= (strtol_end != strbuf && errno != ERANGE);\n> +\n> +\tif (expected_bytes > 0) {\n> +\t\tret &= multi_packet_read(&nbuf, process->out, expected_bytes) == expected_bytes;\n> +\t}\n> +\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tif (ret) {\n> +\t\tstrbuf_swap(dst, &nbuf);\n> +\t} else {\n> +\t\t// Something went wrong with the protocol filter. Force shutdown!\n> +\t\tstop_protocol_filter(&cmd_process_map, entry);\n\nSome error message would be nice... or do we print in down in stack?\n\n> +\t}\n> +\tstrbuf_release(&nbuf);\n> +\treturn ret;\n> +}\n> +\n\n\n[...]\n> @@ -823,7 +1049,10 @@ int would_convert_to_git_filter_fd(const char *path)\n> \tif (!ca.drv->required)\n> \t\treturn 0;\n> \n> -\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n> +\tif (!ca.drv->clean && ca.drv->process)\n> +\t\treturn apply_protocol_filter(path, NULL, 0, -1, NULL, ca.drv->process, \"clean\");\n> +\telse\n> +\t\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n\nSo the rule is: if `clean` is not set, and `process` is, try to use\nprocess for cleaning. It was not clear for me from the documentation.\n\n> }\n> \n> const char *get_convert_attr_ascii(const char *path)\n> @@ -856,17 +1085,22 @@ int convert_to_git(const char *path, const char *src, size_t len,\n> struct strbuf *dst, enum safe_crlf checksafe)\n> {\n> \tint ret = 0;\n> -\tconst char *filter = NULL;\n> +\tconst char *clean_filter = NULL;\n> +\tconst char *process_filter = NULL;\n> \tint required = 0;\n> \tstruct conv_attrs ca;\n> \n> \tconvert_attrs(&ca, path);\n> \tif (ca.drv) {\n> -\t\tfilter = ca.drv->clean;\n> +\t\tclean_filter = ca.drv->clean;\n> +\t\tprocess_filter = ca.drv->process;\n> \t\trequired = ca.drv->required;\n> \t}\n> \n> -\tret |= apply_filter(path, src, len, -1, dst, filter);\n> +\tif (!clean_filter && process_filter)\n> +\t\tret |= apply_protocol_filter(path, src, len, -1, dst, process_filter, \"clean\");\n> +\telse\n> +\t\tret |= apply_filter(path, src, len, -1, dst, clean_filter);\n\nAnd the rule is the same here, as it should.\n\n> \tif (!ret && required)\n> \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n\nIs it a correct error message for `process`? I guess it is, as it prints\nthe name of driver, and not attempted command. Well, we might be using\n\"process\" filter in 'clean' mode,... but that is sophistry.\n\n[...]\n> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n> index b9911a4..c4793ed 100755\n> --- a/t/t0021-conversion.sh\n> +++ b/t/t0021-conversion.sh\n> @@ -4,6 +4,11 @@ test_description='blob conversion via gitattributes'\n> \n> . ./test-lib.sh\n> \n> +if ! test_have_prereq PERL; then\n> +\tskip_all='skipping perl interface tests, perl not available'\n> +\ttest_done\n> +fi\n\nDo all tests require Perl?\n\n> +test_expect_success 'required protocol filter should filter data' '\n[...]\n> +test_expect_success 'protocol filter large file' '\n[...]\n> +test_expect_success 'required protocol filter should fail with clean' '\n[...]\n> +test_expect_success 'protocol filter should restart after failure' '\n[...]\n\n> diff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\n> new file mode 100755\n> index 0000000..7176836\n> --- /dev/null\n> +++ b/t/t0021/rot13-filter.pl\n> @@ -0,0 +1,146 @@\n> +#!/usr/bin/perl\n> +#\n> +# Example implementation for the Git filter protocol version 2\n> +# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n> +#\n> +# This implementation supports two special test cases:\n> +# (1) If data with the filename \"clean-write-fail.r\" is processed with\n> +# a \"clean\" operation then the write operation will die.\n> +# (2) If data with the filename \"smudge-write-fail.r\" is processed with\n> +# a \"smudge\" operation then the write operation will die.\n\nNice.\n\n> +#\n> +\n> +use strict;\n> +use warnings;\n> +\n> +my $MAX_PACKET_CONTENT_SIZE = 65516;\n> +\n> +sub rot13 {\n> + my ($str) = @_;\n> + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n> + return $str;\n> +}\n> +\n> +sub packet_read {\n> + my $buffer;\n> + my $bytes_read = read STDIN, $buffer, 4;\n> + if ( $bytes_read == 0 ) {\n> + return;\n> + }\n> + elsif ( $bytes_read != 4 ) {\n\nThis is a bit untypical bracket style...\n\n> + die \"invalid packet size '$bytes_read' field\";\n> + }\n> + my $pkt_size = hex($buffer);\n> + if ( $pkt_size == 0 ) {\n> + return ( 1, \"\" );\n> + }\n> + elsif ( $pkt_size > 4 ) {\n> + my $content_size = $pkt_size - 4;\n> + $bytes_read = read STDIN, $buffer, $content_size;\n> + if ( $bytes_read != $content_size ) {\n> + die \"invalid packet\";\n> + }\n> + return ( 0, $buffer );\n> + }\n> + else {\n> + die \"invalid packet size\";\n> + }\n> +}\n\nSo packet reading is not that difficult...\n\n> +\n> +sub packet_write {\n> + my ($packet) = @_;\n> + print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n> + print STDOUT $packet;\n> + STDOUT->flush();\n> +}\n\n...and packet write is easy.\n\n> +\n> +sub packet_flush {\n> + print STDOUT sprintf( \"%04x\", 0 );\n> + STDOUT->flush();\n> +}\n> +\n> +open my $debug, \">>\", \"output.log\";\n> +print $debug \"start\\n\";\n> +$debug->flush();\n> +\n> +packet_write(\"git-filter-protocol\\n\");\n> +packet_write(\"version 2\\n\");\n> +packet_write(\"clean smudge\\n\");\n> +print $debug \"wrote filter header\\n\";\n> +$debug->flush();\n\nIsn't $debug flushed automatically?\n\n> +\n> +while (1) {\n> + my $command = packet_read();\n> + unless ( defined($command) ) {\n> + exit();\n> + }\n> + chomp $command;\n> + print $debug \"IN: $command\";\n> + $debug->flush();\n> + my $filename = packet_read();\n> + chomp $filename;\n> + print $debug \" $filename\";\n> + $debug->flush();\n> + my $filelen = packet_read();\n> + chomp $filelen;\n> + print $debug \" $filelen\";\n> + $debug->flush();\n> +\n> + $filelen =~ /\\A\\d+\\z/ or die \"bad filelen: $filelen\";\n> + my $output;\n> +\n> + if ( $filelen > 0 ) {\n> + my $input = \"\";\n> + {\n> + binmode(STDIN);\n> + my $buffer;\n> + my $done = 0;\n> + while ( !$done ) {\n> + ( $done, $buffer ) = packet_read();\n> + $input .= $buffer;\n> + }\n> + print $debug \" [OK] -- \";\n> + $debug->flush();\n> + }\n> +\n> + if ( $command eq \"clean\" ) {\n> + $output = rot13($input);\n> + }\n> + elsif ( $command eq \"smudge\" ) {\n> + $output = rot13($input);\n> + }\n> + else {\n> + die \"bad command\";\n\nPerhaps\n\n die \"bad command $command\";\n\n> + }\n> + }\n> +\n> + my $output_len = length($output);\n> + packet_write(\"$output_len\\n\");\n> + print $debug \"OUT: $output_len \";\n> + $debug->flush();\n> + if ( $output_len > 0 ) {\n> + if ( ( $command eq \"clean\" and $filename eq \"clean-write-fail.r\" )\n\nWhat happened here with whitespace around parentheses?\n\n> + or\n> + ( $command eq \"smudge\" and $filename eq \"smudge-write-fail.r\" ) )\n> + {\n> + print $debug \" [FAIL]\\n\";\n> + $debug->flush();\n> + die \"write error\";\n> + }\n> + else {\n> + while ( length($output) > 0 ) {\n> + my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n> + packet_write($packet);\n> + if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n> + $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n> + }\n> + else {\n> + $output = \"\";\n> + }\n> + }\n> + packet_flush();\n> + print $debug \"[OK]\\n\";\n> + $debug->flush();\n> + }\n> + }\n> +}\n> \n\n"},{"id":"292388","messageId":"BFED7ED8-40DB-46B5-A1B7-4F49624D5A62@gmail.com","threadId":"42904","inReplyTo":"579906C5.1050809@gmail.com","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-28T07:16:18Z","receivedAt":"2016-07-28T07:16:31Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Jul 2016, at 21:08, Jakub Narębski wrote:\n> \n> W dniu 2016-07-27 o 02:06, larsxschneider@gmail.com pisze:\n>> From: Lars Schneider \n>> \n>> Hi,\n>> \n>> thanks a lot for the extensive reviews. I tried to address all mentioned\n>> concerns and summarized them below. The most prominent changes since v1 are\n>> the following:\n>> * Git offers a number of filter capabilities that a filter can request\n>> (right now only \"smudge\" and \"clean\" - in the future maybe \"cleanFromFile\",\n>> \"smudgeToFile\", and/or \"stream\")\n>> * pipe communication uses a packet format (pkt-line) based protocol\n> \n> I wonder if it would make sense to support both whole-file pipe communication,\n> and packet format (pkt-line) pipe communication.\n> \n> The problem with whole-file pipe communication (original proposal for\n> new filter protocol is that it needs file size upfront. For some types\n> of filters it is not a problem:\n> - if a filtered file has the same size as original, like for rot13\n> example in the test for the feature\n> - if you can calculate the resulting file size from original size,\n> like for most if not all encryption formats (that includes GPG,\n> uudecode, base64, quoted-printable, hex, etc.); same for decryption,\n> and from converting between fixed-width encodings\n> - if resulting file size is saved somewhere that is easy to get, like\n> for LFS solutions (I think).\n> \n> For other filters it is serious problem. For example indent, keyword\n> expansion, rezipping with zero compression (well, maybe not this one,\n> but at least the reverse of it), converting between encodings where\n> at least one is variable width (like UTF-8),...\n> \n> IMHO writing whole-file persistent filters is easier than using pkt-line.\n> On the other hand using pkt-line allow for more detailed progress report.\n\nI initially wanted to support only \"while-file\" pipe, too.\n\nBut Peff ($gmane/299902), Duy, and Eric, seemed to prefer the pkt-line\nsolution (gmane is down - otherwise I would have given you the links).\n\nAfter I have looked at it I think the pkt-line solution is indeed nicer\nfor the following reasons:\n\n(1) A stream optimized version (read/write in separate threads) of the filter\n protocol can be implemented in the future without changing the protocol\n(2) pkt-line is a simple and easy to implement format\n(3) Reuse of existing Git communication infrastructure\n -> code and documentation are less surprising to people that know Git\n -> you can use GIT_TRACE_PACKET to easily inspect the\n communication between Git and the filter process\n(4) The overheads is neglect able (4 byte header vs 65516 byte content)\n\n\n>> * a long running filter application is defined with \"filter..process\"\n> \n> I hope that won't confuse Git users into trying to use single-shot\n> filters with a new protocol...\n\nYes, that was my intention for this new config.\n\nThanks,\nLars"},{"id":"292389","messageId":"DAD9422F-43EF-4F98-A373-87BCF5C1FCD1@gmail.com","threadId":"42904","inReplyTo":"57991329.1000709@gmail.com","subject":"Re: [PATCH v2 1/5] convert: quote filter names in error messages","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-28T08:23:39Z","receivedAt":"2016-07-28T08:23:52Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Jul 2016, at 22:01, Jakub Narębski wrote:\n> \n> W dniu 2016-07-27 o 02:06, larsxschneider@gmail.com pisze:\n> \n>> Git filter with spaces (e.g. `filter.sh foo`) are hard to read in\n>> error messages. Quote them to improve the readability.\n> \n> This is not something very important, but the above commit message\n> feels a bit clunky to me. The change is easy to understand, though,\n> so the commit message style is not that important.\n> \n> Perhaps \"Git filter driver command with spaces\"?\n> \n> Well, nevermind; if you have a better idea, good. If not, it is\n> good enough, IMHO.\n\nAgreed. I will fix this with the next roll.\n\nThanks,\nLars"},{"id":"292391","messageId":"df979c36-d8a8-8074-bddc-f67b245e1589@web.de","threadId":"42904","inReplyTo":"20160727000605.49982-6-larsxschneider@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-07-28T10:32:16Z","receivedAt":"2016-07-28T10:33:33Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\nOn 07/27/2016 02:06 AM, larsxschneider@gmail.com wrote:\nSome comments inline\n[]\n > The filter is expected to respond with the result content size as\n > ASCII number in bytes and the result content in packet format with\n > a flush packet at the end:\n > ------------------------\n > packet: git< 57\n > packet: git< SMUDGED_CONTENT\n > packet: git< 0000\nhow does the filter report possible errors here ?\nLet's say I want to convert UTF-8 into ISO-8859-1,\nbut the conversion is impossible.\n\n > packet: git< -1\n > packet: git< Error message, (or empty), followed by a '\\n'\n > packet: git< 0000\n\nSide note: a filter may return length 0.\nSuggestion: add 2 test cases.\n\n\n > ------------------------\n > Please note: In a future version of Git the capability \"stream\"\n > might be supported. In that case the content size must not be\n > part of the filter response.\n >\n > Afterwards the filter is expected to wait for the next command.\n >\n > Signed-off-by: Lars Schneider \n > Helped-by: Martin-Louis Bright \n > Signed-off-by: Lars Schneider \n > ---\n > Documentation/gitattributes.txt | 54 +++++++-\n > convert.c | 269 \n++++++++++++++++++++++++++++++++++++++--\n > t/t0021-conversion.sh | 175 ++++++++++++++++++++++++++\n > t/t0021/rot13-filter.pl | 146 ++++++++++++++++++++++\n > 4 files changed, 631 insertions(+), 13 deletions(-)\n > create mode 100755 t/t0021/rot13-filter.pl\n >\n > diff --git a/Documentation/gitattributes.txt \nb/Documentation/gitattributes.txt\n > index 8882a3e..8fb40d2 100644\n > --- a/Documentation/gitattributes.txt\n > +++ b/Documentation/gitattributes.txt\n > @@ -300,7 +300,11 @@ checkout, when the `smudge` command is \nspecified, the command is\n > fed the blob object from its standard input, and its standard\n > output is used to update the worktree file. Similarly, the\n > `clean` command is used to convert the contents of worktree file\n > -upon checkin.\n > +upon checkin. By default these commands process only a single\n > +blob and terminate. If a long running filter process (see section\n > +below) is used then Git can process all blobs with a single filter\n > +invocation for the entire life of a single Git command (e.g.\n > +`git add .`).\n >\n > One use of the content filtering is to massage the content into a shape\n > that is more convenient for the platform, filesystem, and the user \nto use.\n > @@ -375,6 +379,54 @@ substitution. For example:\n > ------------------------\n >\n >\n > +Long Running Filter Process\n > +^^^^^^^^^^^^^^^^^^^^^^^^^^^\n > +\n > +If the filter command (string value) is defined via\n > +filter..process then Git can process all blobs with a\n > +single filter invocation for the entire life of a single Git\n > +command by talking with the following packet format (pkt-line)\n > +based protocol over standard input and standard output.\n > +\n > +Git starts the filter on first usage and expects a welcome\n > +message, protocol version number, and filter capabilities\n > +seperated by spaces:\n > +------------------------\n > +packet: git< git-filter-protocol\n > +packet: git< version 2\n > +packet: git< clean smudge\n > +------------------------\n > +Supported filter capabilities are \"clean\" and \"smudge\".\n > +\n > +Afterwards Git sends a command (e.g. \"smudge\" or \"clean\" - based\n > +on the supported capabilities), the filename, the content size as\n > +ASCII number in bytes, and the content in packet format with a\n > +flush packet at the end:\n > +------------------------\n > +packet: git> smudge\n > +packet: git> testfile.dat\n > +packet: git> 7\n > +packet: git> CONTENT\n > +packet: git> 0000\n > +------------------------\n > +\n > +The filter is expected to respond with the result content size as\n > +ASCII number in bytes and the result content in packet format with\n > +a flush packet at the end:\n > +------------------------\n > +packet: git< 57\n > +packet: git< SMUDGED_CONTENT\n > +packet: git< 0000\n > +------------------------\n > +Please note: In a future version of Git the capability \"stream\"\n > +might be supported. In that case the content size must not be\n > +part of the filter response.\n > +\n > +Afterwards the filter is expected to wait for the next command.\n > +A demo implementation can be found in `t/t0021/rot13-filter.pl`\n > +located in the Git core repository.\n > +\n > +\n > Interaction between checkin/checkout attributes\n > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n >\n > diff --git a/convert.c b/convert.c\n > index 522e2c5..5ff200b 100644\n > --- a/convert.c\n > +++ b/convert.c\n > @@ -3,6 +3,7 @@\n > #include \"run-command.h\"\n > #include \"quote.h\"\n > #include \"sigchain.h\"\n > +#include \"pkt-line.h\"\n >\n > /*\n > * convert.c - convert a file when checking it out and checking it in.\n > @@ -481,11 +482,232 @@ static int apply_filter(const char *path, \nconst char *src, size_t len, int fd,\n > \treturn ret;\n > }\n >\n > +static off_t multi_packet_read(struct strbuf *sb, const int fd, \nconst size_t size)\n > +{\n > +\toff_t bytes_read;\n > +\toff_t total_bytes_read = 0;\n > +\tstrbuf_grow(sb, size + 1);\t// we need one extra byte for the packet \nflush\n > +\tdo {\n > +\t\tbytes_read = packet_read(\n > +\t\t\tfd, NULL, NULL,\n > +\t\t\tsb->buf + total_bytes_read, sb->len - total_bytes_read - 1,\n > +\t\t\tPACKET_READ_GENTLE_ON_EOF\n > +\t\t);\n > +\t\ttotal_bytes_read += bytes_read;\n > +\t}\n > +\twhile (\n > +\t\tbytes_read > 0 && \t\t\t\t\t// the last packet was no flush\n > +\t\tsb->len - total_bytes_read - 1 > 0 \t// we still have space left in \nthe buffer\n > +\t);\n > +\tstrbuf_setlen(sb, total_bytes_read);\n > +\treturn total_bytes_read;\n > +}\n > +\n > +static int multi_packet_write(const char *src, size_t len, const int \nin, const int out)\n > +{\n > +\tint ret = 1;\n > +\tchar header[4];\n > +\tchar buffer[8192];\n > +\toff_t bytes_to_write;\n > +\twhile (ret) {\n > +\t\tif (in >= 0) {\n > +\t\t\tbytes_to_write = xread(in, buffer, sizeof(buffer));\n > +\t\t\tif (bytes_to_write < 0)\n > +\t\t\t\tret &= 0;\n > +\t\t\tsrc = buffer;\n > +\t\t} else {\n > +\t\t\tbytes_to_write = len > LARGE_PACKET_MAX - 4 ? LARGE_PACKET_MAX - \n4 : len;\n > +\t\t\tlen -= bytes_to_write;\n > +\t\t}\n > +\t\tif (!bytes_to_write)\n > +\t\t\tbreak;\n > +\t\tset_packet_header(header, bytes_to_write + 4);\n > +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n > +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n > +\t}\n > +\tret &= write_in_full(out, \"0000\", 4) == 4;\n > +\treturn ret;\n > +}\n > +\n > +struct cmd2process {\n > +\tstruct hashmap_entry ent; /* must be the first member! */\n > +\tconst char *cmd;\n > +\tint clean;\n > +\tint smudge;\n > +\tstruct child_process process;\n > +};\n > +\n > +static int cmd2process_cmp(const struct cmd2process *e1,\n > +\t\t\t\t\t\t\tconst struct cmd2process *e2,\n > +\t\t\t\t\t\t\tconst void *unused)\n > +{\n > +\treturn strcmp(e1->cmd, e2->cmd);\n > +}\n > +\n > +static struct cmd2process *find_protocol_filter_entry(struct hashmap \n*hashmap, const char *cmd)\n > +{\n > +\tstruct cmd2process k;\n > +\thashmap_entry_init(&k, strhash(cmd));\n > +\tk.cmd = cmd;\n > +\treturn hashmap_get(hashmap, &k, NULL);\n > +}\n > +\n > +static void stop_protocol_filter(struct hashmap *hashmap, struct \ncmd2process *entry) {\n > +\tif (!entry)\n > +\t\treturn;\n > +\tsigchain_push(SIGPIPE, SIG_IGN);\n > +\tclose(entry->process.in);\n > +\tclose(entry->process.out);\n > +\tsigchain_pop(SIGPIPE);\n > +\tfinish_command(&entry->process);\n > +\tchild_process_clear(&entry->process);\n > +\thashmap_remove(hashmap, entry, NULL);\n > +\tfree(entry);\n > +}\n > +\n > +static struct cmd2process *start_protocol_filter(struct hashmap \n*hashmap, const char *cmd)\n > +{\n > +\tint ret = 1;\n > +\tstruct cmd2process *entry;\n > +\tstruct child_process *process;\n > +\tconst char *argv[] = { NULL, NULL };\n > +\tstruct string_list capabilities = STRING_LIST_INIT_NODUP;\n > +\tchar *capabilities_buffer;\n > +\tint i;\n > +\n > +\tentry = xmalloc(sizeof(*entry));\n > +\thashmap_entry_init(entry, strhash(cmd));\n > +\tentry->cmd = cmd;\n > +\tentry->clean = 0;\n > +\tentry->smudge = 0;\n > +\tprocess = &entry->process;\n > +\n > +\tchild_process_init(process);\n > +\targv[0] = cmd;\n > +\tprocess->argv = argv;\n > +\tprocess->use_shell = 1;\n > +\tprocess->in = -1;\n > +\tprocess->out = -1;\n > +\n > +\tif (start_command(process)) {\n > +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n > +\t\tstop_protocol_filter(hashmap, entry);\n > +\t\treturn NULL;\n > +\t}\n > +\n > +\tsigchain_push(SIGPIPE, SIG_IGN);\n > +\tret &= strcmp(packet_read_line(process->out, NULL), \n\"git-filter-protocol\") == 0;\n > +\tret &= strcmp(packet_read_line(process->out, NULL), \"version 2\") == 0;\n > +\tcapabilities_buffer = packet_read_line(process->out, NULL);\n > +\tsigchain_pop(SIGPIPE);\n > +\n > +\tstring_list_split_in_place(&capabilities, capabilities_buffer, ' ', \n-1);\n > +\tfor (i = 0; i < capabilities.nr; i++) {\n > +\t\tconst char *requested = capabilities.items[i].string;\n > +\t\tif (!strcmp(requested, \"clean\")) {\n > +\t\t\tentry->clean = 1;\n > +\t\t} else if (!strcmp(requested, \"smudge\")) {\n > +\t\t\tentry->smudge = 1;\n > +\t\t} else {\n > +\t\t\twarning(\n > +\t\t\t\t\"filter process '%s' requested unsupported filter capability '%s'\",\n > +\t\t\t\tcmd, requested\n > +\t\t\t);\n > +\t\t}\n > +\t}\n > +\tstring_list_clear(&capabilities, 0);\n > +\n > +\tif (!ret) {\n > +\t\terror(\"initialization for external persistent filter '%s' failed\", \ncmd);\n > +\t\tstop_protocol_filter(hashmap, entry);\n > +\t\treturn NULL;\n > +\t}\n > +\n > +\thashmap_add(hashmap, entry);\n > +\treturn entry;\n > +}\n > +\n > +static int cmd_process_map_init = 0;\n > +static struct hashmap cmd_process_map;\n > +\n > +static int apply_protocol_filter(const char *path, const char *src, \nsize_t len,\n > +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n > +\t\t\t\t\t\tconst char *filter_type)\n > +{\n > +\tint ret = 1;\n > +\tstruct cmd2process *entry;\n > +\tstruct child_process *process;\n > +\tstruct stat file_stat;\n > +\tstruct strbuf nbuf = STRBUF_INIT;\n > +\toff_t expected_bytes;\n > +\tchar *strtol_end;\n > +\tchar *strbuf;\n > +\n > +\tif (!cmd || !*cmd)\n > +\t\treturn 0;\n > +\n > +\tif (!dst)\n > +\t\treturn 1;\n > +\n > +\tif (!cmd_process_map_init) {\n > +\t\tcmd_process_map_init = 1;\n > +\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n > +\t\tentry = NULL;\n > +\t} else {\n > +\t\tentry = find_protocol_filter_entry(&cmd_process_map, cmd);\n > +\t}\n > +\n > +\tif (!entry) {\n > +\t\tentry = start_protocol_filter(&cmd_process_map, cmd);\n > +\t\tif (!entry) {\n > +\t\t\treturn 0;\n > +\t\t}\n > +\t}\n > +\tprocess = &entry->process;\n > +\n > +\tif (!(!strcmp(filter_type, \"clean\") && entry->clean) &&\n > +\t\t!(!strcmp(filter_type, \"smudge\") && entry->smudge)) {\n > +\t\treturn 0;\n > +\t}\n > +\n > +\tif (fd >= 0 && !src) {\n > +\t\tret &= fstat(fd, &file_stat) != -1;\n > +\t\tlen = file_stat.st_size;\n > +\t}\n > +\n > +\tsigchain_push(SIGPIPE, SIG_IGN);\n > +\n > +\tpacket_write(process->in, \"%s\\n\", filter_type);\n > +\tpacket_write(process->in, \"%s\\n\", path);\n > +\tpacket_write(process->in, \"%zu\\n\", len);\n%zu is not portable, e.g. Windows\nIt may be better to cast len to an unsigned int64\nand use\npacket_write(process->in, \"%\" PRId64 \"\\n\", (uint64_t)len);\n\nBut this is not ideal either, it could be\n\npacket_write(process->in, \"%\" PRIdMAX \"\\n\", (uintmax_t)len);\n\n\n > +\tret &= multi_packet_write(src, len, fd, process->in);\n > +\n > +\tstrbuf = packet_read_line(process->out, NULL);\n > +\texpected_bytes = (off_t)strtol(strbuf, &strtol_end, 10);\n\nA long is 32 bit under Windows. It could be better to use:\n gitstrtodmax (const char *nptr, char **endptr, int base)\n\n > +\tret &= (strtol_end != strbuf && errno != ERANGE);\n > +\n > +\tif (expected_bytes > 0) {\nAs pointed out earlier, it is possible that the result of\na filter gives a len of 0, which should be handled.\n\n\n > +\t\tret &= multi_packet_read(&nbuf, process->out, expected_bytes) == \nexpected_bytes;\n > +\t}\n > +\n > +\tsigchain_pop(SIGPIPE);\n > +\n > +\tif (ret) {\n > +\t\tstrbuf_swap(dst, &nbuf);\n > +\t} else {\n > +\t\t// Something went wrong with the protocol filter. Force shutdown!\n > +\t\tstop_protocol_filter(&cmd_process_map, entry);\n\nIs there a warning() missing ?\n > +\t}\n > +\tstrbuf_release(&nbuf);\n > +\treturn ret;\n > +}\n > +\n > static struct convert_driver {\n > \tconst char *name;\n > \tstruct convert_driver *next;\n > \tconst char *smudge;\n > \tconst char *clean;\n > +\tconst char *process;\n > \tint required;\n > } *user_convert, **user_convert_tail;\n >\n > @@ -526,6 +748,10 @@ static int read_convert_config(const char *var, \nconst char *value, void *cb)\n > \tif (!strcmp(\"clean\", key))\n > \t\treturn git_config_string(&drv->clean, var, value);\n >\n > +\tif (!strcmp(\"process\", key)) {\n > +\t\treturn git_config_string(&drv->process, var, value);\n > +\t}\n > +\n > \tif (!strcmp(\"required\", key)) {\n > \t\tdrv->required = git_config_bool(var, value);\n > \t\treturn 0;\n > @@ -823,7 +1049,10 @@ int would_convert_to_git_filter_fd(const char \n*path)\n > \tif (!ca.drv->required)\n > \t\treturn 0;\n >\n > -\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n > +\tif (!ca.drv->clean && ca.drv->process)\n > +\t\treturn apply_protocol_filter(path, NULL, 0, -1, NULL, \nca.drv->process, \"clean\");\n > +\telse\n > +\t\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n > }\n >\n > const char *get_convert_attr_ascii(const char *path)\n > @@ -856,17 +1085,22 @@ int convert_to_git(const char *path, const \nchar *src, size_t len,\n > struct strbuf *dst, enum safe_crlf checksafe)\n > {\n > \tint ret = 0;\n > -\tconst char *filter = NULL;\n > +\tconst char *clean_filter = NULL;\n > +\tconst char *process_filter = NULL;\n > \tint required = 0;\n > \tstruct conv_attrs ca;\n >\n > \tconvert_attrs(&ca, path);\n > \tif (ca.drv) {\n > -\t\tfilter = ca.drv->clean;\n > +\t\tclean_filter = ca.drv->clean;\n > +\t\tprocess_filter = ca.drv->process;\n > \t\trequired = ca.drv->required;\n > \t}\n >\n > -\tret |= apply_filter(path, src, len, -1, dst, filter);\n > +\tif (!clean_filter && process_filter)\n > +\t\tret |= apply_protocol_filter(path, src, len, -1, dst, \nprocess_filter, \"clean\");\n > +\telse\n > +\t\tret |= apply_filter(path, src, len, -1, dst, clean_filter);\n > \tif (!ret && required)\n > \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n >\n > @@ -885,13 +1119,19 @@ int convert_to_git(const char *path, const \nchar *src, size_t len,\n > void convert_to_git_filter_fd(const char *path, int fd, struct \nstrbuf *dst,\n > \t\t\t enum safe_crlf checksafe)\n > {\n > +\tint ret = 0;\n > \tstruct conv_attrs ca;\n > \tconvert_attrs(&ca, path);\n >\n > \tassert(ca.drv);\n > -\tassert(ca.drv->clean);\n > +\tassert(ca.drv->clean || ca.drv->process);\n >\n > -\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))\n > +\tif (!ca.drv->clean && ca.drv->process)\n > +\t\tret = apply_protocol_filter(path, NULL, 0, fd, dst, \nca.drv->process, \"clean\");\n > +\telse\n > +\t\tret = apply_filter(path, NULL, 0, fd, dst, ca.drv->clean);\n > +\n > +\tif (!ret)\n > \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n >\n > \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n > @@ -902,14 +1142,16 @@ static int \nconvert_to_working_tree_internal(const char *path, const char *src,\n > \t\t\t\t\t size_t len, struct strbuf *dst,\n > \t\t\t\t\t int normalizing)\n > {\n > -\tint ret = 0, ret_filter = 0;\n > -\tconst char *filter = NULL;\n > +\tint ret = 0, ret_filter;\n > +\tconst char *smudge_filter = NULL;\n > +\tconst char *process_filter = NULL;\n > \tint required = 0;\n > \tstruct conv_attrs ca;\n >\n > \tconvert_attrs(&ca, path);\n > \tif (ca.drv) {\n > -\t\tfilter = ca.drv->smudge;\n > +\t\tprocess_filter = ca.drv->process;\n > +\t\tsmudge_filter = ca.drv->smudge;\n > \t\trequired = ca.drv->required;\n > \t}\n >\n > @@ -922,7 +1164,7 @@ static int \nconvert_to_working_tree_internal(const char *path, const char *src,\n > \t * CRLF conversion can be skipped if normalizing, unless there\n > \t * is a smudge filter. The filter might expect CRLFs.\n > \t */\n > -\tif (filter || !normalizing) {\n > +\tif (smudge_filter || process_filter || !normalizing) {\n > \t\tret |= crlf_to_worktree(path, src, len, dst, ca.crlf_action);\n > \t\tif (ret) {\n > \t\t\tsrc = dst->buf;\n > @@ -930,7 +1172,10 @@ static int \nconvert_to_working_tree_internal(const char *path, const char *src,\n > \t\t}\n > \t}\n >\n > -\tret_filter = apply_filter(path, src, len, -1, dst, filter);\n > +\tif (!smudge_filter && process_filter)\n > +\t\tret_filter = apply_protocol_filter(path, src, len, -1, dst, \nprocess_filter, \"smudge\");\n > +\telse\n > +\t\tret_filter = apply_filter(path, src, len, -1, dst, smudge_filter);\n > \tif (!ret_filter && required)\n > \t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n >\n > @@ -1383,7 +1628,7 @@ struct stream_filter *get_stream_filter(const \nchar *path, const unsigned char *s\n > \tstruct stream_filter *filter = NULL;\n >\n > \tconvert_attrs(&ca, path);\n > -\tif (ca.drv && (ca.drv->smudge || ca.drv->clean))\n > +\tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n > \t\treturn NULL;\n >\n > \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\n > diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n > index b9911a4..c4793ed 100755\n > --- a/t/t0021-conversion.sh\n > +++ b/t/t0021-conversion.sh\n > @@ -4,6 +4,11 @@ test_description='blob conversion via gitattributes'\n >\n > . ./test-lib.sh\n >\n > +if ! test_have_prereq PERL; then\n > +\tskip_all='skipping perl interface tests, perl not available'\n > +\ttest_done\n > +fi\n > +\n > if test_have_prereq EXPENSIVE\n > then\n > \tT0021_LARGE_FILE_SIZE=2048\n > @@ -283,4 +288,174 @@ test_expect_success 'disable filter with empty \noverride' '\n > \ttest_must_be_empty err\n > '\n >\n > +test_expect_success 'required protocol filter should filter data' '\n > +\ttest_config_global filter.protocol.process \n\\\"$TEST_DIRECTORY/t0021/rot13-filter.pl\\\" &&\n > +\ttest_config_global filter.protocol.required true &&\n > +\trm -rf repo &&\n > +\tmkdir repo &&\n > +\t(\n > +\t\tcd repo &&\n > +\t\tgit init &&\n > +\n > +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n > +\t\tgit add . &&\n > +\t\tgit commit . -m \"test commit\" &&\n > +\t\tgit branch empty &&\n > +\n > +\t\tcat ../test.o >test.r &&\n > +\t\techo \"test22\" >test2.r &&\n > +\t\techo \"test333\" >test3.r &&\n > +\n > +\t\trm -f output.log &&\n > +\t\tgit add . &&\n > +\t\tsort output.log | uniq -c | sed \"s/^[ ]*//\" >uniq_output.log &&\n > +\t\tcat >expected_add.log <<-\\EOF &&\n > +\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 [OK]\n > +\t\t\t1 IN: clean test2.r 7 [OK] -- OUT: 7 [OK]\n > +\t\t\t1 IN: clean test3.r 8 [OK] -- OUT: 8 [OK]\n > +\t\t\t1 start\n > +\t\t\t1 wrote filter header\n > +\t\tEOF\n > +\t\ttest_cmp expected_add.log uniq_output.log &&\n > +\n > +\t\t>output.log &&\n > +\t\tgit commit . -m \"test commit\" &&\n > +\t\tsort output.log | uniq -c | sed \"s/^[ ]*//\" |\n > +\t\t\tsed \"s/^\\([0-9]\\) IN: clean/x IN: clean/\" >uniq_output.log &&\n > +\t\tcat >expected_commit.log <<-\\EOF &&\n > +\t\t\tx IN: clean test.r 57 [OK] -- OUT: 57 [OK]\n > +\t\t\tx IN: clean test2.r 7 [OK] -- OUT: 7 [OK]\n > +\t\t\tx IN: clean test3.r 8 [OK] -- OUT: 8 [OK]\n > +\t\t\t1 start\n > +\t\t\t1 wrote filter header\n > +\t\tEOF\n > +\t\ttest_cmp expected_commit.log uniq_output.log &&\n > +\n > +\t\t>output.log &&\n > +\t\trm -f test?.r &&\n > +\t\tgit checkout . &&\n > +\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n > +\t\tcat >expected_checkout.log <<-\\EOF &&\n > +\t\t\tstart\n > +\t\t\twrote filter header\n > +\t\t\tIN: smudge test2.r 7 [OK] -- OUT: 7 [OK]\n > +\t\t\tIN: smudge test3.r 8 [OK] -- OUT: 8 [OK]\n > +\t\tEOF\n > +\t\ttest_cmp expected_checkout.log smudge_output.log &&\n > +\n > +\t\tgit checkout empty &&\n > +\n > +\t\t>output.log &&\n > +\t\tgit checkout master &&\n > +\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n > +\t\tcat >expected_checkout_master.log <<-\\EOF &&\n > +\t\t\tstart\n > +\t\t\twrote filter header\n > +\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 [OK]\n > +\t\t\tIN: smudge test2.r 7 [OK] -- OUT: 7 [OK]\n > +\t\t\tIN: smudge test3.r 8 [OK] -- OUT: 8 [OK]\n > +\t\tEOF\n > +\t\ttest_cmp expected_checkout_master.log smudge_output.log\n > +\t)\n > +'\n > +\n > +test_expect_success 'protocol filter large file' '\n > +\ttest_config_global filter.protocol.process \n\\\"$TEST_DIRECTORY/t0021/rot13-filter.pl\\\" &&\n > +\ttest_config_global filter.protocol.required true &&\n > +\trm -rf repo &&\n > +\tmkdir repo &&\n > +\t(\n > +\t\tcd repo &&\n > +\t\tgit init &&\n > +\n > +\t\techo \"*.file filter=protocol\" >.gitattributes &&\n > +\t\tcp ../generated-test-data/large.file large.file &&\n > +\t\tcp large.file large.original &&\n > +\t\t./../rot13.sh large.rot13 &&\n > +\n > +\t\tgit add large.file .gitattributes &&\n > +\t\tgit commit . -m \"test commit\" &&\n > +\n > +\t\trm -f large.file &&\n > +\t\tgit checkout -- large.file &&\n > +\t\tgit cat-file blob :large.file >actual &&\n > +\t\ttest_cmp large.rot13 actual\n > +\t)\n > +'\n > +\n > +test_expect_success 'required protocol filter should fail with clean' '\n > +\ttest_config_global filter.protocol.process \n\\\"$TEST_DIRECTORY/t0021/rot13-filter.pl\\\" &&\n > +\ttest_config_global filter.protocol.required true &&\n > +\trm -rf repo &&\n > +\tmkdir repo &&\n > +\t(\n > +\t\tcd repo &&\n > +\t\tgit init &&\n > +\n > +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n > +\n > +\t\tcat ../test.o >test.r &&\n > +\t\techo \"this is going to fail\" >clean-write-fail.r &&\n > +\t\techo \"test333\" >test3.r &&\n > +\n > +\t\t# Note: There are three clean paths in convert.c we just test one \nhere.\n > +\t\ttest_must_fail git add .\n > +\t)\n > +'\n > +\n > +test_expect_success 'protocol filter should restart after failure' '\n > +\ttest_config_global filter.protocol.process \n\\\"$TEST_DIRECTORY/t0021/rot13-filter.pl\\\" &&\n > +\trm -rf repo &&\n > +\tmkdir repo &&\n > +\t(\n > +\t\tcd repo &&\n > +\t\tgit init &&\n > +\n > +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n > +\n > +\t\tcat ../test.o >test.r &&\n > +\t\techo \"1234567\" >test2.o &&\n > +\t\tcat test2.o >test2.r &&\n > +\t\techo \"this is going to fail\" >smudge-write-fail.o &&\n > +\t\tcat smudge-write-fail.o >smudge-write-fail.r &&\n > +\t\tgit add . &&\n > +\t\tgit commit . -m \"test commit\" &&\n > +\t\trm -f *.r &&\n > +\n > +\t\tprintf \"\" >output.log &&\n > +\t\tgit checkout . &&\n > +\t\tcat output.log | grep -v \"IN: clean\" >smudge_output.log &&\n > +\t\tcat >expected_checkout_master.log <<-\\EOF &&\n > +\t\t\tstart\n > +\t\t\twrote filter header\n > +\t\t\tIN: smudge smudge-write-fail.r 22 [OK] -- OUT: 22 [FAIL]\n > +\t\t\tstart\n > +\t\t\twrote filter header\n > +\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 [OK]\n > +\t\t\tIN: smudge test2.r 8 [OK] -- OUT: 8 [OK]\n > +\t\tEOF\n > +\t\ttest_cmp expected_checkout_master.log smudge_output.log &&\n > +\n > +\t\ttest_cmp ../test.o test.r &&\n > +\t\t./../rot13.sh <../test.o >expected &&\n > +\t\tgit cat-file blob :test.r >actual &&\n > +\t\ttest_cmp expected actual\n > +\n > +\t\ttest_cmp test2.o test2.r &&\n > +\t\t./../rot13.sh expected &&\n > +\t\tgit cat-file blob :test2.r >actual &&\n > +\t\ttest_cmp expected actual\n > +\n > +\t\ttest_cmp test2.o test2.r &&\n > +\t\t./../rot13.sh expected &&\n > +\t\tgit cat-file blob :test2.r >actual &&\n > +\t\ttest_cmp expected actual\n > +\n > +\t\t! test_cmp smudge-write-fail.o smudge-write-fail.r && # Smudge failed!\n > +\t\t./../rot13.sh expected &&\n > +\t\tgit cat-file blob :smudge-write-fail.r >actual &&\n > +\t\ttest_cmp expected actual\t\t\t\t\t\t\t # Clean worked!\n > +\t)\n > +'\n > +\n > test_done\n > diff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\n > new file mode 100755\n > index 0000000..7176836\n > --- /dev/null\n > +++ b/t/t0021/rot13-filter.pl\n > @@ -0,0 +1,146 @@\n > +#!/usr/bin/perl\n > +#\n > +# Example implementation for the Git filter protocol version 2\n > +# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n > +#\n > +# This implementation supports two special test cases:\n > +# (1) If data with the filename \"clean-write-fail.r\" is processed with\n > +# a \"clean\" operation then the write operation will die.\n > +# (2) If data with the filename \"smudge-write-fail.r\" is processed with\n > +# a \"smudge\" operation then the write operation will die.\n > +#\n > +\n > +use strict;\n > +use warnings;\n > +\n > +my $MAX_PACKET_CONTENT_SIZE = 65516;\n > +\n > +sub rot13 {\n > + my ($str) = @_;\n > + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n > + return $str;\n > +}\n > +\n > +sub packet_read {\n > + my $buffer;\n > + my $bytes_read = read STDIN, $buffer, 4;\n > + if ( $bytes_read == 0 ) {\n > + return;\n > + }\n > + elsif ( $bytes_read != 4 ) {\n > + die \"invalid packet size '$bytes_read' field\";\n > + }\n > + my $pkt_size = hex($buffer);\n > + if ( $pkt_size == 0 ) {\n > + return ( 1, \"\" );\n > + }\n > + elsif ( $pkt_size > 4 ) {\n > + my $content_size = $pkt_size - 4;\n > + $bytes_read = read STDIN, $buffer, $content_size;\n > + if ( $bytes_read != $content_size ) {\n > + die \"invalid packet\";\n > + }\n > + return ( 0, $buffer );\n > + }\n > + else {\n > + die \"invalid packet size\";\n > + }\n > +}\n > +\n > +sub packet_write {\n > + my ($packet) = @_;\n > + print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n > + print STDOUT $packet;\n > + STDOUT->flush();\n > +}\n > +\n > +sub packet_flush {\n > + print STDOUT sprintf( \"%04x\", 0 );\n > + STDOUT->flush();\n > +}\n > +\n > +open my $debug, \">>\", \"output.log\";\n > +print $debug \"start\\n\";\n > +$debug->flush();\n > +\n > +packet_write(\"git-filter-protocol\\n\");\n > +packet_write(\"version 2\\n\");\n > +packet_write(\"clean smudge\\n\");\n > +print $debug \"wrote filter header\\n\";\n > +$debug->flush();\n > +\n > +while (1) {\n > + my $command = packet_read();\n > + unless ( defined($command) ) {\n > + exit();\n > + }\n > + chomp $command;\n > + print $debug \"IN: $command\";\n > + $debug->flush();\n > + my $filename = packet_read();\n > + chomp $filename;\n > + print $debug \" $filename\";\n > + $debug->flush();\n > + my $filelen = packet_read();\n > + chomp $filelen;\n > + print $debug \" $filelen\";\n > + $debug->flush();\n > +\n > + $filelen =~ /\\A\\d+\\z/ or die \"bad filelen: $filelen\";\n > + my $output;\n > +\n > + if ( $filelen > 0 ) {\n > + my $input = \"\";\n > + {\n > + binmode(STDIN);\n > + my $buffer;\n > + my $done = 0;\n > + while ( !$done ) {\n > + ( $done, $buffer ) = packet_read();\n > + $input .= $buffer;\n > + }\n > + print $debug \" [OK] -- \";\n > + $debug->flush();\n > + }\n > +\n > + if ( $command eq \"clean\" ) {\n > + $output = rot13($input);\n > + }\n > + elsif ( $command eq \"smudge\" ) {\n > + $output = rot13($input);\n > + }\n > + else {\n > + die \"bad command\";\n > + }\n > + }\n > +\n > + my $output_len = length($output);\n > + packet_write(\"$output_len\\n\");\n > + print $debug \"OUT: $output_len \";\n > + $debug->flush();\n > + if ( $output_len > 0 ) {\n > + if ( ( $command eq \"clean\" and $filename eq \n\"clean-write-fail.r\" )\n > + or\n > + ( $command eq \"smudge\" and $filename eq \n\"smudge-write-fail.r\" ) )\n > + {\n > + print $debug \" [FAIL]\\n\";\n > + $debug->flush();\n > + die \"write error\";\n > + }\n > + else {\n > + while ( length($output) > 0 ) {\n > + my $packet = substr( $output, 0, \n$MAX_PACKET_CONTENT_SIZE );\n > + packet_write($packet);\n > + if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n > + $output = substr( $output, \n$MAX_PACKET_CONTENT_SIZE );\n > + }\n > + else {\n > + $output = \"\";\n > + }\n > + }\n > + packet_flush();\n > + print $debug \"[OK]\\n\";\n > + $debug->flush();\n > + }\n > + }\n > +}\n >\n\n"},{"id":"292393","messageId":"5799E1A0.2070902@gmail.com","threadId":"42904","inReplyTo":"BFED7ED8-40DB-46B5-A1B7-4F49624D5A62@gmail.com","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-28T10:42:40Z","receivedAt":"2016-07-28T10:43:04Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-28 o 09:16, Lars Schneider pisze:\n>> On 27 Jul 2016, at 21:08, Jakub Narębski wrote:\n>> W dniu 2016-07-27 o 02:06, larsxschneider@gmail.com pisze:\n>>>\n>>> thanks a lot for the extensive reviews. I tried to address all mentioned\n>>> concerns and summarized them below. The most prominent changes since v1 are\n>>> the following:\n>>> * Git offers a number of filter capabilities that a filter can request\n>>> (right now only \"smudge\" and \"clean\" - in the future maybe \"cleanFromFile\",\n>>> \"smudgeToFile\", and/or \"stream\")\n>>> * pipe communication uses a packet format (pkt-line) based protocol\n>>\n>> I wonder if it would make sense to support both whole-file pipe communication,\n>> and packet format (pkt-line) pipe communication.\n>>\n>> The problem with whole-file pipe communication (original proposal for\n>> new filter protocol is that it needs file size upfront. For some types\n>> of filters it is not a problem:\n>> - if a filtered file has the same size as original, like for rot13\n>> example in the test for the feature\n>> - if you can calculate the resulting file size from original size,\n>> like for most if not all encryption formats (that includes GPG,\n>> uudecode, base64, quoted-printable, hex, etc.); same for decryption,\n>> and from converting between fixed-width encodings\n>> - if resulting file size is saved somewhere that is easy to get, like\n>> for LFS solutions (I think).\n>>\n>> For other filters it is serious problem. For example indent, keyword\n>> expansion, rezipping with zero compression (well, maybe not this one,\n>> but at least the reverse of it), converting between encodings where\n>> at least one is variable width (like UTF-8),...\n>>\n>> IMHO writing whole-file persistent filters is easier than using pkt-line.\n>> On the other hand using pkt-line allow for more detailed progress report.\n> \n> I initially wanted to support only \"while-file\" pipe, too.\n> \n> But Peff ($gmane/299902), Duy, and Eric, seemed to prefer the pkt-line\n> solution (gmane is down - otherwise I would have given you the links).\n\nAs GMane is down (at least the web interface; NNTP seems to be running)\nI cannot examine their arguments. Could you summarize?\n\n> \n> After I have looked at it I think the pkt-line solution is indeed nicer\n> for the following reasons:\n> \n> (1) A stream optimized version (read/write in separate threads) of the filter\n> protocol can be implemented in the future without changing the protocol\n\nI think the more important thing is that with pkt-line the filter does\nnot need to know the size of the output upfront. Separate threads are\nindependent of protocol used, I think; and anyway Git never writes to\nfilter and reads from filter in the same command, isn't it? The lifetime\nof filter driver command is one Git command for now.\n\nOh, you meant having separate threads for writing to filter, and\nseparate thread for receiving output, so you don't have to wait to\nsend whole file to filter before starting receiving? Note that\nI think original filter implementation does it; at least async_start()\nused in it hints about that (I need to examine how it works to tell\nmore).\n\n> (2) pkt-line is a simple and easy to implement format\n\nBut it is more complicated than whole-file based protocol. You need\nto loop over packets... well you need that tool with whole-file, but\nit is covered by existing helper functions (read_in_full()). It is\neasy to redirect file descriptors (copy_fd()), while you need to\nconvert contents into packets on write side, and unpack and unsplit\non the receive side in Git.\n\nYou also need to take care documenting if trailing \"\\n\\0\", \"\\n\", \"\\0\"\nis a part of packet.\n\n> (3) Reuse of existing Git communication infrastructure\n> -> code and documentation are less surprising to people that know Git\n\nWhole-file read is not that difficult...\n\n> -> you can use GIT_TRACE_PACKET to easily inspect the\n> communication between Git and the filter process\n\n...but this is a nice advantage.\n\nIf deemed necessary, we could also reuse progress report meters from\nfetch / push side (percent, bandwidth/throughput), I guess.\n\n> (4) The overheads is neglect able (4 byte header vs 65516 byte content)\n\nRight.\n\n> \n> \n>>> * a long running filter application is defined with \"filter..process\"\n>>\n>> I hope that won't confuse Git users into trying to use single-shot\n>> filters with a new protocol...\n> \n> Yes, that was my intention for this new config.\n\nAll right, but you need to document the precedence rules: that is \n\n(1) if, accordingly to the operation, `clean` or `smudge` per-file filter\n is present, it is used; \n(2) if `process` semi-persistent protocol based filter is present,\n and it offers, accordingly to the operation, `clean` or `smudge`\n capabilities, it is used;\n(3) otherwise, no filtering is performed. \n\n`clean` or `smudge` set to empty string means identity filter; I don't know\nabout rule for the new `process` filter if it is set to empty string.\n\nAt least in the commit message you would need to describe why this particular\nsolution was chosen. I guess it is to avoid starting `protocol` filter,\nonly to realize that it does not offer \"smudge\" capability, and that `smudge`\nfilter is to be used because it is set.\n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"292397","messageId":"9AB58AFB-7533-4897-8497-187C6D1239C8@gmail.com","threadId":"42904","inReplyTo":"20160727181148.GC32219@sigill.intra.peff.net","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-28T12:10:12Z","receivedAt":"2016-07-28T12:10:21Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Jul 2016, at 20:11, Jeff King wrote:\n> \n> On Wed, Jul 27, 2016 at 07:31:26PM +0200, Lars Schneider wrote:\n> \n>>>> +\tstrbuf_grow(sb, size + 1);\t// we need one extra byte for the packet flush\n>>> \n>>> What happens if size is the maximum for size_t here (i.e., 4GB-1 on a\n>>> 32-bit system)?\n>> \n>> Would that be an acceptable solution?\n>> \n>> if (size + 1 > SIZE_MAX)\n>> \treturn die(\"unrepresentable length for filter buffer\");\n> \n> No, because by definition \"size\" will wrap to 0. :)\n> \n> You have to do:\n> \n> if (size > SIZE_MAX - 1)\n> \tdie(\"whoops\");\n> \n>> Can you point me to an example in the Git source how this kind of thing should\n>> be handled?\n> \n> The strbuf code itself checks for overflows. So you could do:\n> \n> strbuf_grow(sb, size);\n> ... fill up with size bytes ...\n> strbuf_addch(sb, ...); /* extra byte for whatever */\n> \n> That does mean _possibly_ making a second allocation just to add the\n> extra byte, but in practice it's not likely (unless the input exactly\n> matches the strbuf's growth pattern).\n> \n> If you want to do it yourself, I think:\n> \n> strbuf_grow(sb, st_add(size, 1));\n\nI like that solution! Thanks!\n\n\n> would work.\n> \n>>>> +\twhile (\n>>>> +\t\tbytes_read > 0 && \t\t\t\t\t// the last packet was no flush\n>>>> +\t\tsb->len - total_bytes_read - 1 > 0 \t// we still have space left in the buffer\n>>>> +\t);\n>>> \n>>> And I'm not sure if you need to distinguish between \"0\" and \"-1\" when\n>>> checking byte_read here.\n>> \n>> We want to finish reading in both cases, no?\n> \n> If we get \"-1\", that's from an unexpected EOF during the packet_read(),\n> because you set GENTLE_ON_EOF. So there's nothing left to read, and we\n> should break and return an error.\n\nRight.\n\n\n> I guess \"0\" would come from a flush packet? Why would the filter send\n> back a flush packet (unless you were using them to signal end-of-input,\n> but then why does the filter have to send back the number of bytes ahead\n> of time?).\n\nSending the bytes ahead of time (if available) might be nice for efficient\nbuffer allocation. I am changing the code so that both cases can be handled\n(size ahead of time and no size ahead of time).\n\n\n>>> Why 8K? The pkt-line format naturally restricts us to just under 64K, so\n>>> why not take advantage of that and minimize the framing overhead for\n>>> large data?\n>> \n>> I took inspiration from here for 8K MAX_IO_SIZE:\n>> https://github.com/git/git/blob/master/copy.c#L6\n>> \n>> Is this read limit correct? Should I read 8 times to fill a pkt-line?\n> \n> MAX_IO_SIZE is generally 8 _megabytes_, not 8K. The loop in copy.c just\n> haad to pick an arbitrary size for doing its read/write proxying. I\n> think in practice you are not likely to get much benefit from going\n> beyond 8K or so there, just because operating systems tend to do things\n> in page-sizes anyway, which are usually 4K.\n> \n> But since you are formatting the result into a form that has framing\n> overhead, anything up to LARGE_PACKET_MAX will see benefits (though\n> admittedly even 4 bytes per 8K is not much).\n> \n> I don't think it's worth the complexity of reading 8 times, but just\n> using a buffer size of LARGE_PACKET_MAX-4 would be the most efficient.\n> \n> I doubt it matters _that much_ in practice, but any time I see a magic\n> number I have to wonder at the \"why\". At least basing it on\n> LARGE_PACKET_MAX has some basis, whereas 8K is largely just made-up. :)\n\nSounds good. I will use LARGE_PACKET_MAX-4 !\n\n> \n>>> We do sometimes do \"ret |= something()\" but that is in cases where\n>>> \"ret\" is zero for success, and non-zero (usually -1) otherwise. Perhaps\n>>> your function's error-reporting is inverted from our usual style?\n>> \n>> I thought it makes the code easier to read and the filter doesn't care\n>> at what point the error happens anyways. The filter either succeeds\n>> or fails. What style would you suggest?\n> \n> I think that's orthogonal. I just mean that using zero for success puts\n> you in our usual style, and then accumulating errors can be done with\n> \"|=\".\n\nAh. I guess I was misguided by the way errors are currently handled\nin `apply_filter` (success = 1; failure = 0):\nhttps://github.com/git/git/blob/8c6d1f9807c67532e7fb545a944b064faff0f70b/convert.c#L437-L479\n\nI wouldn't like if the different filter protocols would use different\nerror exit codes. Would it be OK to adjust the existing `apply_filter`\nfunction in a cleanup patch?\n\n\n> I didn't look carefully at whether the accumulating style you're using\n> makes sense or not. But something like:\n> \n>>>> +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n>>>> +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n> \n> does mean that we call the second write() even if the first one failed.\n> That's a waste of time (albeit a minor one), but it also means you could\n> potentially cover up the value of \"errno\" from the first one (though in\n> practice I'd expect the second one to fail for the same reason).\n\nOh. You're right. For some reason I thought the second operator would\nnever be evaluated if the first operator is 0. Apparently that is not\nthe case for bit-wise & ... only for logical & ... thanks for the lesson!\n\n- Lars"},{"id":"292404","messageId":"20160728132906.GA21311@sigill.intra.peff.net","threadId":"42904","inReplyTo":"BFED7ED8-40DB-46B5-A1B7-4F49624D5A62@gmail.com","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-28T13:29:06Z","receivedAt":"2016-07-28T13:29:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 28, 2016 at 09:16:18AM +0200, Lars Schneider wrote:\n\n> But Peff ($gmane/299902), Duy, and Eric, seemed to prefer the pkt-line\n> solution (gmane is down - otherwise I would have given you the links).\n\nFWIW, I think there are arguments for transmitting size + content\n(namely, that it is simpler); the downside is that it doesn't allow\nstreaming.\n\nSo I think there are two viable alternatives:\n\n 1. Total size of data in ASCII decimal, newline, then that many bytes\n of content.\n\n 2. No size header, then a series of pkt-lines followed by a flush\n packet.\n\nAnd you should choose between the two based on whether it's more\nimportant to allow streaming, or more important to make the filter\nimplementations simple[1].\n\nAny solution that is in between those (like sending a size header and\nthen using pktlines anyway) is sacrificing simplicity but not getting\nthe streaming benefits.\n\n-Peff\n\n[1] I haven't thought hard enough about it to have a real opinion. My\n gut says to go with the streaming, just because we've had to\n retrofit streaming in other areas when dealing with blobs, so I\n think we'll end up there eventually. So choosing a simpler protocol\n like (1) would probably mean eventually implementing a next-version\n protocol that does (2), and having to support both.\n\nPS Jakub asked for links, but gmane is down. Here are the relevant threads:\n\n http://public-inbox.org/git/20160720134916.GB19359@sigill.intra.peff.net\n\n http://public-inbox.org/git/20160722154900.19477-1-larsxschneider%40gmail.com/t/#u\n"},{"id":"292405","messageId":"20160728133523.GB21311@sigill.intra.peff.net","threadId":"42904","inReplyTo":"9AB58AFB-7533-4897-8497-187C6D1239C8@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-28T13:35:23Z","receivedAt":"2016-07-28T13:35:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 28, 2016 at 02:10:12PM +0200, Lars Schneider wrote:\n\n> > I think that's orthogonal. I just mean that using zero for success puts\n> > you in our usual style, and then accumulating errors can be done with\n> > \"|=\".\n> \n> Ah. I guess I was misguided by the way errors are currently handled\n> in `apply_filter` (success = 1; failure = 0):\n> https://github.com/git/git/blob/8c6d1f9807c67532e7fb545a944b064faff0f70b/convert.c#L437-L479\n> \n> I wouldn't like if the different filter protocols would use different\n> error exit codes. Would it be OK to adjust the existing `apply_filter`\n> function in a cleanup patch?\n\nAh, I see.\n\nI think those return codes are a little different. They are not \"success\nor error\", but \"did convert or did not convert\" (or \"would convert\" when\nno buffer is given). And unless the filter is required, we quietly turn\nerrors into \"did not convert\" (and if it is, we die()).\n\nSo I'm not sure if changing them is a good idea. I agree with you that\nit's probably inviting confusion to have the two sets of filter\nfunctions have opposite return codes. So I think I retract my\nsuggestion. :)\n\n-Peff\n"},{"id":"292496","messageId":"579B087F.7090108@gmail.com","threadId":"42904","inReplyTo":"20160728132906.GA21311@sigill.intra.peff.net","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-29T07:40:47Z","receivedAt":"2016-07-29T07:41:12Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-28 o 15:29, Jeff King pisze:\n> On Thu, Jul 28, 2016 at 09:16:18AM +0200, Lars Schneider wrote:\n> \n>> But Peff ($gmane/299902), Duy, and Eric, seemed to prefer the pkt-line\n>> solution (gmane is down - otherwise I would have given you the links).\n> \n> FWIW, I think there are arguments for transmitting size + content\n> (namely, that it is simpler); the downside is that it doesn't allow\n> streaming.\n\nAnd that it requires for the filter to know the size of its output\nupfront (which, as I wrote, might be easy to do based on size of input\nand data stored elsewhere, or might need generating whole output to\nknow).\n\nI don't know how parallel Git is, but if it is parallel enough,\nand other limits do not apply (limited amount of CPU cores, I/O limits),\nwithout streaming new filter protocol might be slower, unless startup\ntime dominates (MS Windows?):\n\nCurrent parallel:\n\n | startup | processing 1 |\n | startup | processing 2 |\n | startup | processing 3 |\n | startup | processing 4 |\n\nProtocol v2:\n\n | startup | processing 1 | processing 2 | processing 3 | processing 4 |\n\n> \n> So I think there are two viable alternatives:\n> \n> 1. Total size of data in ASCII decimal, newline, then that many bytes\n> of content.\n> \n> 2. No size header, then a series of pkt-lines followed by a flush\n> packet.\n\n 3. Optional size header[2][3], then a series of pkt-lines followed\n by a flush packet[4].\n\n[2] Git should always provide size, because it is easy to do, and\n I think quite cheap (stored with blob, stored in index, or stat()\n on file away). Filter can provide size if it is easy to calculate,\n or approximation of size / size hint[5] - it helps to avoid\n reallocation.\n[3] It is also a place where filter can pass error conditions that\n are known before starting processing a file.\n[4] On one hand you need to catch cases where real size is larger than\n size sent upfront, or smaller than size sent upfront; on the\n other hand it might be a place where to send warnings and errors...\n unless we utilize stderr of a process (but then there is a problem\n of deadlocking, I think).\n[5] I suggest\n\n \n \"approx\" SPC \n \"unknown\"\n \"fail\"\n\n> And you should choose between the two based on whether it's more\n> important to allow streaming, or more important to make the filter\n> implementations simple[1].\n> \n> Any solution that is in between those (like sending a size header and\n> then using pktlines anyway) is sacrificing simplicity but not getting\n> the streaming benefits.\n> \n> -Peff\n> \n> [1] I haven't thought hard enough about it to have a real opinion. My\n> gut says to go with the streaming, just because we've had to\n> retrofit streaming in other areas when dealing with blobs, so I\n> think we'll end up there eventually. So choosing a simpler protocol\n> like (1) would probably mean eventually implementing a next-version\n> protocol that does (2), and having to support both.\n> \n> PS Jakub asked for links, but gmane is down. Here are the relevant threads:\n> \n> http://public-inbox.org/git/20160720134916.GB19359@sigill.intra.peff.net\n> \n> http://public-inbox.org/git/20160722154900.19477-1-larsxschneider%40gmail.com/t/#u\n> \n\n"},{"id":"292500","messageId":"7F1F1A0E-8FC3-4FBD-81AA-37786DE0EF50@gmail.com","threadId":"42904","inReplyTo":"57994436.4080308@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-29T08:04:31Z","receivedAt":"2016-07-29T08:04:47Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 28 Jul 2016, at 01:31, Jakub Narębski wrote:\n> \n> W dniu 2016-07-27 o 02:06, larsxschneider@gmail.com pisze:\n>> From: Lars Schneider \n>> \n>> Git's clean/smudge mechanism invokes an external filter process for every\n>> single blob that is affected by a filter. If Git filters a lot of blobs\n>> then the startup time of the external filter processes can become a\n>> significant part of the overall Git execution time.\n> \n> It is not strictly necessary... but do we have any benchmarks for this,\n> or is it just the feeling? That is, in what situations Git may filter\n> a large number of files (initial checkout? initial add?, switching\n> to unrelated branch? getting large files from LFS solution?, and when\n> startup time might become significant part of execution time (MS Windows?\n> fast filters?)?\n\nAll the operations you mentioned are slow because they all cause filter\nprocess invocations. I benchmarked the new filter protocol and it is \nalmost 70x faster when switching branches on my local machine (i7, SSD, \nOS X) with a test repository containing 12,000 files that need to be \nfiltered. \nDetails here: https://github.com/github/git-lfs/pull/1382\n\nBased on my previous experience with Git filter invocations I expect even\nmore dramatic results on Windows (I will benchmark this, too, as soon as\nthe list agrees on this approach).\n\n\n>> This patch adds the filter..process string option which, if used,\n> \n> String option... what are possible values? What happens if you use\n> value that is not recognized by Git (it is \"if used\", isn't it)? That's\n> not obvious from the commit message (though it might be from the docs).\n\nThen the process invocation will fail in the same way current filter\nprocess invocations fail. If \"filter..required\" is set then\nGit will fail, otherwise not. \n\n\n> What is missing is the description that it is set to a command, and\n> how it interacts with `clean` and `smudge` options.\n\nRight, I'll add that! I implemented it in a way that the \"filter..clean\" \nand \"filter..smudge\" take presence over \"filter..process\".\n\nThis allows the use of a `single-shot` clean filter and a `long-running` \nsmudge as suggested by Junio in the v1 discussion (no ref, gmane down).\n\n\n>> keeps the external filter process running and processes all blobs with\n>> the following packet format (pkt-line) based protocol over standard input\n>> and standard output.\n>> \n>> Git starts the filter on first usage and expects a welcome\n>> message, protocol version number, and filter capabilities\n>> seperated by spaces:\n> \n> s/seperated/separated/\n\nThanks!\n\n\n> Is there any handling of misconfigured one-shot filters, or would\n> they still hang the execution of a Git command?\n\nThey would still hang. Would it be sufficient to mention that in the\ndocs?\n\n\n>> ------------------------\n>> packet: git< git-filter-protocol\n>> packet: git< version 2\n>> packet: git< clean smudge\n> \n> Wouldn't \"capabilities clean smudge\" be better? Or is it the\n> \"clean smudge\" proposal easier to handle?\n\nGood suggestion! This is easy to handle and I think it mimics\nthe pack-protocol a bit more closely.\n\n\n>> ------------------------\n>> Supported filter capabilities are \"clean\" and \"smudge\".\n>> \n>> Afterwards Git sends a command (e.g. \"smudge\" or \"clean\" - based\n>> on the supported capabilities), the filename, the content size as\n>> ASCII number in bytes, and the content in packet format with a\n>> flush packet at the end:\n>> ------------------------\n>> packet: git> smudge\n>> packet: git> testfile.dat\n> \n> And here we don't have any problems with files containing embedded\n> newlines etc. Also Git should not be sending invalid file names.\n> The question remains: is it absolute file path, or basename?\n\nIt sends a path absolute in context of the Git repo (e.g. subdir/testfile.dat).\nI'll add that to the docs and I'll add a test case to demonstrate it.\n\n> \n>> packet: git> 7\n>> packet: git> CONTENT\n> \n> Can Git send file contents using more than one packet? I think\n> it should be stated upfront.\n\nOK\n\n\n>> packet: git> 0000\n>> ------------------------\n> \n> Why we need to send content size upfront? Well, it is not a problem\n> for Git, but (as I wrote in reply to the cover letter for this\n> series) might be a problem for filter scripts.\n\nI think sending it upfront is nice for buffer allocations of big files\nand it doesn't cost us anything to do it.\n\n\n>> The filter is expected to respond with the result content size as\n>> ASCII number in bytes and the result content in packet format with\n>> a flush packet at the end:\n>> ------------------------\n>> packet: git< 57\n> \n> This is not neccessary (and might be hard for scripts to do) if\n> pkt-line protocol is used.\n> \n> In short: I think pkt-line is not worth the complication on\n> the Git side and on the filter size, unless it is used for\n> streaming, or at least filter not having to calculate output\n> size upfront.\n\nAs I stated in my other response to you, I think there is value\nin having a single protocol for streaming and non-streaming\ncontent. I'll add a capability \"stream\" that doesn't require the \n\"size upfront\" answer from the filter.\n\n\n> \n>> packet: git< SMUDGED_CONTENT\n>> packet: git< 0000\n>> ------------------------\n>> Please note: In a future version of Git the capability \"stream\"\n>> might be supported. In that case the content size must not be\n>> part of the filter response.\n>> \n>> Afterwards the filter is expected to wait for the next command.\n> \n> When filter is supposed to exit, then?\n\nNever by its own. The filter is always shutdown by Git. Something\nfor the docs, I guess :-)\n\n\n>> Signed-off-by: Lars Schneider \n>> Helped-by: Martin-Louis Bright \n>> Signed-off-by: Lars Schneider \n>> ---\n>> Documentation/gitattributes.txt | 54 +++++++-\n>> convert.c | 269 ++++++++++++++++++++++++++++++++++++++--\n>> t/t0021-conversion.sh | 175 ++++++++++++++++++++++++++\n>> t/t0021/rot13-filter.pl | 146 ++++++++++++++++++++++\n>> 4 files changed, 631 insertions(+), 13 deletions(-)\n>> create mode 100755 t/t0021/rot13-filter.pl\n>> \n>> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n>> index 8882a3e..8fb40d2 100644\n>> --- a/Documentation/gitattributes.txt\n>> +++ b/Documentation/gitattributes.txt\n>> @@ -300,7 +300,11 @@ checkout, when the `smudge` command is specified, the command is\n>> fed the blob object from its standard input, and its standard\n>> output is used to update the worktree file. Similarly, the\n>> `clean` command is used to convert the contents of worktree file\n>> -upon checkin.\n>> +upon checkin. By default these commands process only a single\n>> +blob and terminate. If a long running filter process (see section\n>> +below) is used then Git can process all blobs with a single filter\n>> +invocation for the entire life of a single Git command (e.g.\n>> +`git add .`).\n> \n> Ah, all right, here we give an example.\n> \n> But, is \"blob\" term used in this document, or do we use \"file\"\n> and \"file contents\" only?\n\nBoth terms are used in the document. Blob is also already used in a\nsimilar context above.\n\n\n>> \n>> One use of the content filtering is to massage the content into a shape\n>> that is more convenient for the platform, filesystem, and the user to use.\n>> @@ -375,6 +379,54 @@ substitution. For example:\n>> ------------------------\n>> \n>> \n>> +Long Running Filter Process\n>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>> +\n>> +If the filter command (string value) is defined via\n>> +filter..process then Git can process all blobs with a\n>> +single filter invocation for the entire life of a single Git\n>> +command by talking with the following packet format (pkt-line)\n>> +based protocol over standard input and standard output.\n> \n> Ah, so now it is the name of command,\n\nCorrect!\n\n\n> and I assume it is\n> exclusive with `clean` / `smudge`, or does it only takes\n> precedence based on capabilities of filter (that is if\n> for example \"`process`\" does not include 'clean' capability,\n> then `clean` filter is used, using per-file \"protocol\").\n> Or do something different happens (like preference for\n> old-style `clean` and `smudge` filters, and `process`\n> used if any unset)?\n\nThe existing `single-shot` clean/smudge filter takes precedence\nif they are available. The reason is that we would need to\nstart the `long-running` filter to figure out what capabilities\nit has. On the upside I think that is the safest course of action\nfor existing Git installations out there.\n\nI'll add test cases to demonstrate/ensure that behavior.\n\n\n> Anyway, Git command would never (I think) run both\n> \"clean\" and \"smudge\" filters. But I might be wrong.\n\nI think Git can invoke both filters on checkout.\n\n\n> Yeah, I know this going back and forth seems like \n> bike-shedding, but designing good user-facing API\n> is very, very important.\n\nI agree and I really appreciate that you sacrifice your time to think\nit through!\n\n\n>> +\n>> +Git starts the filter on first usage and expects a welcome\n>> +message, protocol version number, and filter capabilities\n>> +seperated by spaces:\n>> +------------------------\n>> +packet: git< git-filter-protocol\n>> +packet: git< version 2\n>> +packet: git< clean smudge\n>> +------------------------\n> \n> Neither of those is terminated by end of line character,\n> that is, \"\\n\", isn't it?\n\nCorrect. Will add that.\n\n> \n>> diff --git a/convert.c b/convert.c\n>> index 522e2c5..5ff200b 100644\n>> --- a/convert.c\n>> +++ b/convert.c\n>> @@ -3,6 +3,7 @@\n>> #include \"run-command.h\"\n>> #include \"quote.h\"\n>> #include \"sigchain.h\"\n>> +#include \"pkt-line.h\"\n>> \n>> /*\n>> * convert.c - convert a file when checking it out and checking it in.\n>> @@ -481,11 +482,232 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n>> \treturn ret;\n>> }\n>> \n>> +static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n> \n> What's the purpose of this function? Is it to gather read whole\n> contents of file into strbuf? Or is it to read at most 'size'\n> bytes of file / of pkt-line stream into strbuf?\n\nThis function reads pkt-lines until a flush packet into the strbuf.\n\n\n> We probably don't want to keep the whole file in memory,\n> if possible.\n\nI understand your concern but this what the original implementation\ndoes, too, and therefore I would like to keep the behavior:\nhttps://github.com/git/git/blob/8c6d1f9807c67532e7fb545a944b064faff0f70b/convert.c#L462\n\n\n>> +{\n>> +\toff_t bytes_read;\n>> +\toff_t total_bytes_read = 0;\n>> +\tstrbuf_grow(sb, size + 1);\t// we need one extra byte for the packet flush\n> \n> Why we put packet flush into strbuf? Or is it only temporarily,\n> and we adjust that at the end... I see that it is.\n\nCorrect.\n\n\n>> +\tdo {\n>> +\t\tbytes_read = packet_read(\n>> +\t\t\tfd, NULL, NULL,\n>> +\t\t\tsb->buf + total_bytes_read, sb->len - total_bytes_read - 1,\n>> +\t\t\tPACKET_READ_GENTLE_ON_EOF\n>> +\t\t);\n>> +\t\ttotal_bytes_read += bytes_read;\n>> +\t}\n>> +\twhile (\n>> +\t\tbytes_read > 0 && \t\t\t\t\t// the last packet was no flush\n>> +\t\tsb->len - total_bytes_read - 1 > 0 \t// we still have space left in the buffer\n>> +\t);\n>> +\tstrbuf_setlen(sb, total_bytes_read);\n>> +\treturn total_bytes_read;\n>> +}\n>> +\n>> +static int multi_packet_write(const char *src, size_t len, const int in, const int out)\n> \n> What's the purpose of this function? What are those 'in' and 'out'\n> parameters? Those names do not describe them well. If they are\n> file descriptors, add fd_* prefix (or whatever Git code uses).\n> Edit: I see that's what existing code uses.\n\nOK\n\n\n> Edit: so we are reading from *src + len or from fd_in, depending on\n> whether fd_in is set to 0 or not? I guess that follows existing\n> code, where it is even worse, because it is hidden...\n\nRight\n\n\n>> +{\n>> +\tint ret = 1;\n>> +\tchar header[4];\n>> +\tchar buffer[8192];\n> \n> Could those two be in one variable? Also, 'header' or 'pkt_header'?\n> \n> Why 8192, and not LARGE_PACKET_MAX - 4?\n\nAgreed. I've changed that.\n\n\n> \n>> +\toff_t bytes_to_write;\n>> +\twhile (ret) {\n>> +\t\tif (in >= 0) {\n>> +\t\t\tbytes_to_write = xread(in, buffer, sizeof(buffer));\n>> +\t\t\tif (bytes_to_write < 0)\n>> +\t\t\t\tret &= 0;\n>> +\t\t\tsrc = buffer;\n>> +\t\t} else {\n>> +\t\t\tbytes_to_write = len > LARGE_PACKET_MAX - 4 ? LARGE_PACKET_MAX - 4 : len;\n>> +\t\t\tlen -= bytes_to_write;\n>> +\t\t}\n>> +\t\tif (!bytes_to_write)\n>> +\t\t\tbreak;\n>> +\t\tset_packet_header(header, bytes_to_write + 4);\n>> +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n>> +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n> \n> These three lines are equivalent to write_packet(), or however\n> it is named, isn't it?\n\nA little different. As I discussed with Peff elsewhere the existing write_packet\nfunctions doesn't write the content of src directly. It creates a buffer which \nwould cause unnecessary copy operations. However, as Junio noted, creating\nthese pkt-lines here is wrong. I will add an function to pkt-line.h that can create\na pkt-line without buffer.\n\n\n>> +\t}\n>> +\tret &= write_in_full(out, \"0000\", 4) == 4;\n> \n> This is equivalent to packet_flush(), or however it is named,\n> isn't it?\n\npacket_flush uses \"write_or_die\" internally. If the a non required filter fails\nthen this is no reason to die. However, using this pkt-line knowledge here is\nwrong. I will add a new function to pkt-line.h. What do you think about\n\"tolerant_packet_flush\" as function name?\n\n\n>> +\treturn ret;\n>> +}\n>> +\n>> +struct cmd2process {\n>> +\tstruct hashmap_entry ent; /* must be the first member! */\n>> +\tconst char *cmd;\n>> +\tint clean;\n>> +\tint smudge;\n> \n> These two are 'int' used as 'bool', isn't it?\n\nYes, but I realized that this becomes cumbersome with more \"bools\". I create a bitmap\nand a few macros to check and set the bitmap (seems to be the Git way for these kind\nof problems).\n\n\n>> +\tstruct child_process process;\n>> +};\n> [...]\n>> +static struct cmd2process *find_protocol_filter_entry(struct hashmap *hashmap, const char *cmd)\n> \n> Wouldn't it be more descriptive to name the first parameter\n> to this function 'cmd_hashmap', or something like that, rather\n> than plain 'hashmap' (it might be the same that is used / was\n> used for a global variable)?\n> \n> Edit: or 'cmd_process_map'.\n\nI could, but wouldn't that shadow the global variable? Wouldn't that be even\nmore confusing?\n\n\n>> +{\n>> +\tstruct cmd2process k;\n>> +\thashmap_entry_init(&k, strhash(cmd));\n>> +\tk.cmd = cmd;\n>> +\treturn hashmap_get(hashmap, &k, NULL);\n>> +}\n> \n> [...]\n>> +static struct cmd2process *start_protocol_filter(struct hashmap *hashmap, const char *cmd)\n>> +{\n>> +\tint ret = 1;\n>> +\tstruct cmd2process *entry;\n>> +\tstruct child_process *process;\n>> +\tconst char *argv[] = { NULL, NULL };\n> \n> Could we initialize it with { cmd, NULL };?\n> \n> Edit: Ah, I see that you follow filter_buffer_or_fd() example from\n> convert.c, isn't it?\n\nCorrect. But filter_buffer_or_fd() requires some extra work on cmd\nthat is not necessary here. I change it to your suggestion!\n\n\n>> +\tstruct string_list capabilities = STRING_LIST_INIT_NODUP;\n>> +\tchar *capabilities_buffer;\n>> +\tint i;\n>> +\n>> +\tentry = xmalloc(sizeof(*entry));\n>> +\thashmap_entry_init(entry, strhash(cmd));\n>> +\tentry->cmd = cmd;\n>> +\tentry->clean = 0;\n>> +\tentry->smudge = 0;\n> \n> Wouldn't\n> \n> \tentry->clean = entry->smudge = 0;\n> \n> be more readable?\n\nI changed it to a bitmap with a single int :)\n\n\n>> +\tprocess = &entry->process;\n>> +\n>> +\tchild_process_init(process);\n>> +\targv[0] = cmd;\n>> +\tprocess->argv = argv;\n>> +\tprocess->use_shell = 1;\n>> +\tprocess->in = -1;\n> \n> Maybe\n> \n> +\tprocess->in = -1;\n> \n> to align, but perhaps it is not worth it.\n\nNot aligned in the original code, therefore I wouldn't do it here.\n\n\n>> +\tprocess->out = -1;\n>> +\n>> +\tif (start_command(process)) {\n>> +\t\terror(\"cannot fork to run external persistent filter '%s'\", cmd);\n> \n> Just a question: is \"cannot fork\" the only reason why start_command()\n> might have failed there?\n> \n> Edit: Ah, I see that you follow filter_buffer_or_fd() example from\n> convert.c, again.\n\nCorrect.\n\n\n>> +\t\tstop_protocol_filter(hashmap, entry);\n>> +\t\treturn NULL;\n>> +\t}\n>> +\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tret &= strcmp(packet_read_line(process->out, NULL), \"git-filter-protocol\") == 0;\n>> +\tret &= strcmp(packet_read_line(process->out, NULL), \"version 2\") == 0;\n> \n> So that's why you need packet_read_line() to return string...\n> \n>> +\tcapabilities_buffer = packet_read_line(process->out, NULL);\n>> +\tsigchain_pop(SIGPIPE);\n>> +\n>> +\tstring_list_split_in_place(&capabilities, capabilities_buffer, ' ', -1);\n> \n> This does not modify capabilities_buffer, does it?\n\nMaybe, but this wouldn't matter as we don't use this buffer afterwards.\n\n\n>> +\tfor (i = 0; i < capabilities.nr; i++) {\n>> +\t\tconst char *requested = capabilities.items[i].string;\n>> +\t\tif (!strcmp(requested, \"clean\")) {\n>> +\t\t\tentry->clean = 1;\n>> +\t\t} else if (!strcmp(requested, \"smudge\")) {\n>> +\t\t\tentry->smudge = 1;\n>> +\t\t} else {\n>> +\t\t\twarning(\n>> +\t\t\t\t\"filter process '%s' requested unsupported filter capability '%s'\",\n>> +\t\t\t\tcmd, requested\n>> +\t\t\t);\n> \n> Nice. This makes it (somewhat) forward- and backward-compatibile.\n\nThanks :)\n\n\n>> +\t\t}\n>> +\t}\n>> +\tstring_list_clear(&capabilities, 0);\n>> +\n>> +\tif (!ret) {\n>> +\t\terror(\"initialization for external persistent filter '%s' failed\", cmd);\n> \n> Do we need more detailed information about the source of error?\n\nNot sure. The beauty of pkt-line is that someone can just set GIT_TRACE_PACKET=1 to\nsee what is going on. I feel that this is enough and we don't need to have fine grained\nerror reporting...\n\n\n>> +\t\tstop_protocol_filter(hashmap, entry);\n>> +\t\treturn NULL;\n>> +\t}\n>> +\n>> +\thashmap_add(hashmap, entry);\n>> +\treturn entry;\n>> +}\n>> +\n>> +static int cmd_process_map_init = 0;\n>> +static struct hashmap cmd_process_map;\n>> +\n>> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n>> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n>> +\t\t\t\t\t\tconst char *filter_type)\n> \n> That is... quite a lot of parameters. But I guess there is precedens.\n\nAgreed. But this mimics the original apply_protocol() interface and therefore \nI would like to keep it.\n\n\n> But I think 'fd' belongs to previous line, as it is alternative to\n> src+len.\n\nYou could also argue that fd is an alternative to src+len and belongs on a \nnew line :-) In any way... if I would move fd up then we would exceed 80 chars...\n\n\n>> +{\n>> +\tint ret = 1;\n>> +\tstruct cmd2process *entry;\n>> +\tstruct child_process *process;\n>> +\tstruct stat file_stat;\n>> +\tstruct strbuf nbuf = STRBUF_INIT;\n>> +\toff_t expected_bytes;\n>> +\tchar *strtol_end;\n>> +\tchar *strbuf;\n>> +\n>> +\tif (!cmd || !*cmd)\n>> +\t\treturn 0;\n>> +\n>> +\tif (!dst)\n>> +\t\treturn 1;\n>> +\n>> +\tif (!cmd_process_map_init) {\n>> +\t\tcmd_process_map_init = 1;\n>> +\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n>> +\t\tentry = NULL;\n> \n> Is it better than having entry NULL-initialized?\n\nI did this based on my understanding of Eric's feedback:\n\n\"Compilers complain about uninitialized variables. Blindly\nsetting them to NULL can allow them to be dereferenced;\ntriggering segfaults; especially if it's passed to a different\ncompilation unit the compiler can't see.\"\n\nSee: http://public-inbox.org/git/20160725072745.GB11634%40starla/\n\n\n>> +\t} else {\n>> +\t\tentry = find_protocol_filter_entry(&cmd_process_map, cmd);\n>> +\t}\n>> +\n>> +\tif (!entry) {\n>> +\t\tentry = start_protocol_filter(&cmd_process_map, cmd);\n> \n> Hmmm... apply_filter() uses start_async() for some reason. Why\n> it does not apply for this new kind of filter?\n\nImplementing the protocol in an async manner would be way harder\nand is only necessary if you want to support true streaming (that\nis reading and writing at the same time). This might be desired but\nis out of scope for this patch series.\n\n\n>> +\t\tif (!entry) {\n>> +\t\t\treturn 0;\n>> +\t\t}\n>> +\t}\n>> +\tprocess = &entry->process;\n>> +\n>> +\tif (!(!strcmp(filter_type, \"clean\") && entry->clean) &&\n>> +\t\t!(!strcmp(filter_type, \"smudge\") && entry->smudge)) {\n> \n> Would it be more readable as !(A || B) rather than (!A && !B)?\n\nI solved it with a bitmask - no DeMorgan anymore :)\n\n\n>> +\t\treturn 0;\n>> +\t}\n>> +\n>> +\tif (fd >= 0 && !src) {\n>> +\t\tret &= fstat(fd, &file_stat) != -1;\n>> +\t\tlen = file_stat.st_size;\n>> +\t}\n> \n> All right, so we either use src+len, or if we use fd we get\n> file size.\n\nRight.\n\n\n>> +\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\n>> +\tpacket_write(process->in, \"%s\\n\", filter_type);\n>> +\tpacket_write(process->in, \"%s\\n\", path);\n>> +\tpacket_write(process->in, \"%zu\\n\", len);\n> \n> So \"\\n\" is included in protocol?\n\nYes. Pkt-line states \"A non-binary line SHOULD BE terminated by an LF\"\n(see Documentation/technical/protocol-common.txt)\n\n\n>> +\tret &= multi_packet_write(src, len, fd, process->in);\n> \n> How git-receive-pack etc. handle multi-packet write?\n\nThis is my solution that avoids an unnecessary buffer. As mentioned\nabove I move this code to its proper place in pkt-line.h/c.\n\n\n>> +\n>> +\tstrbuf = packet_read_line(process->out, NULL);\n>> +\texpected_bytes = (off_t)strtol(strbuf, &strtol_end, 10);\n>> +\tret &= (strtol_end != strbuf && errno != ERANGE);\n>> +\n>> +\tif (expected_bytes > 0) {\n>> +\t\tret &= multi_packet_read(&nbuf, process->out, expected_bytes) == expected_bytes;\n>> +\t}\n>> +\n>> +\tsigchain_pop(SIGPIPE);\n>> +\n>> +\tif (ret) {\n>> +\t\tstrbuf_swap(dst, &nbuf);\n>> +\t} else {\n>> +\t\t// Something went wrong with the protocol filter. Force shutdown!\n>> +\t\tstop_protocol_filter(&cmd_process_map, entry);\n> \n> Some error message would be nice... or do we print in down in stack?\n\nOK, I will add an error(...).\n\n\n>> +\t}\n>> +\tstrbuf_release(&nbuf);\n>> +\treturn ret;\n>> +}\n>> +\n> \n> \n> [...]\n>> @@ -823,7 +1049,10 @@ int would_convert_to_git_filter_fd(const char *path)\n>> \tif (!ca.drv->required)\n>> \t\treturn 0;\n>> \n>> -\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n>> +\tif (!ca.drv->clean && ca.drv->process)\n>> +\t\treturn apply_protocol_filter(path, NULL, 0, -1, NULL, ca.drv->process, \"clean\");\n>> +\telse\n>> +\t\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n> \n> So the rule is: if `clean` is not set, and `process` is, try to use\n> process for cleaning. It was not clear for me from the documentation.\n\nTrue. I will add that to the docs.\n\n\n>> }\n>> \n>> const char *get_convert_attr_ascii(const char *path)\n>> @@ -856,17 +1085,22 @@ int convert_to_git(const char *path, const char *src, size_t len,\n>> struct strbuf *dst, enum safe_crlf checksafe)\n>> {\n>> \tint ret = 0;\n>> -\tconst char *filter = NULL;\n>> +\tconst char *clean_filter = NULL;\n>> +\tconst char *process_filter = NULL;\n>> \tint required = 0;\n>> \tstruct conv_attrs ca;\n>> \n>> \tconvert_attrs(&ca, path);\n>> \tif (ca.drv) {\n>> -\t\tfilter = ca.drv->clean;\n>> +\t\tclean_filter = ca.drv->clean;\n>> +\t\tprocess_filter = ca.drv->process;\n>> \t\trequired = ca.drv->required;\n>> \t}\n>> \n>> -\tret |= apply_filter(path, src, len, -1, dst, filter);\n>> +\tif (!clean_filter && process_filter)\n>> +\t\tret |= apply_protocol_filter(path, src, len, -1, dst, process_filter, \"clean\");\n>> +\telse\n>> +\t\tret |= apply_filter(path, src, len, -1, dst, clean_filter);\n> \n> And the rule is the same here, as it should.\n> \n>> \tif (!ret && required)\n>> \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n> \n> Is it a correct error message for `process`? I guess it is, as it prints\n> the name of driver, and not attempted command. Well, we might be using\n> \"process\" filter in 'clean' mode,... but that is sophistry.\n\nI see where you are coming from. However, it tries to run the \"clean\" filter \nwith the process and therefore I think the message is correct.\n\n\n> [...]\n>> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n>> index b9911a4..c4793ed 100755\n>> --- a/t/t0021-conversion.sh\n>> +++ b/t/t0021-conversion.sh\n>> @@ -4,6 +4,11 @@ test_description='blob conversion via gitattributes'\n>> \n>> . ./test-lib.sh\n>> \n>> +if ! test_have_prereq PERL; then\n>> +\tskip_all='skipping perl interface tests, perl not available'\n>> +\ttest_done\n>> +fi\n> \n> Do all tests require Perl?\n\nNo. I will add the PERL requirement explicitly to the tests that need it.\n\n\n>> +test_expect_success 'required protocol filter should filter data' '\n> [...]\n>> +test_expect_success 'protocol filter large file' '\n> [...]\n>> +test_expect_success 'required protocol filter should fail with clean' '\n> [...]\n>> +test_expect_success 'protocol filter should restart after failure' '\n> [...]\n> \n>> diff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\n>> new file mode 100755\n>> index 0000000..7176836\n>> --- /dev/null\n>> +++ b/t/t0021/rot13-filter.pl\n>> @@ -0,0 +1,146 @@\n>> +#!/usr/bin/perl\n>> +#\n>> +# Example implementation for the Git filter protocol version 2\n>> +# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n>> +#\n>> +# This implementation supports two special test cases:\n>> +# (1) If data with the filename \"clean-write-fail.r\" is processed with\n>> +# a \"clean\" operation then the write operation will die.\n>> +# (2) If data with the filename \"smudge-write-fail.r\" is processed with\n>> +# a \"smudge\" operation then the write operation will die.\n> \n> Nice.\n> \n>> +#\n>> +\n>> +use strict;\n>> +use warnings;\n>> +\n>> +my $MAX_PACKET_CONTENT_SIZE = 65516;\n>> +\n>> +sub rot13 {\n>> + my ($str) = @_;\n>> + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n>> + return $str;\n>> +}\n>> +\n>> +sub packet_read {\n>> + my $buffer;\n>> + my $bytes_read = read STDIN, $buffer, 4;\n>> + if ( $bytes_read == 0 ) {\n>> + return;\n>> + }\n>> + elsif ( $bytes_read != 4 ) {\n> \n> This is a bit untypical bracket style...\n\nWith bracket style you mean the spaces? I used PerlTidy in \nstandard mode. I will adjust it to look more like perl/Git.pm ...\n\nhttp://perltidy.sourceforge.net/\n\n> \n>> + die \"invalid packet size '$bytes_read' field\";\n>> + }\n>> + my $pkt_size = hex($buffer);\n>> + if ( $pkt_size == 0 ) {\n>> + return ( 1, \"\" );\n>> + }\n>> + elsif ( $pkt_size > 4 ) {\n>> + my $content_size = $pkt_size - 4;\n>> + $bytes_read = read STDIN, $buffer, $content_size;\n>> + if ( $bytes_read != $content_size ) {\n>> + die \"invalid packet\";\n>> + }\n>> + return ( 0, $buffer );\n>> + }\n>> + else {\n>> + die \"invalid packet size\";\n>> + }\n>> +}\n> \n> So packet reading is not that difficult...\n> \n>> +\n>> +sub packet_write {\n>> + my ($packet) = @_;\n>> + print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n>> + print STDOUT $packet;\n>> + STDOUT->flush();\n>> +}\n> \n> ...and packet write is easy.\n> \n>> +\n>> +sub packet_flush {\n>> + print STDOUT sprintf( \"%04x\", 0 );\n>> + STDOUT->flush();\n>> +}\n>> +\n>> +open my $debug, \">>\", \"output.log\";\n>> +print $debug \"start\\n\";\n>> +$debug->flush();\n>> +\n>> +packet_write(\"git-filter-protocol\\n\");\n>> +packet_write(\"version 2\\n\");\n>> +packet_write(\"clean smudge\\n\");\n>> +print $debug \"wrote filter header\\n\";\n>> +$debug->flush();\n> \n> Isn't $debug flushed automatically?\n\nMaybe, but autoflush is not explicitly enabled. I will\nenable it again (I disabled it because of Eric's comment\nbut I re-read the comment and he is only talking about\npipes).\n\nhttp://public-inbox.org/git/20160723072721.GA20875%40starla/\n\n\n>> +\n>> +while (1) {\n>> + my $command = packet_read();\n>> + unless ( defined($command) ) {\n>> + exit();\n>> + }\n>> + chomp $command;\n>> + print $debug \"IN: $command\";\n>> + $debug->flush();\n>> + my $filename = packet_read();\n>> + chomp $filename;\n>> + print $debug \" $filename\";\n>> + $debug->flush();\n>> + my $filelen = packet_read();\n>> + chomp $filelen;\n>> + print $debug \" $filelen\";\n>> + $debug->flush();\n>> +\n>> + $filelen =~ /\\A\\d+\\z/ or die \"bad filelen: $filelen\";\n>> + my $output;\n>> +\n>> + if ( $filelen > 0 ) {\n>> + my $input = \"\";\n>> + {\n>> + binmode(STDIN);\n>> + my $buffer;\n>> + my $done = 0;\n>> + while ( !$done ) {\n>> + ( $done, $buffer ) = packet_read();\n>> + $input .= $buffer;\n>> + }\n>> + print $debug \" [OK] -- \";\n>> + $debug->flush();\n>> + }\n>> +\n>> + if ( $command eq \"clean\" ) {\n>> + $output = rot13($input);\n>> + }\n>> + elsif ( $command eq \"smudge\" ) {\n>> + $output = rot13($input);\n>> + }\n>> + else {\n>> + die \"bad command\";\n> \n> Perhaps\n> \n> die \"bad command $command\";\n\nAgreed.\n\n\n>> + }\n>> + }\n>> +\n>> + my $output_len = length($output);\n>> + packet_write(\"$output_len\\n\");\n>> + print $debug \"OUT: $output_len \";\n>> + $debug->flush();\n>> + if ( $output_len > 0 ) {\n>> + if ( ( $command eq \"clean\" and $filename eq \"clean-write-fail.r\" )\n> \n> What happened here with whitespace around parentheses?\n\nPerl tidy :-)\nWill fix!\n\nThanks a lot for this extensive review, again!\n\nBest,\nLars"},{"id":"292501","messageId":"31219A33-CA8A-44D1-9DE0-6448AA1A7571@gmail.com","threadId":"42904","inReplyTo":"579B087F.7090108@gmail.com","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-29T08:14:17Z","receivedAt":"2016-07-29T08:14:38Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Jul 2016, at 09:40, Jakub Narębski wrote:\n> \n> W dniu 2016-07-28 o 15:29, Jeff King pisze:\n>> On Thu, Jul 28, 2016 at 09:16:18AM +0200, Lars Schneider wrote:\n>> \n>>> But Peff ($gmane/299902), Duy, and Eric, seemed to prefer the pkt-line\n>>> solution (gmane is down - otherwise I would have given you the links).\n>> \n>> FWIW, I think there are arguments for transmitting size + content\n>> (namely, that it is simpler); the downside is that it doesn't allow\n>> streaming.\n> \n> And that it requires for the filter to know the size of its output\n> upfront (which, as I wrote, might be easy to do based on size of input\n> and data stored elsewhere, or might need generating whole output to\n> know).\n> \n> I don't know how parallel Git is, but if it is parallel enough,\n> and other limits do not apply (limited amount of CPU cores, I/O limits),\n> without streaming new filter protocol might be slower, unless startup\n> time dominates (MS Windows?):\n> \n> Current parallel:\n> \n> | startup | processing 1 |\n> | startup | processing 2 |\n> | startup | processing 3 |\n> | startup | processing 4 |\n> \n> Protocol v2:\n> \n> | startup | processing 1 | processing 2 | processing 3 | processing 4 |\n\nBased on the current filter design the \"single-shot\" invocations are\nnot executed in parallel.\n\n\n>> So I think there are two viable alternatives:\n>> \n>> 1. Total size of data in ASCII decimal, newline, then that many bytes\n>> of content.\n>> \n>> 2. No size header, then a series of pkt-lines followed by a flush\n>> packet.\n> \n> 3. Optional size header[2][3], then a series of pkt-lines followed\n> by a flush packet[4].\n> \n> [2] Git should always provide size, because it is easy to do, and\n> I think quite cheap (stored with blob, stored in index, or stat()\n> on file away). Filter can provide size if it is easy to calculate,\n> or approximation of size / size hint[5] - it helps to avoid\n> reallocation.\n\nAgreed!\n\n\n> [3] It is also a place where filter can pass error conditions that\n> are known before starting processing a file.\n\nI am not sure I understand what you mean. Can you think of an example?\n\n\n> [4] On one hand you need to catch cases where real size is larger than\n> size sent upfront, or smaller than size sent upfront; on the\n> other hand it might be a place where to send warnings and errors...\n> unless we utilize stderr of a process (but then there is a problem\n> of deadlocking, I think).\n> [5] I suggest\n> \n> \n> \"approx\" SPC \n> \"unknown\"\n> \"fail\"\n\nMy current implementation supports only two cases. Either the filter\nknows the size and sends it back. Or the filter doesn't know the size\nand Git reads until the flush packet (your \"unknown\" case). \"Approx\" is \nprobably hard to do and fail shouldn't be part of the size, no?\n\nThat being said a \"fail\" response is a very good idea! This allows\nthe filter to communicate to git that a non required filter process\nfailed. I will add that to the protocol. Thanks :) \n\n- Lars\n\n\n"},{"id":"292506","messageId":"64C7D52F-9030-460C-8F61-4076F5C1DDF6@gmail.com","threadId":"42904","inReplyTo":"20160727094102.GA31374@starla","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-29T10:38:02Z","receivedAt":"2016-07-29T10:38:14Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Jul 2016, at 11:41, Eric Wong wrote:\n> \n> larsxschneider@gmail.com wrote:\n>> +static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n> \n> I'm no expert in C, but this might be const-correctness taken\n> too far. I think basing this on the read(2) prototype is less\n> surprising:\n> \n> static ssize_t multi_packet_read(int fd, struct strbuf *sb, size_t size)\n\nHm... ok. I like `const` because I think it is usually easier to read/understand\nfunctions that do not change their input variables. This way I can communicate\nmy intention to future people modifying this function!\n\nIf this is frowned upon in the Git community then I will add a comment to the\nCodingGuidelines and remove the const :)\n\nI agree with your reordering of the parameters, though!\n\nSpeaking of coding style... convert.c is already big and gets only bigger \nwith this patch (1720 lines). Would it make sense to add a new file \n\"convert-pipe-protocol.c\"\nor something for my additions?\n\n\n> Also what Jeff said about off_t vs size_t, but my previous\n> emails may have confused you w.r.t. off_t usage...\n> \n>> +static int multi_packet_write(const char *src, size_t len, const int in, const int out)\n> \n> Same comment about over const ints above.\n> len can probably be off_t based on what is below; but you need\n> to process the loop in ssize_t-friendly chunks.\n\nI think I would prefer to keep it an size_t because that is the\ntype we get from Git initially. The code will be more clear in v3.\n\n\n> \n>> +{\n>> +\tint ret = 1;\n>> +\tchar header[4];\n>> +\tchar buffer[8192];\n>> +\toff_t bytes_to_write;\n> \n> What Jeff said, this should be ssize_t to match read(2) and xread\n\nAgreed.\n\n\n> \n>> +\twhile (ret) {\n>> +\t\tif (in >= 0) {\n>> +\t\t\tbytes_to_write = xread(in, buffer, sizeof(buffer));\n>> +\t\t\tif (bytes_to_write < 0)\n>> +\t\t\t\tret &= 0;\n>> +\t\t\tsrc = buffer;\n>> +\t\t} else {\n>> +\t\t\tbytes_to_write = len > LARGE_PACKET_MAX - 4 ? LARGE_PACKET_MAX - 4 : len;\n>> +\t\t\tlen -= bytes_to_write;\n>> +\t\t}\n>> +\t\tif (!bytes_to_write)\n>> +\t\t\tbreak;\n> \n> The whole ret &= .. style error handling is hard-to-follow and\n> here, a source of bugs. I think the expected convention on\n> hitting errors is:\n> \n> \t1) stop whatever you're doing\n> \t2) cleanup\n> \t3) propagate the error to callers\n> \n> \"goto\" is an acceptable way of accomplishing this.\n> \n> For example, byte_to_write may still be negative at this point\n> (and interpreted as a really big number when cast to unsigned\n> size_t) and src/buffer could be stack garbage.\n\nI changed the implementation here so that the &= style\nis not necessary anymore. However, I will look into \"goto\"\nfor the other areas!\n\n\n>> +\t\tset_packet_header(header, bytes_to_write + 4);\n>> +\t\tret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);\n>> +\t\tret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;\n>> +\t}\n>> +\tret &= write_in_full(out, \"0000\", 4) == 4;\n>> +\treturn ret;\n>> +}\n>> +\n> \n>> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n>> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n>> +\t\t\t\t\t\tconst char *filter_type)\n>> +{\n> \n> \n> \n>> +\tif (fd >= 0 && !src) {\n>> +\t\tret &= fstat(fd, &file_stat) != -1;\n>> +\t\tlen = file_stat.st_size;\n> \n> Same truncation bug I noticed earlier; what I originally meant\n> is the `len' arg probably ought to be off_t, here, not size_t.\n> 32-bit x86 Linux systems have 32-bit size_t (unsigned), but\n> large file support means off_t is 64-bits (signed).\n\nOK. Would it be OK to keep size_t for this patch series?\n\n\n> Also, is it worth continuing this function if fstat fails?\n\nNo :-)\n\n\n>> +\t}\n>> +\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\n>> +\tpacket_write(process->in, \"%s\\n\", filter_type);\n>> +\tpacket_write(process->in, \"%s\\n\", path);\n>> +\tpacket_write(process->in, \"%zu\\n\", len);\n> \n> I'm not sure if \"%zu\" is portable since we don't do C99 (yet?)\n> For 64-bit signed off_t, you can probably do:\n> \n> \tpacket_write(process->in, \"%\"PRIuMAX\"\\n\", (uintmax_t)len);\n> \n> Since we don't have PRIiMAX or intmax_t, here, and a negative\n> len would be a bug (probably from failed fstat) anyways.\n\nOK. \"%zu\" is not used in the entire code base. I will go with\nyour suggestion!\n\n\n>> +\tret &= multi_packet_write(src, len, fd, process->in);\n> \n> multi_packet_write will probably fail if fstat failed above...\n\nYes. The error handling is bogus... I thought bitwise \"and\" would\nact the same way as logical \"and\" (a bit embarrassing to admit that...).\n\n\n> \n>> +\tstrbuf = packet_read_line(process->out, NULL);\n> \n> And this may just block or timeout if multi_packet_write failed.\n\nTrue, but unless there is anything easy to do I would leave that.\nOr do you think it is really necessary to introduce \"select\" and\nfriends?\n\n\nThanks a lot for your review,\nLars"},{"id":"292508","messageId":"579B3CFF.6040601@gmail.com","threadId":"42904","inReplyTo":"64C7D52F-9030-460C-8F61-4076F5C1DDF6@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-29T11:24:47Z","receivedAt":"2016-07-29T11:25:12Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-29 o 12:38, Lars Schneider pisze:\n> On 27 Jul 2016, at 11:41, Eric Wong wrote:\n>> larsxschneider@gmail.com wrote:\n\n>>> +static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n>> \n>> I'm no expert in C, but this might be const-correctness taken\n>> too far. I think basing this on the read(2) prototype is less\n>> surprising:\n>> \n>> static ssize_t multi_packet_read(int fd, struct strbuf *sb, size_t size)\n>\n> Hm... ok. I like `const` because I think it is usually easier to read/understand\n> functions that do not change their input variables. This way I can communicate\n> my intention to future people modifying this function!\n\nWell, scalar types like `size_t` are always passed by value, so here `const`\ndoesn't matter, and it makes line longer. I think library functions do not\nuse `const` for `size_t` parameters.\n\nYou are reading from the file descriptor `fd`, so it state would change.\nUsing `const` feels a bit like lying. Also, it is scalar type.\n \n[...] \n> I agree with your reordering of the parameters, though!\n> \n> Speaking of coding style... convert.c is already big and gets only bigger \n> with this patch (1720 lines). Would it make sense to add a new file \n> \"convert-pipe-protocol.c\"\n> or something for my additions?\n\nI wonder if it would be possible to enhance existing functions, instead\nof redoing them (at least in part) for per-command filter driver protocol.\n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"292509","messageId":"C6CC2E23-5E9F-4A88-B80C-FC4AC16F5BEC@gmail.com","threadId":"42904","inReplyTo":"579B3CFF.6040601@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-29T11:31:06Z","receivedAt":"2016-07-29T11:31:14Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Jul 2016, at 13:24, Jakub Narębski wrote:\n> \n> W dniu 2016-07-29 o 12:38, Lars Schneider pisze:\n>> On 27 Jul 2016, at 11:41, Eric Wong wrote:\n>>> larsxschneider@gmail.com wrote:\n> \n>>>> +static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)\n>>> \n>>> I'm no expert in C, but this might be const-correctness taken\n>>> too far. I think basing this on the read(2) prototype is less\n>>> surprising:\n>>> \n>>> static ssize_t multi_packet_read(int fd, struct strbuf *sb, size_t size)\n>> \n>> Hm... ok. I like `const` because I think it is usually easier to read/understand\n>> functions that do not change their input variables. This way I can communicate\n>> my intention to future people modifying this function!\n> \n> Well, scalar types like `size_t` are always passed by value, so here `const`\n> doesn't matter, and it makes line longer. I think library functions do not\n> use `const` for `size_t` parameters.\n> \n> You are reading from the file descriptor `fd`, so it state would change.\n> Using `const` feels a bit like lying. Also, it is scalar type.\n\nOK, since you are the second reviewer arguing against `const` I will remove it.\n\n\n> \n> [...] \n>> I agree with your reordering of the parameters, though!\n>> \n>> Speaking of coding style... convert.c is already big and gets only bigger \n>> with this patch (1720 lines). Would it make sense to add a new file \n>> \"convert-pipe-protocol.c\"\n>> or something for my additions?\n> \n> I wonder if it would be possible to enhance existing functions, instead\n> of redoing them (at least in part) for per-command filter driver protocol.\n\nI think I reused as much as possible.\n\nThanks,\nLars\n"},{"id":"292520","messageId":"20160729155740.GB29773@sigill.intra.peff.net","threadId":"42904","inReplyTo":"31219A33-CA8A-44D1-9DE0-6448AA1A7571@gmail.com","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-29T15:57:40Z","receivedAt":"2016-07-29T15:57:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 29, 2016 at 10:14:17AM +0200, Lars Schneider wrote:\n\n> My current implementation supports only two cases. Either the filter\n> knows the size and sends it back. Or the filter doesn't know the size\n> and Git reads until the flush packet (your \"unknown\" case). \"Approx\" is \n> probably hard to do and fail shouldn't be part of the size, no?\n\nAh, OK, I missed that you could handle both cases. I think that is a\nreasonable approach. It means the filter has to bother with pkt-lines,\nbut beyond that, it can choose the simple or streaming approach as\nappropriate.\n\n> That being said a \"fail\" response is a very good idea! This allows\n> the filter to communicate to git that a non required filter process\n> failed. I will add that to the protocol. Thanks :) \n\nMaybe just send \"ok \", \"ok -1\" (for streaming), or \"fail \"\nfollowed by the content? That is similar to other Git protocols, though\nI am not sure they are good models for sanity or extensibility. :)\n\nI don't know if you would want to leave room for other \"headers\" in the\nresponse, but you could also do something more HTTP-like, with a status\ncode, and arbitrary headers. And presumably git would just ignore\nheaders it doesn't know about. I think that's what Jakub's example was\nleaning towards. I'm just not sure what other headers are really useful,\nbut it does leave room for extensibility.\n\n-Peff\n"},{"id":"292526","messageId":"FA73FFDF-51D1-49FD-A24E-72A2C033E2F3@gmail.com","threadId":"42904","inReplyTo":"20160729155740.GB29773@sigill.intra.peff.net","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-29T16:20:51Z","receivedAt":"2016-07-29T16:20:58Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Jul 2016, at 17:57, Jeff King wrote:\n> \n> On Fri, Jul 29, 2016 at 10:14:17AM +0200, Lars Schneider wrote:\n> \n>> My current implementation supports only two cases. Either the filter\n>> knows the size and sends it back. Or the filter doesn't know the size\n>> and Git reads until the flush packet (your \"unknown\" case). \"Approx\" is \n>> probably hard to do and fail shouldn't be part of the size, no?\n> \n> Ah, OK, I missed that you could handle both cases. I think that is a\n> reasonable approach. It means the filter has to bother with pkt-lines,\n> but beyond that, it can choose the simple or streaming approach as\n> appropriate.\n\nRight.\n\n\n>> That being said a \"fail\" response is a very good idea! This allows\n>> the filter to communicate to git that a non required filter process\n>> failed. I will add that to the protocol. Thanks :) \n> \n> Maybe just send \"ok \", \"ok -1\" (for streaming), or \"fail \"\n> followed by the content? That is similar to other Git protocols, though\n> I am not sure they are good models for sanity or extensibility. :)\n> \n> I don't know if you would want to leave room for other \"headers\" in the\n> response, but you could also do something more HTTP-like, with a status\n> code, and arbitrary headers. And presumably git would just ignore\n> headers it doesn't know about. I think that's what Jakub's example was\n> leaning towards. I'm just not sure what other headers are really useful,\n> but it does leave room for extensibility.\n\nWell, \"ok \" wouldn't make much sense as we already transmitted\nthe size upfront I think. Right now I have implemented the following options:\n\n\"success\\n\" --> everything was alright\n\"reject\\n\" --> the filter rejected the operation but this is no error \n if \"filter..required = false\"\n --> failure that stops/restarts the filter process\n\nI don't think sending any failure reason makes sense because if a failure\nhappens then we are likely in a bad state already (that's why I restart the\nfilter process. I think the filter can report trouble on its own via stdout,\nno? I think this is what Git-LFS already does.\n\nI am working on the docs right now and afterwards I will send a v3 :-)\n\n- Lars\n"},{"id":"292531","messageId":"20160729165018.GA6553@sigill.intra.peff.net","threadId":"42904","inReplyTo":"FA73FFDF-51D1-49FD-A24E-72A2C033E2F3@gmail.com","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-29T16:50:18Z","receivedAt":"2016-07-29T16:50:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 29, 2016 at 06:20:51PM +0200, Lars Schneider wrote:\n\n> >> That being said a \"fail\" response is a very good idea! This allows\n> >> the filter to communicate to git that a non required filter process\n> >> failed. I will add that to the protocol. Thanks :) \n> > \n> > Maybe just send \"ok \", \"ok -1\" (for streaming), or \"fail \"\n> > followed by the content? That is similar to other Git protocols, though\n> > I am not sure they are good models for sanity or extensibility. :)\n> > \n> > I don't know if you would want to leave room for other \"headers\" in the\n> > response, but you could also do something more HTTP-like, with a status\n> > code, and arbitrary headers. And presumably git would just ignore\n> > headers it doesn't know about. I think that's what Jakub's example was\n> > leaning towards. I'm just not sure what other headers are really useful,\n> > but it does leave room for extensibility.\n> \n> Well, \"ok \" wouldn't make much sense as we already transmitted\n> the size upfront I think. Right now I have implemented the following options:\n\nMaybe I'm confused about where in the protocol we are. I was imagining:\n\n git> smudge\n git> \n git> \n git> ...pkt-lines...\n git> pktline-flush\n\n git< ok \n git< ...pkt-lines...\n git< flush\n\nThat is, we should say \"I have something for you\" or \"I do not\" before\nsending a size, because in the \"I do not\" case we have no size to send.\n\nA more extensible protocol might look like:\n\n git> smudge\n git> filename=\n git> size=\n git> pktline-flush\n git> ...pkt-lines of data...\n git> pktline-flush\n\n git< ok (or success, or whatever status code you like)\n git< size=\n git< pkt-line-flush\n git< ...pkt-lines of data...\n git< pktline-flush\n\nThat leaves room for new \"keys\" to be added before the first pkt-flush,\nwithout having to change the parsing at all.\n\n> \"success\\n\" --> everything was alright\n> \"reject\\n\" --> the filter rejected the operation but this is no error \n> if \"filter..required = false\"\n> --> failure that stops/restarts the filter process\n> \n> I don't think sending any failure reason makes sense because if a failure\n> happens then we are likely in a bad state already (that's why I restart the\n> filter process. I think the filter can report trouble on its own via stdout,\n> no? I think this is what Git-LFS already does.\n\nGit-LFS sends to stderr because there's no other option. I wonder if it\nwould be nicer to make it Git's responsibility to talk to the user,\nbecause then it could respect things like \"--quiet\". I guess error\nmessages are generally printed regardless of verbosity, though, so\nprinting them unconditionally is OK.\n\n-Peff\n"},{"id":"292537","messageId":"xmqqshusny86.fsf@gitster.mtv.corp.google.com","threadId":"42904","inReplyTo":"7F1F1A0E-8FC3-4FBD-81AA-37786DE0EF50@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-29T17:35:53Z","receivedAt":"2016-07-29T17:36:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n> I think sending it upfront is nice for buffer allocations of big files\n> and it doesn't cost us anything to do it.\n\nWhile I do NOT think \"total size upfront\" MUST BE avoided at all costs,\nI do not think the above statement to justify it makes ANY sense.\n\nBig files are by definition something you cannot afford to hold its\nentirety in core, so you do not want to be told that you'd be fed 40GB\nand ask xmalloc to allocate that much.\n\nIt allows the reader to be lazy for buffer allocations as long as\nyou know the file fits in-core, at the cost of forcing the writer to\nsomehow come up with the total number of bytes even before sending a\nsingle byte (in other words, if the writer cannot produce and hold\nthe data in-core, it may even have to spool the data in a temporary\nfile only to count, and then play it back after showing the total\nsize).\n\nIt is good that you allow both mode of operations and the size of\nthe data can either be given upfront (which allows a single fixed\nallocation upfront without realloc, as long as the data fits in\ncore), or be left \"(atend)\".\n\nI just don't want to see it oversold as a \"feature\" that the size\nhas to come before data. That is a limitation, not a feature.\n\nThanks.\n"},{"id":"292538","messageId":"9C2A3438-84BA-49FE-87AA-1084995A1E89@gmail.com","threadId":"42904","inReplyTo":"20160729165018.GA6553@sigill.intra.peff.net","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-29T17:43:49Z","receivedAt":"2016-07-29T17:44:01Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Jul 2016, at 18:50, Jeff King wrote:\n> \n> On Fri, Jul 29, 2016 at 06:20:51PM +0200, Lars Schneider wrote:\n> \n>>>> That being said a \"fail\" response is a very good idea! This allows\n>>>> the filter to communicate to git that a non required filter process\n>>>> failed. I will add that to the protocol. Thanks :) \n>>> \n>>> Maybe just send \"ok \", \"ok -1\" (for streaming), or \"fail \"\n>>> followed by the content? That is similar to other Git protocols, though\n>>> I am not sure they are good models for sanity or extensibility. :)\n>>> \n>>> I don't know if you would want to leave room for other \"headers\" in the\n>>> response, but you could also do something more HTTP-like, with a status\n>>> code, and arbitrary headers. And presumably git would just ignore\n>>> headers it doesn't know about. I think that's what Jakub's example was\n>>> leaning towards. I'm just not sure what other headers are really useful,\n>>> but it does leave room for extensibility.\n>> \n>> Well, \"ok \" wouldn't make much sense as we already transmitted\n>> the size upfront I think. Right now I have implemented the following options:\n> \n> Maybe I'm confused about where in the protocol we are. I was imagining:\n> \n> git> smudge\n> git> \n> git> \n> git> ...pkt-lines...\n> git> pktline-flush\n> \n> git< ok \n> git< ...pkt-lines...\n> git< flush\n> \n> That is, we should say \"I have something for you\" or \"I do not\" before\n> sending a size, because in the \"I do not\" case we have no size to send.\n\n\nRight now the protocol is like that in the happy case (non-streaming):\n\ngit> smudge\ngit> \ngit> \ngit> ...pkt-lines...\ngit> pktline-flush\n\ngit< \ngit< ...pkt-lines...\ngit< flush\ngit< success\n\n(diff to your version: no \"ok\" in front of size answer ... plus the\nsize answer is not present in the streaming case)\n\n\nHere is the reject case (non-streaming):\n\ngit> smudge\ngit> \ngit> \ngit> ...pkt-lines...\ngit> pktline-flush\n\ngit< 0\ngit< reject\n\n\nDo you see a problem with this approach?\n\n\n> A more extensible protocol might look like:\n> \n> git> smudge\n> git> filename=\n> git> size=\n> git> pktline-flush\n> git> ...pkt-lines of data...\n> git> pktline-flush\n> \n> git< ok (or success, or whatever status code you like)\n> git< size=\n> git< pkt-line-flush\n> git< ...pkt-lines of data...\n> git< pktline-flush\n> \n> That leaves room for new \"keys\" to be added before the first pkt-flush,\n> without having to change the parsing at all.\n\nAlright. Will be in v3!\n\n\n>> \"success\\n\" --> everything was alright\n>> \"reject\\n\" --> the filter rejected the operation but this is no error \n>> if \"filter..required = false\"\n>> --> failure that stops/restarts the filter process\n>> \n>> I don't think sending any failure reason makes sense because if a failure\n>> happens then we are likely in a bad state already (that's why I restart the\n>> filter process. I think the filter can report trouble on its own via stdout,\n>> no? I think this is what Git-LFS already does.\n> \n> Git-LFS sends to stderr because there's no other option. I wonder if it\n> would be nicer to make it Git's responsibility to talk to the user,\n> because then it could respect things like \"--quiet\". I guess error\n> messages are generally printed regardless of verbosity, though, so\n> printing them unconditionally is OK.\n\nOK!\n\nThanks,\nLars\n"},{"id":"292549","messageId":"20160729182743.GA9877@sigill.intra.peff.net","threadId":"42904","inReplyTo":"9C2A3438-84BA-49FE-87AA-1084995A1E89@gmail.com","subject":"Re: [PATCH v2 0/5] Git filter protocol","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-29T18:27:43Z","receivedAt":"2016-07-29T18:27:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 29, 2016 at 07:43:49PM +0200, Lars Schneider wrote:\n\n> Here is the reject case (non-streaming):\n> \n> git> smudge\n> git> \n> git> \n> git> ...pkt-lines...\n> git> pktline-flush\n> \n> git< 0\n> git< reject\n> \n> \n> Do you see a problem with this approach?\n\nOnly that it seemed a little weird to me to have to write a meaningless\n\"0\" when \"reject\" covers the situation entirely. I don't think it's\nwrong, though (and even in some ways right, because it decouples the\nmeaning of \"reject\" from the syntax of parsing, but I think it's OK for\nthe protocol parser to understand the difference between success and\nfailure codes).\n\n-Peff\n"},{"id":"292572","messageId":"1a009e19-8830-7dea-2811-d475cf482ea3@gmail.com","threadId":"42904","inReplyTo":"xmqqshusny86.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-29T23:11:24Z","receivedAt":"2016-07-29T23:11:51Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-29 o 19:35, Junio C Hamano pisze:\n> Lars Schneider writes:\n> \n>> I think sending it upfront is nice for buffer allocations of big files\n>> and it doesn't cost us anything to do it.\n> \n> While I do NOT think \"total size upfront\" MUST BE avoided at all costs,\n> I do not think the above statement to justify it makes ANY sense.\n> \n> Big files are by definition something you cannot afford to hold its\n> entirety in core, so you do not want to be told that you'd be fed 40GB\n> and ask xmalloc to allocate that much.\n\nI don't know much how filter driver work internally, but in some cases\nGit reads or writes from file (file descriptor), in other cases it reads\nor writes from str+len pair (it probably predates strbuf) - I think in\nthose cases file needs to fit in memory (in size_t). So in some cases\nGit reads file into memory. Whether it uses xmalloc or mmap, I don't\nknow.\n\n> \n> It allows the reader to be lazy for buffer allocations as long as\n> you know the file fits in-core, at the cost of forcing the writer to\n> somehow come up with the total number of bytes even before sending a\n> single byte (in other words, if the writer cannot produce and hold\n> the data in-core, it may even have to spool the data in a temporary\n> file only to count, and then play it back after showing the total\n> size).\n\nFor some types of filters you can know the size upfront:\n - for filters such as rot13, with 1-to-1 transformation, you know\n that the output size is the same as the input size\n - for block encodings, and for constant-width to constant-width\n encoding conversion, filter can calculate output size from the\n input size (e.g. = 2*)\n - filter may have get size from somewhere, for example LFS filter\n stub is constant size, and files are stored in artifactory with\n their length \n\n> \n> It is good that you allow both mode of operations and the size of\n> the data can either be given upfront (which allows a single fixed\n> allocation upfront without realloc, as long as the data fits in\n> core), or be left \"(atend)\".\n\nI think the protocol should be either: + , or\n + + , that is do not use flush\npacket if size is known upfront -- it would be a second point\nof truth (SPOT principle).\n \n> I just don't want to see it oversold as a \"feature\" that the size\n> has to come before data. That is a limitation, not a feature.\n> \n> Thanks.\n> \n\n\n"},{"id":"292584","messageId":"2435ACEE-19BE-4995-B929-BCEF658F278E@gmail.com","threadId":"42904","inReplyTo":"1a009e19-8830-7dea-2811-d475cf482ea3@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-29T23:44:40Z","receivedAt":"2016-07-29T23:44:49Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 30 Jul 2016, at 01:11, Jakub Narębski wrote:\n> \n> W dniu 2016-07-29 o 19:35, Junio C Hamano pisze:\n>> Lars Schneider writes:\n>> \n>>> I think sending it upfront is nice for buffer allocations of big files\n>>> and it doesn't cost us anything to do it.\n>> \n>> While I do NOT think \"total size upfront\" MUST BE avoided at all costs,\n>> I do not think the above statement to justify it makes ANY sense.\n>> \n>> Big files are by definition something you cannot afford to hold its\n>> entirety in core, so you do not want to be told that you'd be fed 40GB\n>> and ask xmalloc to allocate that much.\n> \n> I don't know much how filter driver work internally, but in some cases\n> Git reads or writes from file (file descriptor), in other cases it reads\n> or writes from str+len pair (it probably predates strbuf) - I think in\n> those cases file needs to fit in memory (in size_t). So in some cases\n> Git reads file into memory. Whether it uses xmalloc or mmap, I don't\n> know.\n> \n>> \n>> It allows the reader to be lazy for buffer allocations as long as\n>> you know the file fits in-core, at the cost of forcing the writer to\n>> somehow come up with the total number of bytes even before sending a\n>> single byte (in other words, if the writer cannot produce and hold\n>> the data in-core, it may even have to spool the data in a temporary\n>> file only to count, and then play it back after showing the total\n>> size).\n> \n> For some types of filters you can know the size upfront:\n> - for filters such as rot13, with 1-to-1 transformation, you know\n> that the output size is the same as the input size\n> - for block encodings, and for constant-width to constant-width\n> encoding conversion, filter can calculate output size from the\n> input size (e.g. = 2*)\n> - filter may have get size from somewhere, for example LFS filter\n> stub is constant size, and files are stored in artifactory with\n> their length \n> \n>> \n>> It is good that you allow both mode of operations and the size of\n>> the data can either be given upfront (which allows a single fixed\n>> allocation upfront without realloc, as long as the data fits in\n>> core), or be left \"(atend)\".\n> \n> I think the protocol should be either: + , or\n> + + , that is do not use flush\n> packet if size is known upfront -- it would be a second point\n> of truth (SPOT principle).\n\nAs I mentioned elsewhere a packet is always send right now.\nI have no strong opinion if this is good or bad. The implementation\nwas a little bit simpler and that's why I did it. I will implement \nwhatever option the majority prefers :-)\n\nCheers,\nLars\n\n> \n>> I just don't want to see it oversold as a \"feature\" that the size\n>> has to come before data. That is a limitation, not a feature.\n>> \n>> Thanks.\n>> \n> \n\n"},{"id":"292592","messageId":"8f40f520-2fa5-449a-57b8-e0fed6b92061@gmail.com","threadId":"42904","inReplyTo":"2435ACEE-19BE-4995-B929-BCEF658F278E@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-30T09:32:00Z","receivedAt":"2016-07-30T09:32:31Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-30 o 01:44, Lars Schneider pisze:\n> On 30 Jul 2016, at 01:11, Jakub Narębski wrote:\n\n>> I think the protocol should be either: + , or\n>> + + , that is do not use flush\n>> packet if size is known upfront -- it would be a second point\n>> of truth (SPOT principle).\n>\n> As I mentioned elsewhere a packet is always send right now.\n> I have no strong opinion if this is good or bad. The implementation\n> was a little bit simpler and that's why I did it. I will implement \n> whatever option the majority prefers :-)\n\nWell, if we treat it as a size hint, then it is all right; as you\nsay it makes for a simpler implementation: read till flush. Git\nshould not error out if there is mismatch between specified and\nactual size of return from the filter; filter can do whatever\nit wants.\n\nI see there is v3 series sent, so I'll move the discussion there.\nOne thing: we probably would want for the size / size-hint\npacket to be extensible, either\n\n size= [(SPC =)...] \"\\n\"\n\nor\n\n [(SPC )...] \"\\n\"\n\nthat is, space separated list starting with size / size hint.\n\nUpfront error could be signalled by putting for example \"error\"\nin place of size, e.g.\n\n error \"\\n\"\n\n-- \nJakub Narębski\n"},{"id":"293221","messageId":"20160805185559.GB463@starla","threadId":"42904","inReplyTo":"64C7D52F-9030-460C-8F61-4076F5C1DDF6@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-08-05T18:55:59Z","receivedAt":"2016-08-05T18:56:07Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Lars Schneider wrote:\n> > On 27 Jul 2016, at 11:41, Eric Wong wrote:\n> > larsxschneider@gmail.com wrote:\n> >> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n> >> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n> >> +\t\t\t\t\t\tconst char *filter_type)\n> >> +{\n> > \n> > \n> > \n> >> +\tif (fd >= 0 && !src) {\n> >> +\t\tret &= fstat(fd, &file_stat) != -1;\n> >> +\t\tlen = file_stat.st_size;\n> > \n> > Same truncation bug I noticed earlier; what I originally meant\n> > is the `len' arg probably ought to be off_t, here, not size_t.\n> > 32-bit x86 Linux systems have 32-bit size_t (unsigned), but\n> > large file support means off_t is 64-bits (signed).\n> \n> OK. Would it be OK to keep size_t for this patch series?\n\nI think there should at least be a truncation warning (or die)\nfor larger-than-4GB files on 32-bit. I don't know how common\nthey are for git-lfs users.\n\nPerhaps using xsize_t in git-compat-util.h works for now:\n\n\tlen = xsize_t(file_stat.st_size);\n\n(sorry, I haven't had much time to look at your other updates)\n"},{"id":"293273","messageId":"20160805233838.GA29867@starla","threadId":"42904","inReplyTo":"4499EF4B-974C-4D59-B95F-541EA4CF4078@gmail.com","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-08-05T23:38:38Z","receivedAt":"2016-08-06T20:01:06Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"> >>> larsxschneider@gmail.com wrote:\n> >>>> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n\nLars Schneider wrote:\n> > On 05 Aug 2016, at 20:55, Eric Wong wrote:\n> > Perhaps using xsize_t in git-compat-util.h works for now:\n> > \n> > \tlen = xsize_t(file_stat.st_size);\n> \n> Thanks for the hint! Should I add the same check to sha1_file's use\n> of fstat in line 1002 or is it not needed there?\n> \n> https://github.com/git/git/blob/c6b0597e9ac7277e148e2fd4d7615ac6e0bfb661/sha1_file.c#L1002\n\nNot needed, if you look at the definition of \"struct packed_git\"\nin cache.h, you'll see pack_size is already off_t, not size_t\nlike `len` is.\n"},{"id":"293312","messageId":"4499EF4B-974C-4D59-B95F-541EA4CF4078@gmail.com","threadId":"42904","inReplyTo":"20160805185559.GB463@starla","subject":"Re: [PATCH v2 5/5] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-08-05T23:26:34Z","receivedAt":"2016-08-07T00:13:38Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 05 Aug 2016, at 20:55, Eric Wong wrote:\n> \n> Lars Schneider wrote:\n>>> On 27 Jul 2016, at 11:41, Eric Wong wrote:\n>>> larsxschneider@gmail.com wrote:\n>>>> +static int apply_protocol_filter(const char *path, const char *src, size_t len,\n>>>> +\t\t\t\t\t\tint fd, struct strbuf *dst, const char *cmd,\n>>>> +\t\t\t\t\t\tconst char *filter_type)\n>>>> +{\n>>> \n>>> \n>>> \n>>>> +\tif (fd >= 0 && !src) {\n>>>> +\t\tret &= fstat(fd, &file_stat) != -1;\n>>>> +\t\tlen = file_stat.st_size;\n>>> \n>>> Same truncation bug I noticed earlier; what I originally meant\n>>> is the `len' arg probably ought to be off_t, here, not size_t.\n>>> 32-bit x86 Linux systems have 32-bit size_t (unsigned), but\n>>> large file support means off_t is 64-bits (signed).\n>> \n>> OK. Would it be OK to keep size_t for this patch series?\n> \n> I think there should at least be a truncation warning (or die)\n> for larger-than-4GB files on 32-bit. I don't know how common\n> they are for git-lfs users.\n> \n> Perhaps using xsize_t in git-compat-util.h works for now:\n> \n> \tlen = xsize_t(file_stat.st_size);\n\nThanks for the hint! Should I add the same check to sha1_file's use\nof fstat in line 1002 or is it not needed there?\n\nhttps://github.com/git/git/blob/c6b0597e9ac7277e148e2fd4d7615ac6e0bfb661/sha1_file.c#L1002\n\nThanks,\nLars"}]}