{"thread":{"id":"46298","subject":"[PATCH v9 0/7] convert: add \"status=delayed\" to filter process protocol","startedAt":"2017-06-30T20:41:38Z","lastAt":"2017-06-30T21:31:07Z","messageCount":9,"participants":["Lars Schneider","Junio C Hamano"],"isPatch":true,"patchVersion":9,"patchTotal":7},"messages":[{"id":"323608","messageId":"20170630204128.48708-1-larsxschneider@gmail.com","threadId":"46298","inReplyTo":null,"subject":"[PATCH v9 0/7] convert: add \"status=delayed\" to filter process protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-30T20:41:21Z","receivedAt":"2017-06-30T20:41:38Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Hi,\n\nhere is the 9th iteration of my \"status delayed\" topic. I think that\nmight be the final one :-)\n\nPatch 1 to 3 are minor t0021 test adjustments. Patch 4 to 6 are convert\nrefactorings to prepare the new feature. Patch 7 is the new feature.\n\n### Changes since v7:\n* extracted capabilities negotiation refactoring into dedicated patch 6\n  --> no code changes and therefore no inter-diff\n\nThanks,\nLars\n\n\nRFC: http://public-inbox.org/git/D10F7C47-14E8-465B-8B7A-A09A1B28A39F@gmail.com/\nv1: http://public-inbox.org/git/20170108191736.47359-1-larsxschneider@gmail.com/\nv2: http://public-inbox.org/git/20170226184816.30010-1-larsxschneider@gmail.com/\nv3: http://public-inbox.org/git/20170409191107.20547-1-larsxschneider@gmail.com/\nv4: http://public-inbox.org/git/20170522135001.54506-1-larsxschneider@gmail.com/\nv5: http://public-inbox.org/git/20170601082203.50397-1-larsxschneider@gmail.com/\nv6: http://public-inbox.org/git/20170625182125.6741-1-larsxschneider@gmail.com/\nv7: http://public-inbox.org/git/20170627121027.99209-1-larsxschneider@gmail.com/\nv8: http://public-inbox.org/git/20170628212952.60781-1-larsxschneider@gmail.com/\n\nBase Ref: master\nWeb-Diff: https://github.com/larsxschneider/git/commit/c391b48aa2\nCheckout: git fetch https://github.com/larsxschneider/git filter-process/delay-v9 && git checkout c391b48aa2\n\n\n### Patches\n\nLars Schneider (7):\n  t0021: keep filter log files on comparison\n  t0021: make debug log file name configurable\n  t0021: write \"OUT <size>\" only on success\n  convert: put the flags field before the flag itself for consistent\n    style\n  convert: move multiple file filter error handling to separate function\n  convert: refactor capabilities negotiation\n  convert: add \"status=delayed\" to filter process protocol\n\n Documentation/gitattributes.txt |  69 ++++++++++++-\n builtin/checkout.c              |   3 +\n cache.h                         |   3 +\n convert.c                       | 202 +++++++++++++++++++++++++++----------\n convert.h                       |  26 +++++\n entry.c                         | 132 ++++++++++++++++++++++++-\n t/t0021-conversion.sh           | 178 +++++++++++++++++++++++++++------\n t/t0021/rot13-filter.pl         | 214 +++++++++++++++++++++++++++-------------\n unpack-trees.c                  |   2 +\n 9 files changed, 668 insertions(+), 161 deletions(-)\n\n\nbase-commit: 0339965c70d68fd3831c9a5306443c869de3f6a8\n--\n2.13.2\n\n"},{"id":"323609","messageId":"20170630204128.48708-2-larsxschneider@gmail.com","threadId":"46298","inReplyTo":"20170630204128.48708-1-larsxschneider@gmail.com","subject":"[PATCH v9 1/7] t0021: keep filter log files on comparison","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-30T20:41:22Z","receivedAt":"2017-06-30T20:41:40Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"The filter log files are modified on comparison. That might be\nunexpected by the caller. It would be even undesirable if the caller\nwants to reuse the original log files.\n\nAddress these issues by using temp files for modifications. This is\nuseful for the subsequent patch 'convert: add \"status=delayed\" to\nfilter process protocol'.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n t/t0021-conversion.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 161f560446..ff2424225b 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -42,10 +42,10 @@ test_cmp_count () {\n \tfor FILE in \"$expect\" \"$actual\"\n \tdo\n \t\tsort \"$FILE\" | uniq -c |\n-\t\tsed -e \"s/^ *[0-9][0-9]*[ \t]*IN: /x IN: /\" >\"$FILE.tmp\" &&\n-\t\tmv \"$FILE.tmp\" \"$FILE\" || return\n+\t\tsed -e \"s/^ *[0-9][0-9]*[ \t]*IN: /x IN: /\" >\"$FILE.tmp\"\n \tdone &&\n-\ttest_cmp \"$expect\" \"$actual\"\n+\ttest_cmp \"$expect.tmp\" \"$actual.tmp\" &&\n+\trm \"$expect.tmp\" \"$actual.tmp\"\n }\n \n # Compare two files but exclude all `clean` invocations because Git can\n@@ -56,10 +56,10 @@ test_cmp_exclude_clean () {\n \tactual=$2\n \tfor FILE in \"$expect\" \"$actual\"\n \tdo\n-\t\tgrep -v \"IN: clean\" \"$FILE\" >\"$FILE.tmp\" &&\n-\t\tmv \"$FILE.tmp\" \"$FILE\"\n+\t\tgrep -v \"IN: clean\" \"$FILE\" >\"$FILE.tmp\"\n \tdone &&\n-\ttest_cmp \"$expect\" \"$actual\"\n+\ttest_cmp \"$expect.tmp\" \"$actual.tmp\" &&\n+\trm \"$expect.tmp\" \"$actual.tmp\"\n }\n \n # Check that the contents of two files are equal and that their rot13 version\n-- \n2.13.2\n\n"},{"id":"323610","messageId":"20170630204128.48708-4-larsxschneider@gmail.com","threadId":"46298","inReplyTo":"20170630204128.48708-1-larsxschneider@gmail.com","subject":"[PATCH v9 3/7] t0021: write \"OUT <size>\" only on success","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-30T20:41:24Z","receivedAt":"2017-06-30T20:41:42Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\"rot13-filter.pl\" always writes \"OUT <size>\" to the debug log at the end\nof a response.\n\nThis works perfectly for the existing responses \"abort\", \"error\", and\n\"success\". A new response \"delayed\", that will be introduced in a\nsubsequent patch, accepts the input without giving the filtered result\nright away. At this point we cannot know the size of the response.\nTherefore, we do not write \"OUT <size>\" for \"delayed\" responses.\n\nTo simplify the code we do not write \"OUT <size>\" for \"abort\" and\n\"error\" responses either as their size is always zero.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n t/t0021-conversion.sh   | 6 +++---\n t/t0021/rot13-filter.pl | 6 +++---\n 2 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 0139b460e7..0c04d346a1 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -588,7 +588,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\tcat >expected.log <<-EOF &&\n \t\t\tSTART\n \t\t\tinit handshake complete\n-\t\t\tIN: smudge smudge-write-fail.r $SF [OK] -- OUT: $SF [WRITE FAIL]\n+\t\t\tIN: smudge smudge-write-fail.r $SF [OK] -- [WRITE FAIL]\n \t\t\tSTART\n \t\t\tinit handshake complete\n \t\t\tIN: smudge test.r $S [OK] -- OUT: $S . [OK]\n@@ -634,7 +634,7 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n \t\tcat >expected.log <<-EOF &&\n \t\t\tSTART\n \t\t\tinit handshake complete\n-\t\t\tIN: smudge error.r $SE [OK] -- OUT: 0 [ERROR]\n+\t\t\tIN: smudge error.r $SE [OK] -- [ERROR]\n \t\t\tIN: smudge test.r $S [OK] -- OUT: $S . [OK]\n \t\t\tIN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]\n \t\t\tSTOP\n@@ -673,7 +673,7 @@ test_expect_success PERL 'process filter abort stops processing of all further f\n \t\tcat >expected.log <<-EOF &&\n \t\t\tSTART\n \t\t\tinit handshake complete\n-\t\t\tIN: smudge abort.r $SA [OK] -- OUT: 0 [ABORT]\n+\t\t\tIN: smudge abort.r $SA [OK] -- [ABORT]\n \t\t\tSTOP\n \t\tEOF\n \t\ttest_cmp_exclude_clean expected.log debug.log &&\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 0b943bb377..5e43faeec1 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -153,9 +153,6 @@ while (1) {\n \t\tdie \"bad command '$command'\";\n \t}\n \n-\tprint $debug \"OUT: \" . length($output) . \" \";\n-\t$debug->flush();\n-\n \tif ( $pathname eq \"error.r\" ) {\n \t\tprint $debug \"[ERROR]\\n\";\n \t\t$debug->flush();\n@@ -178,6 +175,9 @@ while (1) {\n \t\t\tdie \"${command} write error\";\n \t\t}\n \n+\t\tprint $debug \"OUT: \" . length($output) . \" \";\n+\t\t$debug->flush();\n+\n \t\twhile ( length($output) > 0 ) {\n \t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n \t\t\tpacket_bin_write($packet);\n-- \n2.13.2\n\n"},{"id":"323611","messageId":"20170630204128.48708-6-larsxschneider@gmail.com","threadId":"46298","inReplyTo":"20170630204128.48708-1-larsxschneider@gmail.com","subject":"[PATCH v9 5/7] convert: move multiple file filter error handling to separate function","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-30T20:41:26Z","receivedAt":"2017-06-30T20:41:44Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Refactoring the filter error handling is useful for the subsequent patch\n'convert: add \"status=delayed\" to filter process protocol'.\n\nIn addition, replace the parentheses around the empty \"if\" block with a\nsingle semicolon to adhere to the Git style guide.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n convert.c | 47 ++++++++++++++++++++++++++---------------------\n 1 file changed, 26 insertions(+), 21 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 9907e3b9ba..e55c034d86 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -565,6 +565,29 @@ static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n \treturn err;\n }\n \n+static void handle_filter_error(const struct strbuf *filter_status,\n+\t\t\t\tstruct cmd2process *entry,\n+\t\t\t\tconst unsigned int wanted_capability) {\n+\tif (!strcmp(filter_status->buf, \"error\"))\n+\t\t; /* The filter signaled a problem with the file. */\n+\telse if (!strcmp(filter_status->buf, \"abort\") && wanted_capability) {\n+\t\t/*\n+\t\t * The filter signaled a permanent problem. Don't try to filter\n+\t\t * files with the same command for the lifetime of the current\n+\t\t * Git process.\n+\t\t */\n+\t\t entry->supported_capabilities &= ~wanted_capability;\n+\t} else {\n+\t\t/*\n+\t\t * Something went wrong with the protocol filter.\n+\t\t * Force shutdown and restart if another blob requires filtering.\n+\t\t */\n+\t\terror(\"external filter '%s' failed\", entry->subprocess.cmd);\n+\t\tsubprocess_stop(&subprocess_map, &entry->subprocess);\n+\t\tfree(entry);\n+\t}\n+}\n+\n static int apply_multi_file_filter(const char *path, const char *src, size_t len,\n \t\t\t\t   int fd, struct strbuf *dst, const char *cmd,\n \t\t\t\t   const unsigned int wanted_capability)\n@@ -656,28 +679,10 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n done:\n \tsigchain_pop(SIGPIPE);\n \n-\tif (err) {\n-\t\tif (!strcmp(filter_status.buf, \"error\")) {\n-\t\t\t/* The filter signaled a problem with the file. */\n-\t\t} else if (!strcmp(filter_status.buf, \"abort\")) {\n-\t\t\t/*\n-\t\t\t * The filter signaled a permanent problem. Don't try to filter\n-\t\t\t * files with the same command for the lifetime of the current\n-\t\t\t * Git process.\n-\t\t\t */\n-\t\t\t entry->supported_capabilities &= ~wanted_capability;\n-\t\t} else {\n-\t\t\t/*\n-\t\t\t * Something went wrong with the protocol filter.\n-\t\t\t * Force shutdown and restart if another blob requires filtering.\n-\t\t\t */\n-\t\t\terror(\"external filter '%s' failed\", cmd);\n-\t\t\tsubprocess_stop(&subprocess_map, &entry->subprocess);\n-\t\t\tfree(entry);\n-\t\t}\n-\t} else {\n+\tif (err)\n+\t\thandle_filter_error(&filter_status, entry, wanted_capability);\n+\telse\n \t\tstrbuf_swap(dst, &nbuf);\n-\t}\n \tstrbuf_release(&nbuf);\n \treturn !err;\n }\n-- \n2.13.2\n\n"},{"id":"323612","messageId":"20170630204128.48708-5-larsxschneider@gmail.com","threadId":"46298","inReplyTo":"20170630204128.48708-1-larsxschneider@gmail.com","subject":"[PATCH v9 4/7] convert: put the flags field before the flag itself for consistent style","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-30T20:41:25Z","receivedAt":"2017-06-30T20:41:55Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Suggested-by: Jeff King <peff@peff.net>\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n convert.c | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex f1e168bc30..9907e3b9ba 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -597,12 +597,12 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \t}\n \tprocess = &entry->subprocess.process;\n \n-\tif (!(wanted_capability & entry->supported_capabilities))\n+\tif (!(entry->supported_capabilities & wanted_capability))\n \t\treturn 0;\n \n-\tif (CAP_CLEAN & wanted_capability)\n+\tif (wanted_capability & CAP_CLEAN)\n \t\tfilter_type = \"clean\";\n-\telse if (CAP_SMUDGE & wanted_capability)\n+\telse if (wanted_capability & CAP_SMUDGE)\n \t\tfilter_type = \"smudge\";\n \telse\n \t\tdie(\"unexpected filter type\");\n@@ -703,9 +703,9 @@ static int apply_filter(const char *path, const char *src, size_t len,\n \tif (!dst)\n \t\treturn 1;\n \n-\tif ((CAP_CLEAN & wanted_capability) && !drv->process && drv->clean)\n+\tif ((wanted_capability & CAP_CLEAN) && !drv->process && drv->clean)\n \t\tcmd = drv->clean;\n-\telse if ((CAP_SMUDGE & wanted_capability) && !drv->process && drv->smudge)\n+\telse if ((wanted_capability & CAP_SMUDGE) && !drv->process && drv->smudge)\n \t\tcmd = drv->smudge;\n \n \tif (cmd && *cmd)\n-- \n2.13.2\n\n"},{"id":"323613","messageId":"20170630204128.48708-8-larsxschneider@gmail.com","threadId":"46298","inReplyTo":"20170630204128.48708-1-larsxschneider@gmail.com","subject":"[PATCH v9 7/7] convert: add \"status=delayed\" to filter process protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-30T20:41:28Z","receivedAt":"2017-06-30T20:41:57Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Some `clean` / `smudge` filters may require a significant amount of\ntime to process a single blob (e.g. the Git LFS smudge filter might\nperform network requests). During this process the Git checkout\noperation is blocked and Git needs to wait until the filter is done to\ncontinue with the checkout.\n\nTeach the filter process protocol, introduced in edcc8581 (\"convert: add\nfilter.<driver>.process option\", 2016-10-16), to accept the status\n\"delayed\" as response to a filter request. Upon this response Git\ncontinues with the checkout operation. After the checkout operation Git\ncalls \"finish_delayed_checkout\" which queries the filter for remaining\nblobs. If the filter is still working on the completion, then the filter\nis expected to block. If the filter has completed all remaining blobs\nthen an empty response is expected.\n\nGit has a multiple code paths that checkout a blob. Support delayed\ncheckouts only in `clone` (in unpack-trees.c) and `checkout` operations\nfor now. The optimization is most effective in these code paths as all\nfiles of the tree are processed.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n Documentation/gitattributes.txt |  69 +++++++++++++-\n builtin/checkout.c              |   3 +\n cache.h                         |   3 +\n convert.c                       | 110 ++++++++++++++++++----\n convert.h                       |  26 +++++\n entry.c                         | 132 +++++++++++++++++++++++++-\n t/t0021-conversion.sh           | 116 +++++++++++++++++++++++\n t/t0021/rot13-filter.pl         | 204 +++++++++++++++++++++++++++-------------\n unpack-trees.c                  |   2 +\n 9 files changed, 575 insertions(+), 90 deletions(-)\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 4736483865..4049a0b9a8 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -425,8 +425,8 @@ packet:          git< capability=clean\n packet:          git< capability=smudge\n packet:          git< 0000\n ------------------------\n-Supported filter capabilities in version 2 are \"clean\" and\n-\"smudge\".\n+Supported filter capabilities in version 2 are \"clean\", \"smudge\",\n+and \"delay\".\n \n Afterwards Git sends a list of \"key=value\" pairs terminated with\n a flush packet. The list will contain at least the filter command\n@@ -512,12 +512,73 @@ the protocol then Git will stop the filter process and restart it\n with the next file that needs to be processed. Depending on the\n `filter.<driver>.required` flag Git will interpret that as error.\n \n-After the filter has processed a blob it is expected to wait for\n-the next \"key=value\" list containing a command. Git will close\n+After the filter has processed a command it is expected to wait for\n+a \"key=value\" list containing the next command. Git will close\n the command pipe on exit. The filter is expected to detect EOF\n and exit gracefully on its own. Git will wait until the filter\n process has stopped.\n \n+Delay\n+^^^^^\n+\n+If the filter supports the \"delay\" capability, then Git can send the\n+flag \"can-delay\" after the filter command and pathname. This flag\n+denotes that the filter can delay filtering the current blob (e.g. to\n+compensate network latencies) by responding with no content but with\n+the status \"delayed\" and a flush packet.\n+------------------------\n+packet:          git> command=smudge\n+packet:          git> pathname=path/testfile.dat\n+packet:          git> can-delay=1\n+packet:          git> 0000\n+packet:          git> CONTENT\n+packet:          git> 0000\n+packet:          git< status=delayed\n+packet:          git< 0000\n+------------------------\n+\n+If the filter supports the \"delay\" capability then it must support the\n+\"list_available_blobs\" command. If Git sends this command, then the\n+filter is expected to return a list of pathnames representing blobs\n+that have been delayed earlier and are now available.\n+The list must be terminated with a flush packet followed\n+by a \"success\" status that is also terminated with a flush packet. If\n+no blobs for the delayed paths are available, yet, then the filter is\n+expected to block the response until at least one blob becomes\n+available. The filter can tell Git that it has no more delayed blobs\n+by sending an empty list. As soon as the filter responds with an empty\n+list, Git stops asking. All blobs that Git has not received at this\n+point are considered missing and will result in an error.\n+\n+------------------------\n+packet:          git> command=list_available_blobs\n+packet:          git> 0000\n+packet:          git< pathname=path/testfile.dat\n+packet:          git< pathname=path/otherfile.dat\n+packet:          git< 0000\n+packet:          git< status=success\n+packet:          git< 0000\n+------------------------\n+\n+After Git received the pathnames, it will request the corresponding\n+blobs again. These requests contain a pathname and an empty content\n+section. The filter is expected to respond with the smudged content\n+in the usual way as explained above.\n+------------------------\n+packet:          git> command=smudge\n+packet:          git> pathname=path/testfile.dat\n+packet:          git> 0000\n+packet:          git> 0000  # empty content!\n+packet:          git< status=success\n+packet:          git< 0000\n+packet:          git< SMUDGED_CONTENT\n+packet:          git< 0000\n+packet:          git< 0000  # empty list, keep \"status=success\" unchanged!\n+------------------------\n+\n+Example\n+^^^^^^^\n+\n A long running filter demo implementation can be found in\n `contrib/long-running-filter/example.pl` located in the Git\n core repository. If you develop your own long running filter\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex a6b2af39d3..c1a256df8d 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -376,6 +376,8 @@ static int checkout_paths(const struct checkout_opts *opts,\n \tstate.force = 1;\n \tstate.refresh_cache = 1;\n \tstate.istate = &the_index;\n+\n+\tenable_delayed_checkout(&state);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n \t\tif (ce->ce_flags & CE_MATCHED) {\n@@ -390,6 +392,7 @@ static int checkout_paths(const struct checkout_opts *opts,\n \t\t\tpos = skip_same_name(ce, pos) - 1;\n \t\t}\n \t}\n+\terrs |= finish_delayed_checkout(&state);\n \n \tif (write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n \t\tdie(_(\"unable to write new index file\"));\ndiff --git a/cache.h b/cache.h\nindex ae4c45d379..69b03b5dc7 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1544,6 +1544,7 @@ struct checkout {\n \tstruct index_state *istate;\n \tconst char *base_dir;\n \tint base_dir_len;\n+\tstruct delayed_checkout *delayed_checkout;\n \tunsigned force:1,\n \t\t quiet:1,\n \t\t not_new:1,\n@@ -1553,6 +1554,8 @@ struct checkout {\n \n #define TEMPORARY_FILENAME_LENGTH 25\n extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);\n+extern void enable_delayed_checkout(struct checkout *state);\n+extern int finish_delayed_checkout(struct checkout *state);\n \n struct cache_def {\n \tstruct strbuf path;\ndiff --git a/convert.c b/convert.c\nindex d13e505dfb..12a0b3eafb 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -496,6 +496,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le\n \n #define CAP_CLEAN    (1u<<0)\n #define CAP_SMUDGE   (1u<<1)\n+#define CAP_DELAY    (1u<<2)\n \n struct cmd2process {\n \tstruct subprocess_entry subprocess; /* must be the first member! */\n@@ -521,6 +522,7 @@ static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n \t} known_caps[] = {\n \t\t{ \"clean\",  CAP_CLEAN  },\n \t\t{ \"smudge\", CAP_SMUDGE },\n+\t\t{ \"delay\",  CAP_DELAY  },\n \t};\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n@@ -605,9 +607,11 @@ static void handle_filter_error(const struct strbuf *filter_status,\n \n static int apply_multi_file_filter(const char *path, const char *src, size_t len,\n \t\t\t\t   int fd, struct strbuf *dst, const char *cmd,\n-\t\t\t\t   const unsigned int wanted_capability)\n+\t\t\t\t   const unsigned int wanted_capability,\n+\t\t\t\t   struct delayed_checkout *dco)\n {\n \tint err;\n+\tint can_delay = 0;\n \tstruct cmd2process *entry;\n \tstruct child_process *process;\n \tstruct strbuf nbuf = STRBUF_INIT;\n@@ -662,6 +666,14 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tif (err)\n \t\tgoto done;\n \n+\tif ((entry->supported_capabilities & CAP_DELAY) &&\n+\t    dco && dco->state == CE_CAN_DELAY) {\n+\t\tcan_delay = 1;\n+\t\terr = packet_write_fmt_gently(process->in, \"can-delay=1\\n\");\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t}\n+\n \terr = packet_flush_gently(process->in);\n \tif (err)\n \t\tgoto done;\n@@ -677,14 +689,73 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tif (err)\n \t\tgoto done;\n \n-\terr = strcmp(filter_status.buf, \"success\");\n+\tif (can_delay && !strcmp(filter_status.buf, \"delayed\")) {\n+\t\tstring_list_insert(&dco->filters, cmd);\n+\t\tstring_list_insert(&dco->paths, path);\n+\t} else {\n+\t\t/* The filter got the blob and wants to send us a response. */\n+\t\terr = strcmp(filter_status.buf, \"success\");\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\terr = read_packetized_to_strbuf(process->out, &nbuf) < 0;\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\terr = subprocess_read_status(process->out, &filter_status);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\terr = strcmp(filter_status.buf, \"success\");\n+\t}\n+\n+done:\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (err)\n+\t\thandle_filter_error(&filter_status, entry, wanted_capability);\n+\telse\n+\t\tstrbuf_swap(dst, &nbuf);\n+\tstrbuf_release(&nbuf);\n+\treturn !err;\n+}\n+\n+\n+int async_query_available_blobs(const char *cmd, struct string_list *available_paths)\n+{\n+\tint err;\n+\tchar *line;\n+\tstruct cmd2process *entry;\n+\tstruct child_process *process;\n+\tstruct strbuf filter_status = STRBUF_INIT;\n+\n+\tassert(subprocess_map_initialized);\n+\tentry = (struct cmd2process *)subprocess_find_entry(&subprocess_map, cmd);\n+\tif (!entry) {\n+\t\terror(\"external filter '%s' is not available anymore although \"\n+\t\t      \"not all paths have been filtered\", cmd);\n+\t\treturn 0;\n+\t}\n+\tprocess = &entry->subprocess.process;\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = packet_write_fmt_gently(\n+\t\tprocess->in, \"command=list_available_blobs\\n\");\n \tif (err)\n \t\tgoto done;\n \n-\terr = read_packetized_to_strbuf(process->out, &nbuf) < 0;\n+\terr = packet_flush_gently(process->in);\n \tif (err)\n \t\tgoto done;\n \n+\twhile ((line = packet_read_line(process->out, NULL))) {\n+\t\tconst char *path;\n+\t\tif (skip_prefix(line, \"pathname=\", &path))\n+\t\t\tstring_list_insert(available_paths, xstrdup(path));\n+\t\telse\n+\t\t\t; /* ignore unknown keys */\n+\t}\n+\n \terr = subprocess_read_status(process->out, &filter_status);\n \tif (err)\n \t\tgoto done;\n@@ -695,10 +766,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tsigchain_pop(SIGPIPE);\n \n \tif (err)\n-\t\thandle_filter_error(&filter_status, entry, wanted_capability);\n-\telse\n-\t\tstrbuf_swap(dst, &nbuf);\n-\tstrbuf_release(&nbuf);\n+\t\thandle_filter_error(&filter_status, entry, 0);\n \treturn !err;\n }\n \n@@ -713,7 +781,8 @@ static struct convert_driver {\n \n static int apply_filter(const char *path, const char *src, size_t len,\n \t\t\tint fd, struct strbuf *dst, struct convert_driver *drv,\n-\t\t\tconst unsigned int wanted_capability)\n+\t\t\tconst unsigned int wanted_capability,\n+\t\t\tstruct delayed_checkout *dco)\n {\n \tconst char *cmd = NULL;\n \n@@ -731,7 +800,8 @@ static int apply_filter(const char *path, const char *src, size_t len,\n \tif (cmd && *cmd)\n \t\treturn apply_single_file_filter(path, src, len, fd, dst, cmd);\n \telse if (drv->process && *drv->process)\n-\t\treturn apply_multi_file_filter(path, src, len, fd, dst, drv->process, wanted_capability);\n+\t\treturn apply_multi_file_filter(path, src, len, fd, dst,\n+\t\t\tdrv->process, wanted_capability, dco);\n \n \treturn 0;\n }\n@@ -1072,7 +1142,7 @@ int would_convert_to_git_filter_fd(const char *path)\n \tif (!ca.drv->required)\n \t\treturn 0;\n \n-\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN);\n+\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL);\n }\n \n const char *get_convert_attr_ascii(const char *path)\n@@ -1109,7 +1179,7 @@ int convert_to_git(const char *path, const char *src, size_t len,\n \n \tconvert_attrs(&ca, path);\n \n-\tret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN);\n+\tret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN, NULL);\n \tif (!ret && ca.drv && ca.drv->required)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n@@ -1134,7 +1204,7 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n \tassert(ca.drv);\n \tassert(ca.drv->clean || ca.drv->process);\n \n-\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN))\n+\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n@@ -1143,7 +1213,7 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n \n static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t\t\t\t    size_t len, struct strbuf *dst,\n-\t\t\t\t\t    int normalizing)\n+\t\t\t\t\t    int normalizing, struct delayed_checkout *dco)\n {\n \tint ret = 0, ret_filter = 0;\n \tstruct conv_attrs ca;\n@@ -1168,21 +1238,29 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n-\tret_filter = apply_filter(path, src, len, -1, dst, ca.drv, CAP_SMUDGE);\n+\tret_filter = apply_filter(\n+\t\tpath, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);\n \tif (!ret_filter && ca.drv && ca.drv->required)\n \t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n \n \treturn ret | ret_filter;\n }\n \n+int async_convert_to_working_tree(const char *path, const char *src,\n+\t\t\t\t  size_t len, struct strbuf *dst,\n+\t\t\t\t  void *dco)\n+{\n+\treturn convert_to_working_tree_internal(path, src, len, dst, 0, dco);\n+}\n+\n int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)\n {\n-\treturn convert_to_working_tree_internal(path, src, len, dst, 0);\n+\treturn convert_to_working_tree_internal(path, src, len, dst, 0, NULL);\n }\n \n int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst)\n {\n-\tint ret = convert_to_working_tree_internal(path, src, len, dst, 1);\n+\tint ret = convert_to_working_tree_internal(path, src, len, dst, 1, NULL);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\ndiff --git a/convert.h b/convert.h\nindex 82871a11d5..643a5be6cc 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -4,6 +4,8 @@\n #ifndef CONVERT_H\n #define CONVERT_H\n \n+#include \"string-list.h\"\n+\n enum safe_crlf {\n \tSAFE_CRLF_FALSE = 0,\n \tSAFE_CRLF_FAIL = 1,\n@@ -32,6 +34,26 @@ enum eol {\n #endif\n };\n \n+enum ce_delay_state {\n+\tCE_NO_DELAY = 0,\n+\tCE_CAN_DELAY = 1,\n+\tCE_RETRY = 2\n+};\n+\n+struct delayed_checkout {\n+\t/*\n+\t * State of the currently processed cache entry. If the state is\n+\t * CE_CAN_DELAY, then the filter can delay the current cache entry.\n+\t * If the state is CE_RETRY, then this signals the filter that the\n+\t * cache entry was requested before.\n+\t */\n+\tenum ce_delay_state state;\n+\t/* List of filter drivers that signaled delayed blobs. */\n+\tstruct string_list filters;\n+\t/* List of delayed blobs identified by their path. */\n+\tstruct string_list paths;\n+};\n+\n extern enum eol core_eol;\n extern const char *get_cached_convert_stats_ascii(const char *path);\n extern const char *get_wt_convert_stats_ascii(const char *path);\n@@ -42,6 +64,10 @@ extern int convert_to_git(const char *path, const char *src, size_t len,\n \t\t\t  struct strbuf *dst, enum safe_crlf checksafe);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t   size_t len, struct strbuf *dst);\n+extern int async_convert_to_working_tree(const char *path, const char *src,\n+\t\t\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t\t\t void *dco);\n+extern int async_query_available_blobs(const char *cmd, struct string_list *available_paths);\n extern int renormalize_buffer(const char *path, const char *src, size_t len,\n \t\t\t      struct strbuf *dst);\n static inline int would_convert_to_git(const char *path)\ndiff --git a/entry.c b/entry.c\nindex d6b263f78e..65458f07a4 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -137,6 +137,105 @@ static int streaming_write_entry(const struct cache_entry *ce, char *path,\n \treturn result;\n }\n \n+void enable_delayed_checkout(struct checkout *state)\n+{\n+\tif (!state->delayed_checkout) {\n+\t\tstate->delayed_checkout = xmalloc(sizeof(*state->delayed_checkout));\n+\t\tstate->delayed_checkout->state = CE_CAN_DELAY;\n+\t\tstring_list_init(&state->delayed_checkout->filters, 0);\n+\t\tstring_list_init(&state->delayed_checkout->paths, 0);\n+\t}\n+}\n+\n+static int remove_available_paths(struct string_list_item *item, void *cb_data)\n+{\n+\tstruct string_list *available_paths = cb_data;\n+\tstruct string_list_item *available;\n+\n+\tavailable = string_list_lookup(available_paths, item->string);\n+\tif (available)\n+\t\tavailable->util = (void *)item->string;\n+\treturn !available;\n+}\n+\n+int finish_delayed_checkout(struct checkout *state)\n+{\n+\tint errs = 0;\n+\tstruct string_list_item *filter, *path;\n+\tstruct delayed_checkout *dco = state->delayed_checkout;\n+\n+\tif (!state->delayed_checkout)\n+\t\treturn errs;\n+\n+\tdco->state = CE_RETRY;\n+\twhile (dco->filters.nr > 0) {\n+\t\tfor_each_string_list_item(filter, &dco->filters) {\n+\t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n+\n+\t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n+\t\t\t\t/* Filter reported an error */\n+\t\t\t\terrs = 1;\n+\t\t\t\tfilter->string = \"\";\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (available_paths.nr <= 0) {\n+\t\t\t\t/*\n+\t\t\t\t * Filter responded with no entries. That means\n+\t\t\t\t * the filter is done and we can remove the\n+\t\t\t\t * filter from the list (see\n+\t\t\t\t * \"string_list_remove_empty_items\" call below).\n+\t\t\t\t */\n+\t\t\t\tfilter->string = \"\";\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * In dco->paths we store a list of all delayed paths.\n+\t\t\t * The filter just send us a list of available paths.\n+\t\t\t * Remove them from the list.\n+\t\t\t */\n+\t\t\tfilter_string_list(&dco->paths, 0,\n+\t\t\t\t&remove_available_paths, &available_paths);\n+\n+\t\t\tfor_each_string_list_item(path, &available_paths) {\n+\t\t\t\tstruct cache_entry* ce;\n+\n+\t\t\t\tif (!path->util) {\n+\t\t\t\t\terror(\"external filter '%s' signaled that '%s' \"\n+\t\t\t\t\t      \"is now available although it has not been \"\n+\t\t\t\t\t      \"delayed earlier\",\n+\t\t\t\t\t      filter->string, path->string);\n+\t\t\t\t\terrs |= 1;\n+\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Do not ask the filter for available blobs,\n+\t\t\t\t\t * again, as the filter is likely buggy.\n+\t\t\t\t\t */\n+\t\t\t\t\tfilter->string = \"\";\n+\t\t\t\t\tcontinue;\n+\t\t\t\t}\n+\t\t\t\tce = index_file_exists(state->istate, path->string,\n+\t\t\t\t\t\t       strlen(path->string), 0);\n+\t\t\t\terrs |= (ce ? checkout_entry(ce, state, NULL) : 1);\n+\t\t\t}\n+\t\t}\n+\t\tstring_list_remove_empty_items(&dco->filters, 0);\n+\t}\n+\tstring_list_clear(&dco->filters, 0);\n+\n+\t/* At this point we should not have any delayed paths anymore. */\n+\terrs |= dco->paths.nr;\n+\tfor_each_string_list_item(path, &dco->paths) {\n+\t\terror(\"'%s' was not filtered properly\", path->string);\n+\t}\n+\tstring_list_clear(&dco->paths, 0);\n+\n+\tfree(dco);\n+\tstate->delayed_checkout = NULL;\n+\n+\treturn errs;\n+}\n+\n static int write_entry(struct cache_entry *ce,\n \t\t       char *path, const struct checkout *state, int to_tempfile)\n {\n@@ -179,11 +278,34 @@ static int write_entry(struct cache_entry *ce,\n \t\t/*\n \t\t * Convert from git internal format to working tree format\n \t\t */\n-\t\tif (ce_mode_s_ifmt == S_IFREG &&\n-\t\t    convert_to_working_tree(ce->name, new, size, &buf)) {\n-\t\t\tfree(new);\n-\t\t\tnew = strbuf_detach(&buf, &newsize);\n-\t\t\tsize = newsize;\n+\t\tif (ce_mode_s_ifmt == S_IFREG) {\n+\t\t\tstruct delayed_checkout *dco = state->delayed_checkout;\n+\t\t\tif (dco && dco->state != CE_NO_DELAY) {\n+\t\t\t\t/* Do not send the blob in case of a retry. */\n+\t\t\t\tif (dco->state == CE_RETRY) {\n+\t\t\t\t\tnew = NULL;\n+\t\t\t\t\tsize = 0;\n+\t\t\t\t}\n+\t\t\t\tret = async_convert_to_working_tree(\n+\t\t\t\t\tce->name, new, size, &buf, dco);\n+\t\t\t\tif (ret && string_list_has_string(&dco->paths, ce->name)) {\n+\t\t\t\t\tfree(new);\n+\t\t\t\t\tgoto finish;\n+\t\t\t\t}\n+\t\t\t} else\n+\t\t\t\tret = convert_to_working_tree(\n+\t\t\t\t\tce->name, new, size, &buf);\n+\n+\t\t\tif (ret) {\n+\t\t\t\tfree(new);\n+\t\t\t\tnew = strbuf_detach(&buf, &newsize);\n+\t\t\t\tsize = newsize;\n+\t\t\t}\n+\t\t\t/*\n+\t\t\t * No \"else\" here as errors from convert are OK at this\n+\t\t\t * point. If the error would have been fatal (e.g.\n+\t\t\t * filter is required), then we would have died already.\n+\t\t\t */\n \t\t}\n \n \t\tfd = open_output_fd(path, ce, to_tempfile);\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 0c04d346a1..eb3d83744a 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -701,4 +701,120 @@ test_expect_success PERL 'invalid process filter must fail (and not hang!)' '\n \t)\n '\n \n+test_expect_success PERL 'delayed checkout in process filter' '\n+\ttest_config_global filter.a.process \"rot13-filter.pl a.log clean smudge delay\" &&\n+\ttest_config_global filter.a.required true &&\n+\ttest_config_global filter.b.process \"rot13-filter.pl b.log clean smudge delay\" &&\n+\ttest_config_global filter.b.required true &&\n+\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\t\techo \"*.a filter=a\" >.gitattributes &&\n+\t\techo \"*.b filter=b\" >>.gitattributes &&\n+\t\tcp \"$TEST_ROOT/test.o\" test.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" test-delay10.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" test-delay11.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" test-delay20.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" test-delay10.b &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"test commit\"\n+\t) &&\n+\n+\tS=$(file_size \"$TEST_ROOT/test.o\") &&\n+\tcat >a.exp <<-EOF &&\n+\t\tSTART\n+\t\tinit handshake complete\n+\t\tIN: smudge test.a $S [OK] -- OUT: $S . [OK]\n+\t\tIN: smudge test-delay10.a $S [OK] -- [DELAYED]\n+\t\tIN: smudge test-delay11.a $S [OK] -- [DELAYED]\n+\t\tIN: smudge test-delay20.a $S [OK] -- [DELAYED]\n+\t\tIN: list_available_blobs test-delay10.a test-delay11.a [OK]\n+\t\tIN: smudge test-delay10.a 0 [OK] -- OUT: $S . [OK]\n+\t\tIN: smudge test-delay11.a 0 [OK] -- OUT: $S . [OK]\n+\t\tIN: list_available_blobs test-delay20.a [OK]\n+\t\tIN: smudge test-delay20.a 0 [OK] -- OUT: $S . [OK]\n+\t\tIN: list_available_blobs [OK]\n+\t\tSTOP\n+\tEOF\n+\tcat >b.exp <<-EOF &&\n+\t\tSTART\n+\t\tinit handshake complete\n+\t\tIN: smudge test-delay10.b $S [OK] -- [DELAYED]\n+\t\tIN: list_available_blobs test-delay10.b [OK]\n+\t\tIN: smudge test-delay10.b 0 [OK] -- OUT: $S . [OK]\n+\t\tIN: list_available_blobs [OK]\n+\t\tSTOP\n+\tEOF\n+\n+\trm -rf repo-cloned &&\n+\tfilter_git clone repo repo-cloned &&\n+\ttest_cmp_count a.exp repo-cloned/a.log &&\n+\ttest_cmp_count b.exp repo-cloned/b.log &&\n+\n+\t(\n+\t\tcd repo-cloned &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay10.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay11.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay20.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay10.b &&\n+\n+\t\trm *.a *.b &&\n+\t\tfilter_git checkout . &&\n+\t\ttest_cmp_count ../a.exp a.log &&\n+\t\ttest_cmp_count ../b.exp b.log &&\n+\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay10.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay11.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay20.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay10.b\n+\t)\n+'\n+\n+test_expect_success PERL 'missing file in delayed checkout' '\n+\ttest_config_global filter.bug.process \"rot13-filter.pl bug.log clean smudge delay\" &&\n+\ttest_config_global filter.bug.required true &&\n+\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\t\techo \"*.a filter=bug\" >.gitattributes &&\n+\t\tcp \"$TEST_ROOT/test.o\" missing-delay.a\n+\t\tgit add . &&\n+\t\tgit commit -m \"test commit\"\n+\t) &&\n+\n+\trm -rf repo-cloned &&\n+\ttest_must_fail git clone repo repo-cloned 2>git-stderr.log &&\n+\tcat git-stderr.log &&\n+\tgrep \"error: .missing-delay\\.a. was not filtered properly\" git-stderr.log\n+'\n+\n+test_expect_success PERL 'invalid file in delayed checkout' '\n+\ttest_config_global filter.bug.process \"rot13-filter.pl bug.log clean smudge delay\" &&\n+\ttest_config_global filter.bug.required true &&\n+\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\t\techo \"*.a filter=bug\" >.gitattributes &&\n+\t\tcp \"$TEST_ROOT/test.o\" invalid-delay.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" unfiltered\n+\t\tgit add . &&\n+\t\tgit commit -m \"test commit\"\n+\t) &&\n+\n+\trm -rf repo-cloned &&\n+\ttest_must_fail git clone repo repo-cloned 2>git-stderr.log &&\n+\tgrep \"error: external filter .* signaled that .unfiltered. is now available although it has not been delayed earlier\" git-stderr.log\n+'\n+\n test_done\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 5e43faeec1..ad685d92f8 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -18,6 +18,16 @@\n #     operation then the filter signals that it cannot or does not want\n #     to process the file and any file after that is processed with the\n #     same command.\n+# (5) If data with a pathname that is a key in the DELAY hash is\n+#     requested (e.g. \"test-delay10.a\") then the filter responds with\n+#     a \"delay\" status and sets the \"requested\" field in the DELAY hash.\n+#     The filter will signal the availability of this object after\n+#     \"count\" (field in DELAY hash) \"list_available_blobs\" commands.\n+# (6) If data with the pathname \"missing-delay.a\" is processed that the\n+#     filter will drop the path from the \"list_available_blobs\" response.\n+# (7) If data with the pathname \"invalid-delay.a\" is processed that the\n+#     filter will add the path \"unfiltered\" which was not delayed before\n+#     to the \"list_available_blobs\" response.\n #\n \n use strict;\n@@ -30,6 +40,15 @@ my @capabilities            = @ARGV;\n \n open my $debug, \">>\", $log_file or die \"cannot open log file: $!\";\n \n+my %DELAY = (\n+\t'test-delay10.a' => { \"requested\" => 0, \"count\" => 1 },\n+\t'test-delay11.a' => { \"requested\" => 0, \"count\" => 1 },\n+\t'test-delay20.a' => { \"requested\" => 0, \"count\" => 2 },\n+\t'test-delay10.b' => { \"requested\" => 0, \"count\" => 1 },\n+\t'missing-delay.a' => { \"requested\" => 0, \"count\" => 1 },\n+\t'invalid-delay.a' => { \"requested\" => 0, \"count\" => 1 },\n+);\n+\n sub rot13 {\n \tmy $str = shift;\n \t$str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n@@ -66,7 +85,7 @@ sub packet_bin_read {\n \n sub packet_txt_read {\n \tmy ( $res, $buf ) = packet_bin_read();\n-\tunless ( $buf =~ s/\\n$// ) {\n+\tunless ( $buf eq '' or $buf =~ s/\\n$// ) {\n \t\tdie \"A non-binary line MUST be terminated by an LF.\";\n \t}\n \treturn ( $res, $buf );\n@@ -101,6 +120,7 @@ packet_flush();\n \n ( packet_txt_read() eq ( 0, \"capability=clean\" ) )  || die \"bad capability\";\n ( packet_txt_read() eq ( 0, \"capability=smudge\" ) ) || die \"bad capability\";\n+( packet_txt_read() eq ( 0, \"capability=delay\" ) )  || die \"bad capability\";\n ( packet_bin_read() eq ( 1, \"\" ) )                  || die \"bad capability end\";\n \n foreach (@capabilities) {\n@@ -111,88 +131,142 @@ print $debug \"init handshake complete\\n\";\n $debug->flush();\n \n while (1) {\n-\tmy ($command) = packet_txt_read() =~ /^command=(.+)$/;\n+\tmy ( $command ) = packet_txt_read() =~ /^command=(.+)$/;\n \tprint $debug \"IN: $command\";\n \t$debug->flush();\n \n-\tmy ($pathname) = packet_txt_read() =~ /^pathname=(.+)$/;\n-\tprint $debug \" $pathname\";\n-\t$debug->flush();\n-\n-\tif ( $pathname eq \"\" ) {\n-\t\tdie \"bad pathname '$pathname'\";\n-\t}\n+\tif ( $command eq \"list_available_blobs\" ) {\n+\t\t# Flush\n+\t\tpacket_bin_read();\n \n-\t# Flush\n-\tpacket_bin_read();\n-\n-\tmy $input = \"\";\n-\t{\n-\t\tbinmode(STDIN);\n-\t\tmy $buffer;\n-\t\tmy $done = 0;\n-\t\twhile ( !$done ) {\n-\t\t\t( $done, $buffer ) = packet_bin_read();\n-\t\t\t$input .= $buffer;\n+\t\tforeach my $pathname ( sort keys %DELAY ) {\n+\t\t\tif ( $DELAY{$pathname}{\"requested\"} >= 1 ) {\n+\t\t\t\t$DELAY{$pathname}{\"count\"} = $DELAY{$pathname}{\"count\"} - 1;\n+\t\t\t\tif ( $pathname eq \"invalid-delay.a\" ) {\n+\t\t\t\t\t# Send Git a pathname that was not delayed earlier\n+\t\t\t\t\tpacket_txt_write(\"pathname=unfiltered\");\n+\t\t\t\t}\n+\t\t\t\tif ( $pathname eq \"missing-delay.a\" ) {\n+\t\t\t\t\t# Do not signal Git that this file is available\n+\t\t\t\t} elsif ( $DELAY{$pathname}{\"count\"} == 0 ) {\n+\t\t\t\t\tprint $debug \" $pathname\";\n+\t\t\t\t\tpacket_txt_write(\"pathname=$pathname\");\n+\t\t\t\t}\n+\t\t\t}\n \t\t}\n-\t\tprint $debug \" \" . length($input) . \" [OK] -- \";\n-\t\t$debug->flush();\n-\t}\n-\n-\tmy $output;\n-\tif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n-\t\t$output = \"\";\n-\t}\n-\telsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n-\t\t$output = rot13($input);\n-\t}\n-\telsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n-\t\t$output = rot13($input);\n-\t}\n-\telse {\n-\t\tdie \"bad command '$command'\";\n-\t}\n \n-\tif ( $pathname eq \"error.r\" ) {\n-\t\tprint $debug \"[ERROR]\\n\";\n-\t\t$debug->flush();\n-\t\tpacket_txt_write(\"status=error\");\n \t\tpacket_flush();\n-\t}\n-\telsif ( $pathname eq \"abort.r\" ) {\n-\t\tprint $debug \"[ABORT]\\n\";\n+\n+\t\tprint $debug \" [OK]\\n\";\n \t\t$debug->flush();\n-\t\tpacket_txt_write(\"status=abort\");\n+\t\tpacket_txt_write(\"status=success\");\n \t\tpacket_flush();\n \t}\n \telse {\n-\t\tpacket_txt_write(\"status=success\");\n-\t\tpacket_flush();\n+\t\tmy ( $pathname ) = packet_txt_read() =~ /^pathname=(.+)$/;\n+\t\tprint $debug \" $pathname\";\n+\t\t$debug->flush();\n+\n+\t\tif ( $pathname eq \"\" ) {\n+\t\t\tdie \"bad pathname '$pathname'\";\n+\t\t}\n+\n+\t\t# Read until flush\n+\t\tmy ( $done, $buffer ) = packet_txt_read();\n+\t\twhile ( $buffer ne '' ) {\n+\t\t\tif ( $buffer eq \"can-delay=1\" ) {\n+\t\t\t\tif ( exists $DELAY{$pathname} and $DELAY{$pathname}{\"requested\"} == 0 ) {\n+\t\t\t\t\t$DELAY{$pathname}{\"requested\"} = 1;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tdie \"Unknown message '$buffer'\";\n+\t\t\t}\n \n-\t\tif ( $pathname eq \"${command}-write-fail.r\" ) {\n-\t\t\tprint $debug \"[WRITE FAIL]\\n\";\n+\t\t\t( $done, $buffer ) = packet_txt_read();\n+\t\t}\n+\n+\t\tmy $input = \"\";\n+\t\t{\n+\t\t\tbinmode(STDIN);\n+\t\t\tmy $buffer;\n+\t\t\tmy $done = 0;\n+\t\t\twhile ( !$done ) {\n+\t\t\t\t( $done, $buffer ) = packet_bin_read();\n+\t\t\t\t$input .= $buffer;\n+\t\t\t}\n+\t\t\tprint $debug \" \" . length($input) . \" [OK] -- \";\n \t\t\t$debug->flush();\n-\t\t\tdie \"${command} write error\";\n \t\t}\n \n-\t\tprint $debug \"OUT: \" . length($output) . \" \";\n-\t\t$debug->flush();\n+\t\tmy $output;\n+\t\tif ( exists $DELAY{$pathname} and exists $DELAY{$pathname}{\"output\"} ) {\n+\t\t\t$output = $DELAY{$pathname}{\"output\"}\n+\t\t}\n+\t\telsif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n+\t\t\t$output = \"\";\n+\t\t}\n+\t\telsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n+\t\t\t$output = rot13($input);\n+\t\t}\n+\t\telsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n+\t\t\t$output = rot13($input);\n+\t\t}\n+\t\telse {\n+\t\t\tdie \"bad command '$command'\";\n+\t\t}\n+\n+\t\tif ( $pathname eq \"error.r\" ) {\n+\t\t\tprint $debug \"[ERROR]\\n\";\n+\t\t\t$debug->flush();\n+\t\t\tpacket_txt_write(\"status=error\");\n+\t\t\tpacket_flush();\n+\t\t}\n+\t\telsif ( $pathname eq \"abort.r\" ) {\n+\t\t\tprint $debug \"[ABORT]\\n\";\n+\t\t\t$debug->flush();\n+\t\t\tpacket_txt_write(\"status=abort\");\n+\t\t\tpacket_flush();\n+\t\t}\n+\t\telsif ( $command eq \"smudge\" and\n+\t\t\texists $DELAY{$pathname} and\n+\t\t\t$DELAY{$pathname}{\"requested\"} == 1\n+\t\t) {\n+\t\t\tprint $debug \"[DELAYED]\\n\";\n+\t\t\t$debug->flush();\n+\t\t\tpacket_txt_write(\"status=delayed\");\n+\t\t\tpacket_flush();\n+\t\t\t$DELAY{$pathname}{\"requested\"} = 2;\n+\t\t\t$DELAY{$pathname}{\"output\"} = $output;\n+\t\t}\n+\t\telse {\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t\tpacket_flush();\n \n-\t\twhile ( length($output) > 0 ) {\n-\t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\tpacket_bin_write($packet);\n-\t\t\t# dots represent the number of packets\n-\t\t\tprint $debug \".\";\n-\t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n-\t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n+\t\t\tif ( $pathname eq \"${command}-write-fail.r\" ) {\n+\t\t\t\tprint $debug \"[WRITE FAIL]\\n\";\n+\t\t\t\t$debug->flush();\n+\t\t\t\tdie \"${command} write error\";\n \t\t\t}\n-\t\t\telse {\n-\t\t\t\t$output = \"\";\n+\n+\t\t\tprint $debug \"OUT: \" . length($output) . \" \";\n+\t\t\t$debug->flush();\n+\n+\t\t\twhile ( length($output) > 0 ) {\n+\t\t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n+\t\t\t\tpacket_bin_write($packet);\n+\t\t\t\t# dots represent the number of packets\n+\t\t\t\tprint $debug \".\";\n+\t\t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n+\t\t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n+\t\t\t\t}\n+\t\t\t\telse {\n+\t\t\t\t\t$output = \"\";\n+\t\t\t\t}\n \t\t\t}\n+\t\t\tpacket_flush();\n+\t\t\tprint $debug \" [OK]\\n\";\n+\t\t\t$debug->flush();\n+\t\t\tpacket_flush();\n \t\t}\n-\t\tpacket_flush();\n-\t\tprint $debug \" [OK]\\n\";\n-\t\t$debug->flush();\n-\t\tpacket_flush();\n \t}\n }\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d38c37e38c..491a2562ad 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -379,6 +379,7 @@ static int check_updates(struct unpack_trees_options *o)\n \tif (should_update_submodules() && o->update && !o->dry_run)\n \t\treload_gitmodules_file(index, &state);\n \n+\tenable_delayed_checkout(&state);\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -393,6 +394,7 @@ static int check_updates(struct unpack_trees_options *o)\n \t\t\t}\n \t\t}\n \t}\n+\terrs |= finish_delayed_checkout(&state);\n \tstop_progress(&progress);\n \tif (o->update)\n \t\tgit_attr_set_direction(GIT_ATTR_CHECKIN, NULL);\n-- \n2.13.2\n\n"},{"id":"323614","messageId":"20170630204128.48708-7-larsxschneider@gmail.com","threadId":"46298","inReplyTo":"20170630204128.48708-1-larsxschneider@gmail.com","subject":"[PATCH v9 6/7] convert: refactor capabilities negotiation","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-30T20:41:27Z","receivedAt":"2017-06-30T20:41:59Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"The code to negotiate long running filter capabilities was very\nrepetitive for new capabilities. Replace the repetitive conditional\nstatements with a table-driven approach. This is useful for the\nsubsequent patch 'convert: add \"status=delayed\" to filter process\nprotocol'.\n\nSuggested-by: Junio C Hamano <gitster@pobox.com>\n---\n convert.c | 39 +++++++++++++++++++++++++++------------\n 1 file changed, 27 insertions(+), 12 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex e55c034d86..d13e505dfb 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -507,7 +507,7 @@ static struct hashmap subprocess_map;\n \n static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n {\n-\tint err;\n+\tint err, i;\n \tstruct cmd2process *entry = (struct cmd2process *)subprocess;\n \tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n \tchar *cap_buf;\n@@ -515,6 +515,14 @@ static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n \tstruct child_process *process = &subprocess->process;\n \tconst char *cmd = subprocess->cmd;\n \n+\tstatic const struct {\n+\t\tconst char *name;\n+\t\tunsigned int cap;\n+\t} known_caps[] = {\n+\t\t{ \"clean\",  CAP_CLEAN  },\n+\t\t{ \"smudge\", CAP_SMUDGE },\n+\t};\n+\n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n \terr = packet_writel(process->in, \"git-filter-client\", \"version=2\", NULL);\n@@ -533,7 +541,15 @@ static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n \tif (err)\n \t\tgoto done;\n \n-\terr = packet_writel(process->in, \"capability=clean\", \"capability=smudge\", NULL);\n+\tfor (i = 0; i < ARRAY_SIZE(known_caps); ++i) {\n+\t\terr = packet_write_fmt_gently(\n+\t\t\tprocess->in, \"capability=%s\\n\", known_caps[i].name);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t}\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\tgoto done;\n \n \tfor (;;) {\n \t\tcap_buf = packet_read_line(process->out, NULL);\n@@ -545,16 +561,15 @@ static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n \t\t\tcontinue;\n \n \t\tcap_name = cap_list.items[1].string;\n-\t\tif (!strcmp(cap_name, \"clean\")) {\n-\t\t\tentry->supported_capabilities |= CAP_CLEAN;\n-\t\t} else if (!strcmp(cap_name, \"smudge\")) {\n-\t\t\tentry->supported_capabilities |= CAP_SMUDGE;\n-\t\t} else {\n-\t\t\twarning(\n-\t\t\t\t\"external filter '%s' requested unsupported filter capability '%s'\",\n-\t\t\t\tcmd, cap_name\n-\t\t\t);\n-\t\t}\n+\t\ti = ARRAY_SIZE(known_caps) - 1;\n+\t\twhile (i >= 0 && strcmp(cap_name, known_caps[i].name))\n+\t\t\ti--;\n+\n+\t\tif (i >= 0)\n+\t\t\tentry->supported_capabilities |= known_caps[i].cap;\n+\t\telse\n+\t\t\twarning(\"external filter '%s' requested unsupported filter capability '%s'\",\n+\t\t\tcmd, cap_name);\n \n \t\tstring_list_clear(&cap_list, 0);\n \t}\n-- \n2.13.2\n\n"},{"id":"323615","messageId":"20170630204128.48708-3-larsxschneider@gmail.com","threadId":"46298","inReplyTo":"20170630204128.48708-1-larsxschneider@gmail.com","subject":"[PATCH v9 2/7] t0021: make debug log file name configurable","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-30T20:41:23Z","receivedAt":"2017-06-30T20:42:24Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"The \"rot13-filter.pl\" helper wrote its debug logs always to \"rot13-filter.log\".\nMake this configurable by defining the log file as first parameter of\n\"rot13-filter.pl\".\n\nThis is useful if \"rot13-filter.pl\" is configured multiple times similar to the\nsubsequent patch 'convert: add \"status=delayed\" to filter process protocol'.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n t/t0021-conversion.sh   | 44 ++++++++++++++++++++++----------------------\n t/t0021/rot13-filter.pl |  8 +++++---\n 2 files changed, 27 insertions(+), 25 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex ff2424225b..0139b460e7 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -28,7 +28,7 @@ file_size () {\n }\n \n filter_git () {\n-\trm -f rot13-filter.log &&\n+\trm -f *.log &&\n \tgit \"$@\"\n }\n \n@@ -342,7 +342,7 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '\n '\n \n test_expect_success PERL 'required process filter should filter data' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -375,7 +375,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \t\t\tIN: clean testsubdir/test3 '\\''sq'\\'',\\$x=.r $S3 [OK] -- OUT: $S3 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_count expected.log rot13-filter.log &&\n+\t\ttest_cmp_count expected.log debug.log &&\n \n \t\tgit commit -m \"test commit 2\" &&\n \t\trm -f test2.r \"testsubdir/test3 '\\''sq'\\'',\\$x=.r\" &&\n@@ -388,7 +388,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \t\t\tIN: smudge testsubdir/test3 '\\''sq'\\'',\\$x=.r $S3 [OK] -- OUT: $S3 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\tfilter_git checkout --quiet --no-progress empty-branch &&\n \t\tcat >expected.log <<-EOF &&\n@@ -397,7 +397,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \t\t\tIN: clean test.r $S [OK] -- OUT: $S . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\tfilter_git checkout --quiet --no-progress master &&\n \t\tcat >expected.log <<-EOF &&\n@@ -409,7 +409,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \t\t\tIN: smudge testsubdir/test3 '\\''sq'\\'',\\$x=.r $S3 [OK] -- OUT: $S3 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.r &&\n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test2.o\" test2.r &&\n@@ -419,7 +419,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \n test_expect_success PERL 'required process filter takes precedence' '\n \ttest_config_global filter.protocol.clean false &&\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -439,12 +439,12 @@ test_expect_success PERL 'required process filter takes precedence' '\n \t\t\tIN: clean test.r $S [OK] -- OUT: $S . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_count expected.log rot13-filter.log\n+\t\ttest_cmp_count expected.log debug.log\n \t)\n '\n \n test_expect_success PERL 'required process filter should be used only for \"clean\" operation only' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -462,7 +462,7 @@ test_expect_success PERL 'required process filter should be used only for \"clean\n \t\t\tIN: clean test.r $S [OK] -- OUT: $S . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_count expected.log rot13-filter.log &&\n+\t\ttest_cmp_count expected.log debug.log &&\n \n \t\trm test.r &&\n \n@@ -474,12 +474,12 @@ test_expect_success PERL 'required process filter should be used only for \"clean\n \t\t\tinit handshake complete\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log\n+\t\ttest_cmp_exclude_clean expected.log debug.log\n \t)\n '\n \n test_expect_success PERL 'required process filter should process multiple packets' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \n \trm -rf repo &&\n@@ -514,7 +514,7 @@ test_expect_success PERL 'required process filter should process multiple packet\n \t\t\tIN: clean 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_count expected.log rot13-filter.log &&\n+\t\ttest_cmp_count expected.log debug.log &&\n \n \t\trm -f *.file &&\n \n@@ -529,7 +529,7 @@ test_expect_success PERL 'required process filter should process multiple packet\n \t\t\tIN: smudge 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\tfor FILE in *.file\n \t\tdo\n@@ -539,7 +539,7 @@ test_expect_success PERL 'required process filter should process multiple packet\n '\n \n test_expect_success PERL 'required process filter with clean error should fail' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -558,7 +558,7 @@ test_expect_success PERL 'required process filter with clean error should fail'\n '\n \n test_expect_success PERL 'process filter should restart after unexpected write failure' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -579,7 +579,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\tgit add . &&\n \t\trm -f *.r &&\n \n-\t\trm -f rot13-filter.log &&\n+\t\trm -f debug.log &&\n \t\tgit checkout --quiet --no-progress . 2>git-stderr.log &&\n \n \t\tgrep \"smudge write error at\" git-stderr.log &&\n@@ -595,7 +595,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\t\tIN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.r &&\n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test2.o\" test2.r &&\n@@ -609,7 +609,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n '\n \n test_expect_success PERL 'process filter should not be restarted if it signals an error' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -639,7 +639,7 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n \t\t\tIN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.r &&\n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test2.o\" test2.r &&\n@@ -648,7 +648,7 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n '\n \n test_expect_success PERL 'process filter abort stops processing of all further files' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -676,7 +676,7 @@ test_expect_success PERL 'process filter abort stops processing of all further f\n \t\t\tIN: smudge abort.r $SA [OK] -- OUT: 0 [ABORT]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\ttest_cmp \"$TEST_ROOT/test.o\" test.r &&\n \t\ttest_cmp \"$TEST_ROOT/test2.o\" test2.r &&\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 617f581e56..0b943bb377 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -2,8 +2,9 @@\n # Example implementation for the Git filter protocol version 2\n # See Documentation/gitattributes.txt, section \"Filter Protocol\"\n #\n-# The script takes the list of supported protocol capabilities as\n-# arguments (\"clean\", \"smudge\", etc).\n+# The first argument defines a debug log file that the script write to.\n+# All remaining arguments define a list of supported protocol\n+# capabilities (\"clean\", \"smudge\", etc).\n #\n # This implementation supports special test cases:\n # (1) If data with the pathname \"clean-write-fail.r\" is processed with\n@@ -24,9 +25,10 @@ use warnings;\n use IO::File;\n \n my $MAX_PACKET_CONTENT_SIZE = 65516;\n+my $log_file                = shift @ARGV;\n my @capabilities            = @ARGV;\n \n-open my $debug, \">>\", \"rot13-filter.log\" or die \"cannot open log file: $!\";\n+open my $debug, \">>\", $log_file or die \"cannot open log file: $!\";\n \n sub rot13 {\n \tmy $str = shift;\n-- \n2.13.2\n\n"},{"id":"323645","messageId":"xmqqwp7tp3ho.fsf@gitster.mtv.corp.google.com","threadId":"46298","inReplyTo":"20170630204128.48708-1-larsxschneider@gmail.com","subject":"Re: [PATCH v9 0/7] convert: add \"status=delayed\" to filter process protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-30T21:30:59Z","receivedAt":"2017-06-30T21:31:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Will queue.  I personally feel that this is polished enough for 'next'.\n\nThanks.\n"}]}