{"thread":{"id":"46020","subject":"[PATCH v4 0/4] convert: add \"status=delayed\" to filter process protocol","startedAt":"2017-05-22T13:50:16Z","lastAt":"2017-05-30T01:56:04Z","messageCount":15,"participants":["Lars Schneider","Taylor Blau","Torsten Bögershausen","Junio C Hamano"],"isPatch":true,"patchVersion":4,"patchTotal":4},"messages":[{"id":"320412","messageId":"20170522135001.54506-1-larsxschneider@gmail.com","threadId":"46020","inReplyTo":null,"subject":"[PATCH v4 0/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-22T13:49:57Z","receivedAt":"2017-05-22T13:50:16Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Hi,\n\nhere is the 4th iteration of my \"status delayed\" topic. I am sorry for\nthe large delays between iterations. Patch 1 to 3 are minor t0021 test\nadjustments and haven't been changed since v3. Patch 4 is the new feature.\n\nMost significant changes since v3:\n* removed \"delay-id\" approach. Delayed blobs are identified by their pathname\n  (suggested by Taylor, the main user of this new API, see\n   http://public-inbox.org/git/20170412173404.GA49694@Ida/ )\n* rebased the topic onto master (2.13)\n* moved \"struct delayed_checkout\" and \"enum ce_delay_state\" to convert.h\n\nIf you review this series then please read the \"Delay\" section in\n\"Documentation/gitattributes.txt\" first for an overview of the delay mechanism.\nThe changes in \"t/t0021/rot13-filter.pl\" are easier to review if you ignore\nwhitespace changes.\n\nThanks,\nLars\n\nRFC: http://public-inbox.org/git/D10F7C47-14E8-465B-8B7A-A09A1B28A39F@gmail.com/\nv1: http://public-inbox.org/git/20170108191736.47359-1-larsxschneider@gmail.com/\nv2: http://public-inbox.org/git/20170226184816.30010-1-larsxschneider@gmail.com/\nv3: http://public-inbox.org/git/20170409191107.20547-1-larsxschneider@gmail.com/\n\n\nLars Schneider (4):\n  t0021: keep filter log files on comparison\n  t0021: make debug log file name configurable\n  t0021: write \"OUT\" only on success\n  convert: add \"status=delayed\" to filter process protocol\n\n Documentation/gitattributes.txt |  65 ++++++++++++-\n builtin/checkout.c              |   3 +\n cache.h                         |   6 +-\n convert.c                       | 131 ++++++++++++++++++++++----\n convert.h                       |  21 +++++\n entry.c                         | 110 +++++++++++++++++++++-\n t/t0021-conversion.sh           | 136 ++++++++++++++++++++-------\n t/t0021/rot13-filter.pl         | 199 ++++++++++++++++++++++++++--------------\n unpack-trees.c                  |   2 +\n 9 files changed, 544 insertions(+), 129 deletions(-)\n\n\nbase-commit: 10c78a162fa821ee85203165b805ff46be454091\n--\n2.13.0\n\n"},{"id":"320413","messageId":"20170522135001.54506-3-larsxschneider@gmail.com","threadId":"46020","inReplyTo":"20170522135001.54506-1-larsxschneider@gmail.com","subject":"[PATCH v4 2/4] t0021: make debug log file name configurable","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-22T13:49:59Z","receivedAt":"2017-05-22T13:50:19Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"The \"rot13-filter.pl\" helper wrote its debug logs always to \"rot13-filter.log\".\nMake this configurable by defining the log file as first parameter of\n\"rot13-filter.pl\".\n\nThis is useful if \"rot13-filter.pl\" is configured multiple times similar to the\nsubsequent patch 'convert: add \"status=delayed\" to filter process protocol'.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n t/t0021-conversion.sh   | 44 ++++++++++++++++++++++----------------------\n t/t0021/rot13-filter.pl |  8 +++++---\n 2 files changed, 27 insertions(+), 25 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex ff2424225b..0139b460e7 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -28,7 +28,7 @@ file_size () {\n }\n \n filter_git () {\n-\trm -f rot13-filter.log &&\n+\trm -f *.log &&\n \tgit \"$@\"\n }\n \n@@ -342,7 +342,7 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '\n '\n \n test_expect_success PERL 'required process filter should filter data' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -375,7 +375,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \t\t\tIN: clean testsubdir/test3 '\\''sq'\\'',\\$x=.r $S3 [OK] -- OUT: $S3 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_count expected.log rot13-filter.log &&\n+\t\ttest_cmp_count expected.log debug.log &&\n \n \t\tgit commit -m \"test commit 2\" &&\n \t\trm -f test2.r \"testsubdir/test3 '\\''sq'\\'',\\$x=.r\" &&\n@@ -388,7 +388,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \t\t\tIN: smudge testsubdir/test3 '\\''sq'\\'',\\$x=.r $S3 [OK] -- OUT: $S3 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\tfilter_git checkout --quiet --no-progress empty-branch &&\n \t\tcat >expected.log <<-EOF &&\n@@ -397,7 +397,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \t\t\tIN: clean test.r $S [OK] -- OUT: $S . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\tfilter_git checkout --quiet --no-progress master &&\n \t\tcat >expected.log <<-EOF &&\n@@ -409,7 +409,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \t\t\tIN: smudge testsubdir/test3 '\\''sq'\\'',\\$x=.r $S3 [OK] -- OUT: $S3 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.r &&\n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test2.o\" test2.r &&\n@@ -419,7 +419,7 @@ test_expect_success PERL 'required process filter should filter data' '\n \n test_expect_success PERL 'required process filter takes precedence' '\n \ttest_config_global filter.protocol.clean false &&\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -439,12 +439,12 @@ test_expect_success PERL 'required process filter takes precedence' '\n \t\t\tIN: clean test.r $S [OK] -- OUT: $S . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_count expected.log rot13-filter.log\n+\t\ttest_cmp_count expected.log debug.log\n \t)\n '\n \n test_expect_success PERL 'required process filter should be used only for \"clean\" operation only' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -462,7 +462,7 @@ test_expect_success PERL 'required process filter should be used only for \"clean\n \t\t\tIN: clean test.r $S [OK] -- OUT: $S . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_count expected.log rot13-filter.log &&\n+\t\ttest_cmp_count expected.log debug.log &&\n \n \t\trm test.r &&\n \n@@ -474,12 +474,12 @@ test_expect_success PERL 'required process filter should be used only for \"clean\n \t\t\tinit handshake complete\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log\n+\t\ttest_cmp_exclude_clean expected.log debug.log\n \t)\n '\n \n test_expect_success PERL 'required process filter should process multiple packets' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \n \trm -rf repo &&\n@@ -514,7 +514,7 @@ test_expect_success PERL 'required process filter should process multiple packet\n \t\t\tIN: clean 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_count expected.log rot13-filter.log &&\n+\t\ttest_cmp_count expected.log debug.log &&\n \n \t\trm -f *.file &&\n \n@@ -529,7 +529,7 @@ test_expect_success PERL 'required process filter should process multiple packet\n \t\t\tIN: smudge 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\tfor FILE in *.file\n \t\tdo\n@@ -539,7 +539,7 @@ test_expect_success PERL 'required process filter should process multiple packet\n '\n \n test_expect_success PERL 'required process filter with clean error should fail' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -558,7 +558,7 @@ test_expect_success PERL 'required process filter with clean error should fail'\n '\n \n test_expect_success PERL 'process filter should restart after unexpected write failure' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -579,7 +579,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\tgit add . &&\n \t\trm -f *.r &&\n \n-\t\trm -f rot13-filter.log &&\n+\t\trm -f debug.log &&\n \t\tgit checkout --quiet --no-progress . 2>git-stderr.log &&\n \n \t\tgrep \"smudge write error at\" git-stderr.log &&\n@@ -595,7 +595,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\t\tIN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.r &&\n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test2.o\" test2.r &&\n@@ -609,7 +609,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n '\n \n test_expect_success PERL 'process filter should not be restarted if it signals an error' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -639,7 +639,7 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n \t\t\tIN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.r &&\n \t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test2.o\" test2.r &&\n@@ -648,7 +648,7 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n '\n \n test_expect_success PERL 'process filter abort stops processing of all further files' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -676,7 +676,7 @@ test_expect_success PERL 'process filter abort stops processing of all further f\n \t\t\tIN: smudge abort.r $SA [OK] -- OUT: 0 [ABORT]\n \t\t\tSTOP\n \t\tEOF\n-\t\ttest_cmp_exclude_clean expected.log rot13-filter.log &&\n+\t\ttest_cmp_exclude_clean expected.log debug.log &&\n \n \t\ttest_cmp \"$TEST_ROOT/test.o\" test.r &&\n \t\ttest_cmp \"$TEST_ROOT/test2.o\" test2.r &&\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 617f581e56..0b943bb377 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -2,8 +2,9 @@\n # Example implementation for the Git filter protocol version 2\n # See Documentation/gitattributes.txt, section \"Filter Protocol\"\n #\n-# The script takes the list of supported protocol capabilities as\n-# arguments (\"clean\", \"smudge\", etc).\n+# The first argument defines a debug log file that the script write to.\n+# All remaining arguments define a list of supported protocol\n+# capabilities (\"clean\", \"smudge\", etc).\n #\n # This implementation supports special test cases:\n # (1) If data with the pathname \"clean-write-fail.r\" is processed with\n@@ -24,9 +25,10 @@ use warnings;\n use IO::File;\n \n my $MAX_PACKET_CONTENT_SIZE = 65516;\n+my $log_file                = shift @ARGV;\n my @capabilities            = @ARGV;\n \n-open my $debug, \">>\", \"rot13-filter.log\" or die \"cannot open log file: $!\";\n+open my $debug, \">>\", $log_file or die \"cannot open log file: $!\";\n \n sub rot13 {\n \tmy $str = shift;\n-- \n2.13.0\n\n"},{"id":"320414","messageId":"20170522135001.54506-2-larsxschneider@gmail.com","threadId":"46020","inReplyTo":"20170522135001.54506-1-larsxschneider@gmail.com","subject":"[PATCH v4 1/4] t0021: keep filter log files on comparison","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-22T13:49:58Z","receivedAt":"2017-05-22T13:50:20Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"The filter log files are modified on comparison. Write the modified log files\nto temp files for comparison to fix this.\n\nThis is useful for the subsequent patch 'convert: add \"status=delayed\" to\nfilter process protocol'.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n t/t0021-conversion.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 161f560446..ff2424225b 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -42,10 +42,10 @@ test_cmp_count () {\n \tfor FILE in \"$expect\" \"$actual\"\n \tdo\n \t\tsort \"$FILE\" | uniq -c |\n-\t\tsed -e \"s/^ *[0-9][0-9]*[ \t]*IN: /x IN: /\" >\"$FILE.tmp\" &&\n-\t\tmv \"$FILE.tmp\" \"$FILE\" || return\n+\t\tsed -e \"s/^ *[0-9][0-9]*[ \t]*IN: /x IN: /\" >\"$FILE.tmp\"\n \tdone &&\n-\ttest_cmp \"$expect\" \"$actual\"\n+\ttest_cmp \"$expect.tmp\" \"$actual.tmp\" &&\n+\trm \"$expect.tmp\" \"$actual.tmp\"\n }\n \n # Compare two files but exclude all `clean` invocations because Git can\n@@ -56,10 +56,10 @@ test_cmp_exclude_clean () {\n \tactual=$2\n \tfor FILE in \"$expect\" \"$actual\"\n \tdo\n-\t\tgrep -v \"IN: clean\" \"$FILE\" >\"$FILE.tmp\" &&\n-\t\tmv \"$FILE.tmp\" \"$FILE\"\n+\t\tgrep -v \"IN: clean\" \"$FILE\" >\"$FILE.tmp\"\n \tdone &&\n-\ttest_cmp \"$expect\" \"$actual\"\n+\ttest_cmp \"$expect.tmp\" \"$actual.tmp\" &&\n+\trm \"$expect.tmp\" \"$actual.tmp\"\n }\n \n # Check that the contents of two files are equal and that their rot13 version\n-- \n2.13.0\n\n"},{"id":"320415","messageId":"20170522135001.54506-4-larsxschneider@gmail.com","threadId":"46020","inReplyTo":"20170522135001.54506-1-larsxschneider@gmail.com","subject":"[PATCH v4 3/4] t0021: write \"OUT\" only on success","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-22T13:50:00Z","receivedAt":"2017-05-22T13:50:21Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\"rot13-filter.pl\" used to write \"OUT <size>\" to the debug log even in case of\nan abort or error. Fix this by writing \"OUT <size>\" to the debug log only in\nthe successful case if output is actually written.\n\nThis is useful for the subsequent patch 'convert: add \"status=delayed\" to\nfilter process protocol'.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n t/t0021-conversion.sh   | 6 +++---\n t/t0021/rot13-filter.pl | 6 +++---\n 2 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 0139b460e7..0c04d346a1 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -588,7 +588,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\tcat >expected.log <<-EOF &&\n \t\t\tSTART\n \t\t\tinit handshake complete\n-\t\t\tIN: smudge smudge-write-fail.r $SF [OK] -- OUT: $SF [WRITE FAIL]\n+\t\t\tIN: smudge smudge-write-fail.r $SF [OK] -- [WRITE FAIL]\n \t\t\tSTART\n \t\t\tinit handshake complete\n \t\t\tIN: smudge test.r $S [OK] -- OUT: $S . [OK]\n@@ -634,7 +634,7 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n \t\tcat >expected.log <<-EOF &&\n \t\t\tSTART\n \t\t\tinit handshake complete\n-\t\t\tIN: smudge error.r $SE [OK] -- OUT: 0 [ERROR]\n+\t\t\tIN: smudge error.r $SE [OK] -- [ERROR]\n \t\t\tIN: smudge test.r $S [OK] -- OUT: $S . [OK]\n \t\t\tIN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]\n \t\t\tSTOP\n@@ -673,7 +673,7 @@ test_expect_success PERL 'process filter abort stops processing of all further f\n \t\tcat >expected.log <<-EOF &&\n \t\t\tSTART\n \t\t\tinit handshake complete\n-\t\t\tIN: smudge abort.r $SA [OK] -- OUT: 0 [ABORT]\n+\t\t\tIN: smudge abort.r $SA [OK] -- [ABORT]\n \t\t\tSTOP\n \t\tEOF\n \t\ttest_cmp_exclude_clean expected.log debug.log &&\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 0b943bb377..5e43faeec1 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -153,9 +153,6 @@ while (1) {\n \t\tdie \"bad command '$command'\";\n \t}\n \n-\tprint $debug \"OUT: \" . length($output) . \" \";\n-\t$debug->flush();\n-\n \tif ( $pathname eq \"error.r\" ) {\n \t\tprint $debug \"[ERROR]\\n\";\n \t\t$debug->flush();\n@@ -178,6 +175,9 @@ while (1) {\n \t\t\tdie \"${command} write error\";\n \t\t}\n \n+\t\tprint $debug \"OUT: \" . length($output) . \" \";\n+\t\t$debug->flush();\n+\n \t\twhile ( length($output) > 0 ) {\n \t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n \t\t\tpacket_bin_write($packet);\n-- \n2.13.0\n\n"},{"id":"320416","messageId":"20170522135001.54506-5-larsxschneider@gmail.com","threadId":"46020","inReplyTo":"20170522135001.54506-1-larsxschneider@gmail.com","subject":"[PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-22T13:50:01Z","receivedAt":"2017-05-22T13:50:27Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Some `clean` / `smudge` filters might require a significant amount of\ntime to process a single blob (e.g. the Git LFS smudge filter might\nperform network requests). During this process the Git checkout\noperation is blocked and Git needs to wait until the filter is done to\ncontinue with the checkout.\n\nTeach the filter process protocol (introduced in edcc858) to accept the\nstatus \"delayed\" as response to a filter request. Upon this response Git\ncontinues with the checkout operation. After the checkout operation Git\ncalls \"finish_delayed_checkout\" which queries the filter for remaining\nblobs. If the filter is still working on the completion, then the filter\nis expected to block. If the filter has completed all remaining blobs\nthen an empty response is expected.\n\nGit has a multiple code paths that checkout a blob. Support delayed\ncheckouts only in `clone` (in unpack-trees.c) and `checkout` operations.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n Documentation/gitattributes.txt |  65 +++++++++++++-\n builtin/checkout.c              |   3 +\n cache.h                         |   6 +-\n convert.c                       | 131 +++++++++++++++++++++++-----\n convert.h                       |  21 +++++\n entry.c                         | 110 +++++++++++++++++++++--\n t/t0021-conversion.sh           |  74 ++++++++++++++++\n t/t0021/rot13-filter.pl         | 189 ++++++++++++++++++++++++++--------------\n unpack-trees.c                  |   2 +\n 9 files changed, 507 insertions(+), 94 deletions(-)\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 4736483865..ec92d3e3fa 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -425,8 +425,8 @@ packet:          git< capability=clean\n packet:          git< capability=smudge\n packet:          git< 0000\n ------------------------\n-Supported filter capabilities in version 2 are \"clean\" and\n-\"smudge\".\n+Supported filter capabilities in version 2 are \"clean\", \"smudge\",\n+and \"delay\".\n \n Afterwards Git sends a list of \"key=value\" pairs terminated with\n a flush packet. The list will contain at least the filter command\n@@ -512,12 +512,69 @@ the protocol then Git will stop the filter process and restart it\n with the next file that needs to be processed. Depending on the\n `filter.<driver>.required` flag Git will interpret that as error.\n \n-After the filter has processed a blob it is expected to wait for\n-the next \"key=value\" list containing a command. Git will close\n+After the filter has processed a command it is expected to wait for\n+a \"key=value\" list containing the next command. Git will close\n the command pipe on exit. The filter is expected to detect EOF\n and exit gracefully on its own. Git will wait until the filter\n process has stopped.\n \n+Delay\n+^^^^^\n+\n+If the filter supports the \"delay\" capability, then Git can send the\n+flag \"can-delay\" after the filter command and pathname. This flag\n+denotes that the filter can delay filtering the current blob (e.g. to\n+compensate network latencies) by responding with no content but with\n+the status \"delayed\" and a flush packet.\n+------------------------\n+packet:          git> command=smudge\n+packet:          git> pathname=path/testfile.dat\n+packet:          git> can-delay=1\n+packet:          git> 0000\n+packet:          git> CONTENT\n+packet:          git> 0000\n+packet:          git< status=delayed\n+packet:          git< 0000\n+------------------------\n+\n+If the filter supports the \"delay\" capability then it must support the\n+\"list_available_blobs\" command. If Git sends this command, then the\n+filter is expected to return a list of pathnames of blobs that are\n+available. The list must be terminated with a flush packet followed\n+by a \"success\" status that is also terminated with a flush packet. If\n+no blobs for the delayed paths are available, yet, then the filter is\n+expected to block the response until at least one blob becomes\n+available. The filter can tell Git that it has no more delayed blobs\n+by sending an empty list.\n+------------------------\n+packet:          git> command=list_available_blobs\n+packet:          git> 0000\n+packet:          git< pathname=path/testfile.dat\n+packet:          git< pathname=path/otherfile.dat\n+packet:          git< 0000\n+packet:          git< status=success\n+packet:          git< 0000\n+------------------------\n+\n+After Git received the pathnames, it will request the corresponding\n+blobs again. These requests contain a pathname and an empty content\n+section. The filter is expected to respond with the smudged content\n+in the usual way as explained above.\n+------------------------\n+packet:          git> command=smudge\n+packet:          git> pathname=path/testfile.dat\n+packet:          git> 0000\n+packet:          git> 0000  # empty content!\n+packet:          git< status=success\n+packet:          git< 0000\n+packet:          git< SMUDGED_CONTENT\n+packet:          git< 0000\n+packet:          git< 0000\n+------------------------\n+\n+Example\n+^^^^^^^\n+\n A long running filter demo implementation can be found in\n `contrib/long-running-filter/example.pl` located in the Git\n core repository. If you develop your own long running filter\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex bfa5419f33..dc5638a3bd 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -374,6 +374,8 @@ static int checkout_paths(const struct checkout_opts *opts,\n \tstate.force = 1;\n \tstate.refresh_cache = 1;\n \tstate.istate = &the_index;\n+\n+\tenable_delayed_checkout(&state);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n \t\tif (ce->ce_flags & CE_MATCHED) {\n@@ -388,6 +390,7 @@ static int checkout_paths(const struct checkout_opts *opts,\n \t\t\tpos = skip_same_name(ce, pos) - 1;\n \t\t}\n \t}\n+\terrs |= finish_delayed_checkout(&state);\n \n \tif (write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n \t\tdie(_(\"unable to write new index file\"));\ndiff --git a/cache.h b/cache.h\nindex 188811920c..2a5945a25c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1542,16 +1542,20 @@ extern int ident_cmp(const struct ident_split *, const struct ident_split *);\n struct checkout {\n \tstruct index_state *istate;\n \tconst char *base_dir;\n+\tstruct delayed_checkout *delayed_checkout;\n \tint base_dir_len;\n \tunsigned force:1,\n \t\t quiet:1,\n \t\t not_new:1,\n \t\t refresh_cache:1;\n };\n-#define CHECKOUT_INIT { NULL, \"\" }\n+#define CHECKOUT_INIT { NULL, \"\", NULL }\n+\n \n #define TEMPORARY_FILENAME_LENGTH 25\n extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);\n+extern void enable_delayed_checkout(struct checkout *state);\n+extern int finish_delayed_checkout(struct checkout *state);\n \n struct cache_def {\n \tstruct strbuf path;\ndiff --git a/convert.c b/convert.c\nindex 8d652bf27c..bc83933c62 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -495,6 +495,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le\n \n #define CAP_CLEAN    (1u<<0)\n #define CAP_SMUDGE   (1u<<1)\n+#define CAP_DELAY    (1u<<2)\n \n struct cmd2process {\n \tstruct hashmap_entry ent; /* must be the first member! */\n@@ -632,7 +633,8 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n \tif (err)\n \t\tgoto done;\n \n-\terr = packet_write_list(process->in, \"capability=clean\", \"capability=smudge\", NULL);\n+\terr = packet_write_list(process->in,\n+\t\t\"capability=clean\", \"capability=smudge\", \"capability=delay\", NULL);\n \n \tfor (;;) {\n \t\tcap_buf = packet_read_line(process->out, NULL);\n@@ -648,6 +650,8 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n \t\t\tentry->supported_capabilities |= CAP_CLEAN;\n \t\t} else if (!strcmp(cap_name, \"smudge\")) {\n \t\t\tentry->supported_capabilities |= CAP_SMUDGE;\n+\t\t} else if (!strcmp(cap_name, \"delay\")) {\n+\t\t\tentry->supported_capabilities |= CAP_DELAY;\n \t\t} else {\n \t\t\twarning(\n \t\t\t\t\"external filter '%s' requested unsupported filter capability '%s'\",\n@@ -673,9 +677,11 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n \n static int apply_multi_file_filter(const char *path, const char *src, size_t len,\n \t\t\t\t   int fd, struct strbuf *dst, const char *cmd,\n-\t\t\t\t   const unsigned int wanted_capability)\n+\t\t\t\t   const unsigned int wanted_capability,\n+\t\t\t\t   struct delayed_checkout *dco)\n {\n \tint err;\n+\tint can_delay = 0;\n \tstruct cmd2process *entry;\n \tstruct child_process *process;\n \tstruct strbuf nbuf = STRBUF_INIT;\n@@ -726,6 +732,14 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tif (err)\n \t\tgoto done;\n \n+\tif (CAP_DELAY & entry->supported_capabilities &&\n+\t    dco && dco->state == CE_CAN_DELAY) {\n+\t\tcan_delay = 1;\n+\t\terr = packet_write_fmt_gently(process->in, \"can-delay=1\\n\");\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t}\n+\n \terr = packet_flush_gently(process->in);\n \tif (err)\n \t\tgoto done;\n@@ -738,16 +752,22 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \t\tgoto done;\n \n \tread_multi_file_filter_status(process->out, &filter_status);\n-\terr = strcmp(filter_status.buf, \"success\");\n-\tif (err)\n-\t\tgoto done;\n-\n-\terr = read_packetized_to_strbuf(process->out, &nbuf) < 0;\n-\tif (err)\n-\t\tgoto done;\n+\tif (can_delay && !strcmp(filter_status.buf, \"delayed\")) {\n+\t\tdco->state = CE_DELAYED;\n+\t\tstring_list_insert(&dco->filters, cmd);\n+\t\tstring_list_insert(&dco->paths, path);\n+\t} else {\n+\t\t/* The filter got the blob and wants to send us a response. */\n+\t\terr = strcmp(filter_status.buf, \"success\");\n+\t\tif (err)\n+\t\t\tgoto done;\n \n-\tread_multi_file_filter_status(process->out, &filter_status);\n-\terr = strcmp(filter_status.buf, \"success\");\n+\t\terr = read_packetized_to_strbuf(process->out, &nbuf) < 0;\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tread_multi_file_filter_status(process->out, &filter_status);\n+\t\terr = strcmp(filter_status.buf, \"success\");\n+\t}\n \n done:\n \tsigchain_pop(SIGPIPE);\n@@ -777,6 +797,67 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \treturn !err;\n }\n \n+\n+int async_query_available_blobs(const char *cmd, struct string_list *delayed_paths)\n+{\n+\tint err;\n+\tchar *line;\n+\tstruct cmd2process *entry;\n+\tstruct child_process *process;\n+\tstruct strbuf filter_status = STRBUF_INIT;\n+\n+\tentry = find_multi_file_filter_entry(&cmd_process_map, cmd);\n+\tif (!entry) {\n+\t\terror(\"external filter '%s' is not available anymore although \"\n+\t\t      \"not all paths have been filtered\", cmd);\n+\t\treturn 0;\n+\t}\n+\tprocess = &entry->process;\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = packet_write_fmt_gently(\n+\t\tprocess->in, \"command=list_available_blobs\\n\");\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tfor (;;) {\n+\t\tconst char* pre = \"pathname=\";\n+\t\tconst int pre_len = strlen(pre);\n+\t\tline = packet_read_line(process->out, NULL);\n+\t\tif (!line)\n+\t\t\tbreak;\n+\t\terr = strlen(line) <= pre_len || strncmp(line, pre, pre_len);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstring_list_insert(delayed_paths, xstrdup(line+pre_len));\n+\t}\n+\n+\tread_multi_file_filter_status(process->out, &filter_status);\n+\terr = strcmp(filter_status.buf, \"success\");\n+\n+done:\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (err || errno == EPIPE) {\n+\t\tif (!strcmp(filter_status.buf, \"error\")) {\n+\t\t\t/* The filter signaled a problem with the file. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * Something went wrong with the protocol filter.\n+\t\t\t * Force shutdown and restart if another blob requires\n+\t\t\t * filtering.\n+\t\t\t */\n+\t\t\terror(\"external filter '%s' failed\", cmd);\n+\t\t\tkill_multi_file_filter(&cmd_process_map, entry);\n+\t\t}\n+\t}\n+\treturn !err;\n+}\n+\n static struct convert_driver {\n \tconst char *name;\n \tstruct convert_driver *next;\n@@ -788,7 +869,8 @@ static struct convert_driver {\n \n static int apply_filter(const char *path, const char *src, size_t len,\n \t\t\tint fd, struct strbuf *dst, struct convert_driver *drv,\n-\t\t\tconst unsigned int wanted_capability)\n+\t\t\tconst unsigned int wanted_capability,\n+\t\t\tstruct delayed_checkout *dco)\n {\n \tconst char *cmd = NULL;\n \n@@ -806,7 +888,8 @@ static int apply_filter(const char *path, const char *src, size_t len,\n \tif (cmd && *cmd)\n \t\treturn apply_single_file_filter(path, src, len, fd, dst, cmd);\n \telse if (drv->process && *drv->process)\n-\t\treturn apply_multi_file_filter(path, src, len, fd, dst, drv->process, wanted_capability);\n+\t\treturn apply_multi_file_filter(path, src, len, fd, dst,\n+\t\t\tdrv->process, wanted_capability, dco);\n \n \treturn 0;\n }\n@@ -1147,7 +1230,7 @@ int would_convert_to_git_filter_fd(const char *path)\n \tif (!ca.drv->required)\n \t\treturn 0;\n \n-\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN);\n+\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL);\n }\n \n const char *get_convert_attr_ascii(const char *path)\n@@ -1184,7 +1267,7 @@ int convert_to_git(const char *path, const char *src, size_t len,\n \n \tconvert_attrs(&ca, path);\n \n-\tret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN);\n+\tret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN, NULL);\n \tif (!ret && ca.drv && ca.drv->required)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n@@ -1209,7 +1292,7 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n \tassert(ca.drv);\n \tassert(ca.drv->clean || ca.drv->process);\n \n-\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN))\n+\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n@@ -1218,7 +1301,7 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n \n static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t\t\t\t    size_t len, struct strbuf *dst,\n-\t\t\t\t\t    int normalizing)\n+\t\t\t\t\t    int normalizing, struct delayed_checkout *dco)\n {\n \tint ret = 0, ret_filter = 0;\n \tstruct conv_attrs ca;\n@@ -1243,21 +1326,29 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n-\tret_filter = apply_filter(path, src, len, -1, dst, ca.drv, CAP_SMUDGE);\n+\tret_filter = apply_filter(\n+\t\tpath, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);\n \tif (!ret_filter && ca.drv && ca.drv->required)\n \t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n \n \treturn ret | ret_filter;\n }\n \n+int async_convert_to_working_tree(const char *path, const char *src,\n+\t\t\t\t  size_t len, struct strbuf *dst,\n+\t\t\t\t  void *dco)\n+{\n+\treturn convert_to_working_tree_internal(path, src, len, dst, 0, dco);\n+}\n+\n int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)\n {\n-\treturn convert_to_working_tree_internal(path, src, len, dst, 0);\n+\treturn convert_to_working_tree_internal(path, src, len, dst, 0, NULL);\n }\n \n int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst)\n {\n-\tint ret = convert_to_working_tree_internal(path, src, len, dst, 1);\n+\tint ret = convert_to_working_tree_internal(path, src, len, dst, 1, NULL);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\ndiff --git a/convert.h b/convert.h\nindex 82871a11d5..c4beaa5101 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -4,6 +4,8 @@\n #ifndef CONVERT_H\n #define CONVERT_H\n \n+#include \"string-list.h\"\n+\n enum safe_crlf {\n \tSAFE_CRLF_FALSE = 0,\n \tSAFE_CRLF_FAIL = 1,\n@@ -32,6 +34,21 @@ enum eol {\n #endif\n };\n \n+enum ce_delay_state {\n+\tCE_NO_DELAY = 0,\n+\tCE_CAN_DELAY = 1,\n+\tCE_DELAYED = 2,\n+\tCE_RETRY = 3\n+};\n+\n+struct delayed_checkout {\n+\tenum ce_delay_state state;\n+\t/* List of filter drivers that signaled delayed blobs. */\n+\tstruct string_list filters;\n+\t/* List of delayed blobs identified by their path. */\n+\tstruct string_list paths;\n+};\n+\n extern enum eol core_eol;\n extern const char *get_cached_convert_stats_ascii(const char *path);\n extern const char *get_wt_convert_stats_ascii(const char *path);\n@@ -42,6 +59,10 @@ extern int convert_to_git(const char *path, const char *src, size_t len,\n \t\t\t  struct strbuf *dst, enum safe_crlf checksafe);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t   size_t len, struct strbuf *dst);\n+extern int async_convert_to_working_tree(const char *path, const char *src,\n+\t\t\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t\t\t void *dco);\n+extern int async_query_available_blobs(const char *cmd, struct string_list *delayed_paths);\n extern int renormalize_buffer(const char *path, const char *src, size_t len,\n \t\t\t      struct strbuf *dst);\n static inline int would_convert_to_git(const char *path)\ndiff --git a/entry.c b/entry.c\nindex d2b512da90..73068117f4 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -137,6 +137,81 @@ static int streaming_write_entry(const struct cache_entry *ce, char *path,\n \treturn result;\n }\n \n+void enable_delayed_checkout(struct checkout *state)\n+{\n+\tif (!state->delayed_checkout) {\n+\t\tstate->delayed_checkout = xmalloc(sizeof(*state->delayed_checkout));\n+\t\tstate->delayed_checkout->state = CE_CAN_DELAY;\n+\t\tstring_list_init(&state->delayed_checkout->filters, 0);\n+\t\tstring_list_init(&state->delayed_checkout->paths, 0);\n+\t}\n+}\n+\n+static int remove_available_paths(struct string_list_item *item, void *cb_data)\n+{\n+\tstruct string_list *available_paths = cb_data;\n+\treturn !string_list_has_string(available_paths, item->string);\n+}\n+\n+int finish_delayed_checkout(struct checkout *state)\n+{\n+\tint errs = 0;\n+\tstruct string_list_item *filter, *path;\n+\tstruct delayed_checkout *dco = state->delayed_checkout;\n+\n+\tif (!state->delayed_checkout) {\n+\t\treturn errs;\n+\t}\n+\n+\twhile (dco->filters.nr > 0) {\n+\t\tfor_each_string_list_item(filter, &dco->filters) {\n+\t\t\tstruct string_list available_paths;\n+\t\t\tstring_list_init(&available_paths, 0);\n+\n+\t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n+\t\t\t\t/* Filter reported an error */\n+\t\t\t\terrs = 1;\n+\t\t\t\tfilter->string = \"\";\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (available_paths.nr <= 0) {\n+\t\t\t\t/* Filter responded with no entries. That means\n+\t\t\t\t   the filter is done and we can remove the\n+\t\t\t\t   filter from the list (see\n+\t\t\t\t   \"string_list_remove_empty_items\" call below).\n+\t\t\t\t*/\n+\t\t\t\tfilter->string = \"\";\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* In dco->paths we store a list of all delayed paths.\n+\t\t\t   The filter just send us a list of available paths.\n+\t\t\t   Remove them from the list.\n+\t\t\t*/\n+\t\t\tfilter_string_list(&dco->paths, 0,\n+\t\t\t\t&remove_available_paths, &available_paths);\n+\n+\t\t\tfor_each_string_list_item(path, &available_paths) {\n+\t\t\t\tstruct cache_entry* ce = index_file_exists(\n+\t\t\t\t\tstate->istate, path->string,\n+\t\t\t\t\tstrlen(path->string), 0);\n+\t\t\t\tdco->state = CE_RETRY;\n+\t\t\t\terrs |= (ce ? checkout_entry(ce, state, NULL) : 1);\n+\t\t\t}\n+\t\t}\n+\t\tstring_list_remove_empty_items(&dco->filters, 0);\n+\t}\n+\tstring_list_clear(&dco->filters, 0);\n+\n+\t/* At this point we should not have any delayed paths anymore. */\n+\terrs |= dco->paths.nr;\n+\n+\tfree(dco);\n+\tstate->delayed_checkout = NULL;\n+\n+\treturn errs;\n+}\n+\n static int write_entry(struct cache_entry *ce,\n \t\t       char *path, const struct checkout *state, int to_tempfile)\n {\n@@ -179,11 +254,36 @@ static int write_entry(struct cache_entry *ce,\n \t\t/*\n \t\t * Convert from git internal format to working tree format\n \t\t */\n-\t\tif (ce_mode_s_ifmt == S_IFREG &&\n-\t\t    convert_to_working_tree(ce->name, new, size, &buf)) {\n-\t\t\tfree(new);\n-\t\t\tnew = strbuf_detach(&buf, &newsize);\n-\t\t\tsize = newsize;\n+\t\tif (ce_mode_s_ifmt == S_IFREG) {\n+\t\t\tstruct delayed_checkout *dco = state->delayed_checkout;\n+\t\t\tif (dco && dco->state != CE_NO_DELAY) {\n+\t\t\t\t/* Do not send the blob in case of a retry. */\n+\t\t\t\tif (dco->state == CE_RETRY) {\n+\t\t\t\t\tnew = NULL;\n+\t\t\t\t\tsize = 0;\n+\t\t\t\t}\n+\t\t\t\tret = async_convert_to_working_tree(\n+\t\t\t\t\tce->name, new, size, &buf, dco);\n+\t\t\t\tif (ret && dco->state == CE_DELAYED) {\n+\t\t\t\t\tfree(new);\n+\t\t\t\t\t/* Reset the state of the next blob */\n+\t\t\t\t\tdco->state = CE_CAN_DELAY;\n+\t\t\t\t\tgoto finish;\n+\t\t\t\t}\n+\t\t\t} else\n+\t\t\t\tret = convert_to_working_tree(\n+\t\t\t\t\tce->name, new, size, &buf);\n+\n+\t\t\tif (ret) {\n+\t\t\t\tfree(new);\n+\t\t\t\tnew = strbuf_detach(&buf, &newsize);\n+\t\t\t\tsize = newsize;\n+\t\t\t}\n+\t\t\t/*\n+\t\t\t * No \"else\" here as errors from convert are OK at this\n+\t\t\t * point. If the error would have been fatal (e.g.\n+\t\t\t * filter is required), then we would have died already.\n+\t\t\t */\n \t\t}\n \n \t\tfd = open_output_fd(path, ce, to_tempfile);\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 0c04d346a1..4b5a45fd43 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -701,4 +701,78 @@ test_expect_success PERL 'invalid process filter must fail (and not hang!)' '\n \t)\n '\n \n+test_expect_success PERL 'delayed checkout in process filter' '\n+\ttest_config_global filter.a.process \"rot13-filter.pl a.log clean smudge delay\" &&\n+\ttest_config_global filter.a.required true &&\n+\ttest_config_global filter.b.process \"rot13-filter.pl b.log clean smudge delay\" &&\n+\ttest_config_global filter.b.required true &&\n+\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\t\techo \"*.a filter=a\" >.gitattributes &&\n+\t\techo \"*.b filter=b\" >>.gitattributes &&\n+\t\tcp \"$TEST_ROOT/test.o\" test.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" test-delay10.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" test-delay11.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" test-delay20.a &&\n+\t\tcp \"$TEST_ROOT/test.o\" test-delay10.b &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"test commit 1\"\n+\t) &&\n+\n+\tS=$(file_size \"$TEST_ROOT/test.o\") &&\n+\tcat >a.exp <<-EOF &&\n+\t\tSTART\n+\t\tinit handshake complete\n+\t\tIN: smudge test.a $S [OK] -- OUT: $S . [OK]\n+\t\tIN: smudge test-delay10.a $S [OK] -- [DELAYED]\n+\t\tIN: smudge test-delay11.a $S [OK] -- [DELAYED]\n+\t\tIN: smudge test-delay20.a $S [OK] -- [DELAYED]\n+\t\tIN: list_available_blobs test-delay10.a test-delay11.a [OK]\n+\t\tIN: smudge test-delay10.a 0 [OK] -- OUT: $S . [OK]\n+\t\tIN: smudge test-delay11.a 0 [OK] -- OUT: $S . [OK]\n+\t\tIN: list_available_blobs test-delay20.a [OK]\n+\t\tIN: smudge test-delay20.a 0 [OK] -- OUT: $S . [OK]\n+\t\tIN: list_available_blobs [OK]\n+\t\tSTOP\n+\tEOF\n+\tcat >b.exp <<-EOF &&\n+\t\tSTART\n+\t\tinit handshake complete\n+\t\tIN: smudge test-delay10.b $S [OK] -- [DELAYED]\n+\t\tIN: list_available_blobs test-delay10.b [OK]\n+\t\tIN: smudge test-delay10.b 0 [OK] -- OUT: $S . [OK]\n+\t\tIN: list_available_blobs [OK]\n+\t\tSTOP\n+\tEOF\n+\n+\trm -rf repo-cloned &&\n+\tfilter_git clone repo repo-cloned &&\n+\ttest_cmp_count a.exp repo-cloned/a.log &&\n+\ttest_cmp_count b.exp repo-cloned/b.log &&\n+\n+\t(\n+\t\tcd repo-cloned &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay10.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay11.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay20.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay10.b &&\n+\n+\t\trm *.a *.b &&\n+\t\tfilter_git checkout . &&\n+\t\ttest_cmp_count ../a.exp a.log &&\n+\t\ttest_cmp_count ../b.exp b.log &&\n+\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay10.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay11.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay20.a &&\n+\t\ttest_cmp_committed_rot13 \"$TEST_ROOT/test.o\" test-delay10.b\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 5e43faeec1..f0dc0aff4a 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -18,6 +18,11 @@\n #     operation then the filter signals that it cannot or does not want\n #     to process the file and any file after that is processed with the\n #     same command.\n+# (5) If data with a pathname that is a key in the DELAY hash is\n+#     requested (e.g. 'test-delay10.a') then the filter responds with\n+#     a \"delay\" status and sets the \"requested\" field in the DELAY hash.\n+#     The filter will signal the availability of this object after\n+#     \"count\" (field in DELAY hash) \"list_available_blobs\" commands.\n #\n \n use strict;\n@@ -30,6 +35,13 @@ my @capabilities            = @ARGV;\n \n open my $debug, \">>\", $log_file or die \"cannot open log file: $!\";\n \n+my %DELAY = (\n+\t'test-delay10.a' => { \"requested\" => 0, \"count\" => 1 },\n+\t'test-delay11.a' => { \"requested\" => 0, \"count\" => 1 },\n+\t'test-delay20.a' => { \"requested\" => 0, \"count\" => 2 },\n+\t'test-delay10.b' => { \"requested\" => 0, \"count\" => 1 },\n+);\n+\n sub rot13 {\n \tmy $str = shift;\n \t$str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n@@ -66,7 +78,7 @@ sub packet_bin_read {\n \n sub packet_txt_read {\n \tmy ( $res, $buf ) = packet_bin_read();\n-\tunless ( $buf =~ s/\\n$// ) {\n+\tunless ( $buf eq '' or $buf =~ s/\\n$// ) {\n \t\tdie \"A non-binary line MUST be terminated by an LF.\";\n \t}\n \treturn ( $res, $buf );\n@@ -101,6 +113,7 @@ packet_flush();\n \n ( packet_txt_read() eq ( 0, \"capability=clean\" ) )  || die \"bad capability\";\n ( packet_txt_read() eq ( 0, \"capability=smudge\" ) ) || die \"bad capability\";\n+( packet_txt_read() eq ( 0, \"capability=delay\" ) )  || die \"bad capability\";\n ( packet_bin_read() eq ( 1, \"\" ) )                  || die \"bad capability end\";\n \n foreach (@capabilities) {\n@@ -115,84 +128,132 @@ while (1) {\n \tprint $debug \"IN: $command\";\n \t$debug->flush();\n \n-\tmy ($pathname) = packet_txt_read() =~ /^pathname=(.+)$/;\n-\tprint $debug \" $pathname\";\n-\t$debug->flush();\n-\n-\tif ( $pathname eq \"\" ) {\n-\t\tdie \"bad pathname '$pathname'\";\n-\t}\n+\tif ( $command eq \"list_available_blobs\" ) {\n+\t\t# Flush\n+\t\tpacket_bin_read();\n \n-\t# Flush\n-\tpacket_bin_read();\n-\n-\tmy $input = \"\";\n-\t{\n-\t\tbinmode(STDIN);\n-\t\tmy $buffer;\n-\t\tmy $done = 0;\n-\t\twhile ( !$done ) {\n-\t\t\t( $done, $buffer ) = packet_bin_read();\n-\t\t\t$input .= $buffer;\n+\t\tforeach my $pathname (sort keys %DELAY) {\n+\t\t\tif ( $DELAY{$pathname}{\"requested\"} >= 1 ) {\n+\t\t\t\t$DELAY{$pathname}{\"count\"} = $DELAY{$pathname}{\"count\"} - 1;\n+\t\t\t\tif ($DELAY{$pathname}{\"count\"} == 0 ) {\n+\t\t\t\t\tprint $debug \" $pathname\";\n+\t\t\t\t\tpacket_txt_write(\"pathname=$pathname\");\n+\t\t\t\t}\n+\t\t\t}\n \t\t}\n-\t\tprint $debug \" \" . length($input) . \" [OK] -- \";\n-\t\t$debug->flush();\n-\t}\n-\n-\tmy $output;\n-\tif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n-\t\t$output = \"\";\n-\t}\n-\telsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n-\t\t$output = rot13($input);\n-\t}\n-\telsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n-\t\t$output = rot13($input);\n-\t}\n-\telse {\n-\t\tdie \"bad command '$command'\";\n-\t}\n \n-\tif ( $pathname eq \"error.r\" ) {\n-\t\tprint $debug \"[ERROR]\\n\";\n-\t\t$debug->flush();\n-\t\tpacket_txt_write(\"status=error\");\n \t\tpacket_flush();\n-\t}\n-\telsif ( $pathname eq \"abort.r\" ) {\n-\t\tprint $debug \"[ABORT]\\n\";\n+\n+\t\tprint $debug \" [OK]\\n\";\n \t\t$debug->flush();\n-\t\tpacket_txt_write(\"status=abort\");\n+\t\tpacket_txt_write(\"status=success\");\n \t\tpacket_flush();\n \t}\n \telse {\n-\t\tpacket_txt_write(\"status=success\");\n-\t\tpacket_flush();\n+\t\tmy ($pathname) = packet_txt_read() =~ /^pathname=(.+)$/;\n+\t\tprint $debug \" $pathname\";\n+\t\t$debug->flush();\n+\n+\t\tif ( $pathname eq \"\" ) {\n+\t\t\tdie \"bad pathname '$pathname'\";\n+\t\t}\n+\n+\t\t# Read until flush\n+\t\tmy ( $done, $buffer ) = packet_txt_read();\n+\t\twhile ( $buffer ne '' ) {\n+\t\t\tif ( $buffer eq \"can-delay=1\" ) {\n+\t\t\t\tif ( exists $DELAY{$pathname} and $DELAY{$pathname}{\"requested\"} == 0 ) {\n+\t\t\t\t\t$DELAY{$pathname}{\"requested\"} = 1;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tdie \"Unknown message '$buffer'\";\n+\t\t\t}\n \n-\t\tif ( $pathname eq \"${command}-write-fail.r\" ) {\n-\t\t\tprint $debug \"[WRITE FAIL]\\n\";\n+\t\t\t( $done, $buffer ) = packet_txt_read();\n+\t\t}\n+\n+\t\tmy $input = \"\";\n+\t\t{\n+\t\t\tbinmode(STDIN);\n+\t\t\tmy $buffer;\n+\t\t\tmy $done = 0;\n+\t\t\twhile ( !$done ) {\n+\t\t\t\t( $done, $buffer ) = packet_bin_read();\n+\t\t\t\t$input .= $buffer;\n+\t\t\t}\n+\t\t\tprint $debug \" \" . length($input) . \" [OK] -- \";\n \t\t\t$debug->flush();\n-\t\t\tdie \"${command} write error\";\n \t\t}\n \n-\t\tprint $debug \"OUT: \" . length($output) . \" \";\n-\t\t$debug->flush();\n+\t\tmy $output;\n+\t\tif ( exists $DELAY{$pathname} and exists $DELAY{$pathname}{\"output\"} ) {\n+\t\t\t$output = $DELAY{$pathname}{\"output\"}\n+\t\t}\n+\t\telsif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n+\t\t\t$output = \"\";\n+\t\t}\n+\t\telsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n+\t\t\t$output = rot13($input);\n+\t\t}\n+\t\telsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n+\t\t\t$output = rot13($input);\n+\t\t}\n+\t\telse {\n+\t\t\tdie \"bad command '$command'\";\n+\t\t}\n+\n+\t\tif ( $pathname eq \"error.r\" ) {\n+\t\t\tprint $debug \"[ERROR]\\n\";\n+\t\t\t$debug->flush();\n+\t\t\tpacket_txt_write(\"status=error\");\n+\t\t\tpacket_flush();\n+\t\t}\n+\t\telsif ( $pathname eq \"abort.r\" ) {\n+\t\t\tprint $debug \"[ABORT]\\n\";\n+\t\t\t$debug->flush();\n+\t\t\tpacket_txt_write(\"status=abort\");\n+\t\t\tpacket_flush();\n+\t\t}\n+\t\telsif ( $command eq \"smudge\" and\n+\t\t\texists $DELAY{$pathname} and\n+\t\t\t$DELAY{$pathname}{\"requested\"} == 1\n+\t\t) {\n+\t\t\tprint $debug \"[DELAYED]\\n\";\n+\t\t\t$debug->flush();\n+\t\t\tpacket_txt_write(\"status=delayed\");\n+\t\t\tpacket_flush();\n+\t\t\t$DELAY{$pathname}{\"requested\"} = 2;\n+\t\t\t$DELAY{$pathname}{\"output\"} = $output;\n+\t\t}\n+\t\telse {\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t\tpacket_flush();\n \n-\t\twhile ( length($output) > 0 ) {\n-\t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\tpacket_bin_write($packet);\n-\t\t\t# dots represent the number of packets\n-\t\t\tprint $debug \".\";\n-\t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n-\t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n+\t\t\tif ( $pathname eq \"${command}-write-fail.r\" ) {\n+\t\t\t\tprint $debug \"[WRITE FAIL]\\n\";\n+\t\t\t\t$debug->flush();\n+\t\t\t\tdie \"${command} write error\";\n \t\t\t}\n-\t\t\telse {\n-\t\t\t\t$output = \"\";\n+\n+\t\t\tprint $debug \"OUT: \" . length($output) . \" \";\n+\t\t\t$debug->flush();\n+\n+\t\t\twhile ( length($output) > 0 ) {\n+\t\t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n+\t\t\t\tpacket_bin_write($packet);\n+\t\t\t\t# dots represent the number of packets\n+\t\t\t\tprint $debug \".\";\n+\t\t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n+\t\t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n+\t\t\t\t}\n+\t\t\t\telse {\n+\t\t\t\t\t$output = \"\";\n+\t\t\t\t}\n \t\t\t}\n+\t\t\tpacket_flush();\n+\t\t\tprint $debug \" [OK]\\n\";\n+\t\t\t$debug->flush();\n+\t\t\tpacket_flush();\n \t\t}\n-\t\tpacket_flush();\n-\t\tprint $debug \" [OK]\\n\";\n-\t\t$debug->flush();\n-\t\tpacket_flush();\n \t}\n }\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex aa15111fef..009fcacb4b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -374,6 +374,7 @@ static int check_updates(struct unpack_trees_options *o)\n \tif (should_update_submodules() && o->update && !o->dry_run)\n \t\treload_gitmodules_file(index, &state);\n \n+\tenable_delayed_checkout(&state);\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -388,6 +389,7 @@ static int check_updates(struct unpack_trees_options *o)\n \t\t\t}\n \t\t}\n \t}\n+\terrs |= finish_delayed_checkout(&state);\n \tstop_progress(&progress);\n \tif (o->update)\n \t\tgit_attr_set_direction(GIT_ATTR_CHECKIN, NULL);\n-- \n2.13.0\n\n"},{"id":"320462","messageId":"20170522192749.GA31160@syl.local","threadId":"46020","inReplyTo":"20170522135001.54506-5-larsxschneider@gmail.com","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Taylor Blau","fromEmail":"ttaylorr@github.com","sentAt":"2017-05-22T19:30:43Z","receivedAt":"2017-05-22T19:30:52Z","isPatch":true,"sender":{"key":"ttaylorr@github.com","avatar":"https://gravatar.com/avatar/d5f3476f26b6f99cbb6b467e7ed7482f5762c8157bc73f569196e428bdcbea25?d=mp&s=160"},"body":"I have no remaining concerns about the protocol specification in terms of\nimplementing a filter with this capability.\n"},{"id":"320474","messageId":"0e2eb0e4-6efd-349e-21be-ae1133a6047b@web.de","threadId":"46020","inReplyTo":"20170522135001.54506-5-larsxschneider@gmail.com","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-05-22T19:52:21Z","receivedAt":"2017-05-22T19:52:57Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2017-05-22 15:50, Lars Schneider wrote:\n> +After Git received the pathnames, it will request the corresponding\n> +blobs again. These requests contain a pathname and an empty content\n> +section. The filter is expected to respond with the smudged content\n> +in the usual way as explained above.\n> +------------------------\n> +packet:          git> command=smudge\n> +packet:          git> pathname=path/testfile.dat\n> +packet:          git> 0000\n> +packet:          git> 0000  # empty content!\n> +packet:          git< status=success\n> +packet:          git< 0000\n> +packet:          git< SMUDGED_CONTENT\n> +packet:          git< 0000\n> +packet:          git< 0000\n> +------------------------\n\nThe documentation mentions \"0000\" 2 times.\nIs this a bug in the docu ? Or a feature which may need a comment ?\n\n"},{"id":"320476","messageId":"64b1fda4-9f79-1bd8-ad6d-43196b808d61@web.de","threadId":"46020","inReplyTo":"20170522135001.54506-5-larsxschneider@gmail.com","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-05-22T19:58:14Z","receivedAt":"2017-05-22T19:58:38Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2017-05-22 15:50, Lars Schneider wrote:\n> +\n> +int async_query_available_blobs(const char *cmd, struct string_list *delayed_paths)\n> +{\n> +\tint err;\n> +\tchar *line;\n> +\tstruct cmd2process *entry;\n> +\tstruct child_process *process;\n> +\tstruct strbuf filter_status = STRBUF_INIT;\n> +\n> +\tentry = find_multi_file_filter_entry(&cmd_process_map, cmd);\n> +\tif (!entry) {\n> +\t\terror(\"external filter '%s' is not available anymore although \"\n> +\t\t      \"not all paths have been filtered\", cmd);\n> +\t\treturn 0;\n> +\t}\n> +\tprocess = &entry->process;\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\n> +\terr = packet_write_fmt_gently(\n> +\t\tprocess->in, \"command=list_available_blobs\\n\");\n> +\tif (err)\n> +\t\tgoto done;\n> +\n> +\terr = packet_flush_gently(process->in);\n> +\tif (err)\n> +\t\tgoto done;\n> +\n> +\tfor (;;) {\n> +\t\tconst char* pre = \"pathname=\";\n> +\t\tconst int pre_len = strlen(pre);\n> +\t\tline = packet_read_line(process->out, NULL);\n> +\t\tif (!line)\n> +\t\t\tbreak;\n> +\t\terr = strlen(line) <= pre_len || strncmp(line, pre, pre_len);\n> +\t\tif (err)\n> +\t\t\tgoto done;\n> +\t\tstring_list_insert(delayed_paths, xstrdup(line+pre_len));\n> +\t}\n> +\n> +\tread_multi_file_filter_status(process->out, &filter_status);\n> +\terr = strcmp(filter_status.buf, \"success\");\n> +\n> +done:\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tif (err || errno == EPIPE) {\n\nThis looks strange, at first glance.\nDo we set errno to 0 before ?\nOr is there a trick that EPIPE can only be reached,\nif it is \"our\" error ?\n\n\n> +\t\tif (!strcmp(filter_status.buf, \"error\")) {\n> +\t\t\t/* The filter signaled a problem with the file. */\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * Something went wrong with the protocol filter.\n> +\t\t\t * Force shutdown and restart if another blob requires\n> +\t\t\t * filtering.\n> +\t\t\t */\n> +\t\t\terror(\"external filter '%s' failed\", cmd);\n> +\t\t\tkill_multi_file_filter(&cmd_process_map, entry);\n> +\t\t}\n> +\t}\n> +\treturn !err;\n> +}\n> +\n\n"},{"id":"320480","messageId":"E2C871B0-DCDF-40C5-A559-C396F3C8AA66@gmail.com","threadId":"46020","inReplyTo":"0e2eb0e4-6efd-349e-21be-ae1133a6047b@web.de","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-22T20:33:52Z","receivedAt":"2017-05-22T20:34:02Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 22 May 2017, at 21:52, Torsten Bögershausen <tboegi@web.de> wrote:\n> \n> On 2017-05-22 15:50, Lars Schneider wrote:\n>> +After Git received the pathnames, it will request the corresponding\n>> +blobs again. These requests contain a pathname and an empty content\n>> +section. The filter is expected to respond with the smudged content\n>> +in the usual way as explained above.\n>> +------------------------\n>> +packet:          git> command=smudge\n>> +packet:          git> pathname=path/testfile.dat\n>> +packet:          git> 0000\n>> +packet:          git> 0000  # empty content!\n>> +packet:          git< status=success\n>> +packet:          git< 0000\n>> +packet:          git< SMUDGED_CONTENT\n>> +packet:          git< 0000\n>> +packet:          git< 0000\n>> +------------------------\n> \n> The documentation mentions \"0000\" 2 times.\n> Is this a bug in the docu ? Or a feature which may need a comment ?\n\nThe first 0000 marks the end of the content and the second 0000\nmarks the end of an empty status list.\n\nExplained in the existing protocol here:\nhttps://github.com/git/git/blob/10c78a162fa821ee85203165b805ff46be454091/Documentation/gitattributes.txt#L451-L457\nhttps://github.com/git/git/blob/10c78a162fa821ee85203165b805ff46be454091/Documentation/gitattributes.txt#L464\n\nFor clarity I should probably change it to this:\n\n...\npacket:          git< status=success\npacket:          git< 0000\npacket:          git< SMUDGED_CONTENT\npacket:          git< 0000\npacket:          git< 0000  # empty list, keep \"status=success\" unchanged!\n\nThanks,\nLars"},{"id":"320481","messageId":"04ECEBE5-CAC8-4344-B07A-6B7A7C795C94@gmail.com","threadId":"46020","inReplyTo":"64b1fda4-9f79-1bd8-ad6d-43196b808d61@web.de","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-22T20:37:29Z","receivedAt":"2017-05-22T20:37:39Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 22 May 2017, at 21:58, Torsten Bögershausen <tboegi@web.de> wrote:\n> \n> On 2017-05-22 15:50, Lars Schneider wrote:\n>> +\n>> +int async_query_available_blobs(const char *cmd, struct string_list *delayed_paths)\n>> +{\n>> +\tint err;\n>> +\tchar *line;\n>> +\tstruct cmd2process *entry;\n>> +\tstruct child_process *process;\n>> +\tstruct strbuf filter_status = STRBUF_INIT;\n>> +\n>> +\tentry = find_multi_file_filter_entry(&cmd_process_map, cmd);\n>> +\tif (!entry) {\n>> +\t\terror(\"external filter '%s' is not available anymore although \"\n>> +\t\t      \"not all paths have been filtered\", cmd);\n>> +\t\treturn 0;\n>> +\t}\n>> +\tprocess = &entry->process;\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\n>> +\terr = packet_write_fmt_gently(\n>> +\t\tprocess->in, \"command=list_available_blobs\\n\");\n>> +\tif (err)\n>> +\t\tgoto done;\n>> +\n>> +\terr = packet_flush_gently(process->in);\n>> +\tif (err)\n>> +\t\tgoto done;\n>> +\n>> +\tfor (;;) {\n>> +\t\tconst char* pre = \"pathname=\";\n>> +\t\tconst int pre_len = strlen(pre);\n>> +\t\tline = packet_read_line(process->out, NULL);\n>> +\t\tif (!line)\n>> +\t\t\tbreak;\n>> +\t\terr = strlen(line) <= pre_len || strncmp(line, pre, pre_len);\n>> +\t\tif (err)\n>> +\t\t\tgoto done;\n>> +\t\tstring_list_insert(delayed_paths, xstrdup(line+pre_len));\n>> +\t}\n>> +\n>> +\tread_multi_file_filter_status(process->out, &filter_status);\n>> +\terr = strcmp(filter_status.buf, \"success\");\n>> +\n>> +done:\n>> +\tsigchain_pop(SIGPIPE);\n>> +\n>> +\tif (err || errno == EPIPE) {\n> \n> This looks strange, at first glance.\n> Do we set errno to 0 before ?\n> Or is there a trick that EPIPE can only be reached,\n> if it is \"our\" error ?\n\nYou are right and I'll fix it! \nThanks for reminding me! \nPeff also noticed that some time ago:\nhttp://public-inbox.org/git/20170411200520.oivytvlzkdu7bfh5@sigill.intra.peff.net/\n\nThanks,\nLars"},{"id":"320521","messageId":"xmqq1srgm9kq.fsf@gitster.mtv.corp.google.com","threadId":"46020","inReplyTo":"04ECEBE5-CAC8-4344-B07A-6B7A7C795C94@gmail.com","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-23T05:22:13Z","receivedAt":"2017-05-23T05:22:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider <larsxschneider@gmail.com> writes:\n\n>>> +\tsigchain_pop(SIGPIPE);\n>>> +\n>>> +\tif (err || errno == EPIPE) {\n>> \n>> This looks strange, at first glance.\n>> Do we set errno to 0 before ?\n>> Or is there a trick that EPIPE can only be reached,\n>> if it is \"our\" error ?\n>\n> You are right and I'll fix it! \n> Thanks for reminding me! \n> Peff also noticed that some time ago:\n> http://public-inbox.org/git/20170411200520.oivytvlzkdu7bfh5@sigill.intra.peff.net/\n\nBen Peart's bp/sub-process-convert-filter topic also had the same\nEPIPE issues in its earlier incarnation, IIRC.  I haven't looked at\nthis topic for some time, but I wonder if we can share code with it.\n\n"},{"id":"320534","messageId":"DB31E695-B7D5-4337-BACD-5191EB642F40@gmail.com","threadId":"46020","inReplyTo":"xmqq1srgm9kq.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-23T08:43:02Z","receivedAt":"2017-05-23T08:43:19Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 23 May 2017, at 07:22, Junio C Hamano <gitster@pobox.com> wrote:\n> \n> Lars Schneider <larsxschneider@gmail.com> writes:\n> \n>>>> +\tsigchain_pop(SIGPIPE);\n>>>> +\n>>>> +\tif (err || errno == EPIPE) {\n>>> \n>>> This looks strange, at first glance.\n>>> Do we set errno to 0 before ?\n>>> Or is there a trick that EPIPE can only be reached,\n>>> if it is \"our\" error ?\n>> \n>> You are right and I'll fix it! \n>> Thanks for reminding me! \n>> Peff also noticed that some time ago:\n>> http://public-inbox.org/git/20170411200520.oivytvlzkdu7bfh5@sigill.intra.peff.net/\n> \n> Ben Peart's bp/sub-process-convert-filter topic also had the same\n> EPIPE issues in its earlier incarnation, IIRC.  I haven't looked at\n> this topic for some time, but I wonder if we can share code with it.\n\nThat's right. There might be some code sharing opportunity with Ben's\ncode that is already in \"next\":\nhttps://github.com/git/git/blob/next/convert.c#L660-L677\n\nWould it be useful for you if I send v5 with the changes rebased \nonto \"next\"?\n\nThanks,\nLars\n\n\n"},{"id":"320945","messageId":"832A4D04-F32F-4D1F-A5CE-3514F8469E63@gmail.com","threadId":"46020","inReplyTo":"DB31E695-B7D5-4337-BACD-5191EB642F40@gmail.com","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-29T11:11:55Z","receivedAt":"2017-05-29T11:12:04Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 23 May 2017, at 10:43, Lars Schneider <larsxschneider@gmail.com> wrote:\n> \n> \n>> On 23 May 2017, at 07:22, Junio C Hamano <gitster@pobox.com> wrote:\n>> \n>> Lars Schneider <larsxschneider@gmail.com> writes:\n>> \n>>>>> +\tsigchain_pop(SIGPIPE);\n>>>>> +\n>>>>> +\tif (err || errno == EPIPE) {\n>>>> \n>>>> This looks strange, at first glance.\n>>>> Do we set errno to 0 before ?\n>>>> Or is there a trick that EPIPE can only be reached,\n>>>> if it is \"our\" error ?\n>>> \n>>> You are right and I'll fix it! \n>>> Thanks for reminding me! \n>>> Peff also noticed that some time ago:\n>>> http://public-inbox.org/git/20170411200520.oivytvlzkdu7bfh5@sigill.intra.peff.net/\n>> \n>> Ben Peart's bp/sub-process-convert-filter topic also had the same\n>> EPIPE issues in its earlier incarnation, IIRC.  I haven't looked at\n>> this topic for some time, but I wonder if we can share code with it.\n> \n> That's right. There might be some code sharing opportunity with Ben's\n> code that is already in \"next\":\n> https://github.com/git/git/blob/next/convert.c#L660-L677\n> \n> Would it be useful for you if I send v5 with the changes rebased \n> onto \"next\"?\n\nHi Junio,\n\nsorry for bugging you again, but Ben's topic did not make it to \"master\"\ntoday. Is it OK if I rebase my topic onto \"next\" and resend?\n\nThanks,\nLars"},{"id":"320980","messageId":"xmqqshjndul2.fsf@gitster.mtv.corp.google.com","threadId":"46020","inReplyTo":"832A4D04-F32F-4D1F-A5CE-3514F8469E63@gmail.com","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-30T01:03:37Z","receivedAt":"2017-05-30T01:03:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider <larsxschneider@gmail.com> writes:\n\n>> That's right. There might be some code sharing opportunity with Ben's\n>> code that is already in \"next\":\n>> https://github.com/git/git/blob/next/convert.c#L660-L677\n>> \n>> Would it be useful for you if I send v5 with the changes rebased \n>> onto \"next\"?\n>\n> Hi Junio,\n>\n> sorry for bugging you again, but Ben's topic did not make it to \"master\"\n> today. Is it OK if I rebase my topic onto \"next\" and resend?\n\nSorry, your earlier question was lost in the noise and I should have\npicked it up during my last sweep of leftover bits.\n\nIf you are depending on a single topic in 'next', it is better to\nbuild on the tip of that topic, not on 'next', if you can figure out\nwhere the tip is.  In practice, while we are exchanging patches via\ne-mail, there should be no noticeable difference either way [*1*],\nbut once you start throwing a complex and long series, you may want\nto publish it to a public repository for reviewers and the\nmaintainer to pull, and that workflow might give us an easier way to\nreview, but a topic based on 'next' will never have a chance to be\npulled to be merged for real, as merging its tip to 'master' means\nit will bring all other junk that may not ready.  So if you anticipate\nthat to happen someday, practicing to build on things that are only\nneeded (e.g. if you depend on two topics, you may start by merging\nthem on top of 'master' and then building your change on top) is a\ngood idea.\n\nNo matter what you do, please mention on top of what you built your\nwork.\n\nThanks.\n"},{"id":"320982","messageId":"xmqqd1ards5v.fsf@gitster.mtv.corp.google.com","threadId":"46020","inReplyTo":"xmqqshjndul2.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 4/4] convert: add \"status=delayed\" to filter process protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-30T01:55:56Z","receivedAt":"2017-05-30T01:56:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> If you are depending on a single topic in 'next', it is better to\n> build on the tip of that topic, not on 'next', if you can figure out\n> where the tip is.  In practice, while we are exchanging patches via\n> e-mail, there should be no noticeable difference either way [*1*],\n> ...\n\nAnd the forgotten foot note would have said something like this:\n\n *1* If you are depending only on a single topic in 'next', and\n     other topics in 'next' do not interfere with your work, then by\n     definition, your patches that apply cleanly on 'next' ought to\n     apply cleanly on the tip of that single topic.\n\n;-)\n"}]}