{"thread":{"id":"44122","subject":"[PATCH v8 00/11] Git filter protocol","startedAt":"2016-09-20T19:02:59Z","lastAt":"2016-10-06T21:27:45Z","messageCount":71,"participants":["larsxschneider@gmail.com","Jakub Narębski","Lars Schneider","Jeff King","Junio C Hamano","Torsten Bögershausen","Johannes Sixt"],"isPatch":true,"patchVersion":8,"patchTotal":11},"messages":[{"id":"302232","messageId":"20160920190247.82189-1-larsxschneider@gmail.com","threadId":"44122","inReplyTo":null,"subject":"[PATCH v8 00/11] Git filter protocol","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:36Z","receivedAt":"2016-09-20T19:02:59Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nThe goal of this series is to avoid launching a new clean/smudge filter\nprocess for each file that is filtered.\n\nA short summary about v1 to v5 can be found here:\nhttps://git.github.io/rev_news/2016/08/17/edition-18/\n\nThis series is also published on web:\nhttps://github.com/larsxschneider/git/pull/12\n\nThanks a lot to\n Stefan, Torsten, Junio, Jeff, and Ramsay\nfor very helpful reviews,\nLars\n\n\n\n## Major changes since v7\n\n* explicitly define all packets as text packets terminated by an LF (except CONTENT and flush)\n* move check_pipe() from write_or_die to run_command and reuse it\n\n\n\n## All changes since v7\n\n### Stefan\n\n* http://public-inbox.org/git/CAGZ79kY0GaWuuh_MzKL6FZ7KWF2Kwhfh9qnEYd-qX8VDQWNmCQ@mail.gmail.com/\n * move check_pipe() from write_or_die to run_command and reuse it\n * use error() (== -1) as return value\n\n* http://public-inbox.org/git/CAGZ79kZdroDdD5SHP+-9svSTYbJfn2vsFXAwC4aen3hMVEOOPA@mail.gmail.com/\n * remove verbose return value explanation in commit messages\n * on \"packet_flush_gently\" introduction, mention that the function is used later\n\n\n### Torsten\n\n* http://public-inbox.org/git/20160910164056.GA14646@tb-raspi/\n * remove unnecessary parenthesis\n\n\n* http://public-inbox.org/git/20160910062919.GB11001@tb-raspi/\n * explicitly define all packets as text packets terminated by an LF (except CONTENT and flush)\n\n\n\n### Junio\n\n* http://public-inbox.org/git/xmqq8tuvx1sz.fsf@gitster.mtv.corp.google.com/\n * fix SP in Perl script\n * use `unsigned int` for CAP_CLEAN and CAP_SMUDGE\n * fix pointer notation\n * remove invalid \"convert.h\" include\n\n\n### Ramsay\n\n* http://public-inbox.org/git/6373d68b-574d-59f3-7b8d-60dd3a673806@ramsayjones.plus.com\n * declare packet_write_gently() static\n\n\n### Lars\n* add SP in paths for test case\n* fix \"{\" code formatting\n\n\n\n## Interdiff (v7..v8)\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex ac000ea..946dcad 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -385,9 +385,11 @@ Long Running Filter Process\n If the filter command (a string value) is defined via\n `filter..process` then Git can process all blobs with a\n single filter invocation for the entire life of a single Git\n-command. This is achieved by using the following packet format\n-(pkt-line, see technical/protocol-common.txt) based protocol over\n-standard input and standard output.\n+command. This is achieved by using a packet format (pkt-line,\n+see technical/protocol-common.txt) based protocol over standard\n+input and standard output as follows. All packets are considered\n+text and therefore are terminated by an LF. Exceptions are the\n+\"*CONTENT\" packets and the flush packet.\n\n Git starts the filter when it encounters the first file\n that needs to be cleaned or smudged. After the filter started\n@@ -430,8 +432,8 @@ to filter relative to the repository root. Right after these packets\n Git sends the content split in zero or more pkt-line packets and a\n flush packet to terminate content.\n ------------------------\n-packet: git> command=smudge\\n\n-packet: git> pathname=path/testfile.dat\\n\n+packet: git> command=smudge\n+packet: git> pathname=path/testfile.dat\n packet: git> 0000\n packet: git> CONTENT\n packet: git> 0000\n@@ -445,7 +447,7 @@ or more pkt-line packets and a flush packet at the end. Finally, a\n second list of \"key=value\" pairs terminated with a flush packet\n is expected. The filter can change the status in the second list.\n ------------------------\n-packet: git< status=success\\n\n+packet: git< status=success\n packet: git< 0000\n packet: git< SMUDGED_CONTENT\n packet: git< 0000\n@@ -455,7 +457,7 @@ packet: git< 0000 # empty list!\n If the result content is empty then the filter is expected to respond\n with a success status and an empty list.\n ------------------------\n-packet: git< status=success\\n\n+packet: git< status=success\n packet: git< 0000\n packet: git< 0000 # empty content!\n packet: git< 0000 # empty list!\n@@ -466,7 +468,7 @@ it is expected to respond with an \"error\" status. Depending on the\n `filter..required` flag Git will interpret that as error\n but it will not stop or restart the filter process.\n ------------------------\n-packet: git< status=error\\n\n+packet: git< status=error\n packet: git< 0000\n ------------------------\n\n@@ -476,11 +478,11 @@ completely) sent. Depending on the `filter..required` flag\n Git will interpret that as error but it will not stop or restart the\n filter process.\n ------------------------\n-packet: git< status=success\\n\n+packet: git< status=success\n packet: git< 0000\n packet: git< HALF_WRITTEN_ERRONEOUS_CONTENT\n packet: git< 0000\n-packet: git< status=error\\n\n+packet: git< status=error\n packet: git< 0000\n ------------------------\n\n@@ -500,7 +502,7 @@ the `filter..required` flag Git will interpret that as error\n for the content as well as any future content for the lifetime of the\n Git process but it will not stop or restart the filter process.\n ------------------------\n-packet: git< status=abort\\n\n+packet: git< status=abort\n packet: git< 0000\n ------------------------\n\n@@ -510,8 +512,8 @@ the command pipe on exit. The filter is expected to detect EOF\n and exit gracefully on its own.\n\n A long running filter demo implementation can be found in\n-`contrib/long-running-filter/example.pl` located in the Git\n-core repository. If you develop your own long running filter\n+`contrib/long-running-filter/example.pl` located in the Git\n+core repository. If you develop your own long running filter\n process then the `GIT_TRACE_PACKET` environment variables can be\n very helpful for debugging (see linkgit:git[1]).\n\ndiff --git a/contrib/long-running-filter/example.pl b/contrib/long-running-filter/example.pl\nindex 279fbfb..c13a631 100755\n--- a/contrib/long-running-filter/example.pl\n+++ b/contrib/long-running-filter/example.pl\n@@ -9,7 +9,7 @@ use warnings;\n\n my $MAX_PACKET_CONTENT_SIZE = 65516;\n\n-sub packet_read {\n+sub packet_bin_read {\n my $buffer;\n my $bytes_read = read STDIN, $buffer, 4;\n if ( $bytes_read == 0 ) {\n@@ -37,38 +37,50 @@ sub packet_read {\n }\n }\n\n-sub packet_write {\n+sub packet_txt_read {\n+ my ( $res, $buf ) = packet_bin_read();\n+ unless ( $buf =~ /\\n$/ ) {\n+ die \"A non-binary line SHOULD BE terminated by an LF.\";\n+ }\n+ return ( $res, substr( $buf, 0, -1 ) );\n+}\n+\n+sub packet_bin_write {\n my ($packet) = @_;\n print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n print STDOUT $packet;\n STDOUT->flush();\n }\n\n+sub packet_txt_write {\n+ packet_bin_write( $_[0] . \"\\n\" );\n+}\n+\n sub packet_flush {\n print STDOUT sprintf( \"%04x\", 0 );\n STDOUT->flush();\n }\n\n-( packet_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialization\";\n-( packet_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n-( packet_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n+( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n+( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n+( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n\n-packet_write(\"git-filter-server\\n\");\n-packet_write(\"version=2\\n\");\n+packet_txt_write(\"git-filter-server\");\n+packet_txt_write(\"version=2\");\n\n-( packet_read() eq ( 0, \"clean=true\" ) ) || die \"bad capability\";\n-( packet_read() eq ( 0, \"smudge=true\" ) ) || die \"bad capability\";\n-( packet_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n+( packet_txt_read() eq ( 0, \"clean=true\" ) ) || die \"bad capability\";\n+( packet_txt_read() eq ( 0, \"smudge=true\" ) ) || die \"bad capability\";\n+( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n\n-packet_write( \"clean=true\\n\" );\n-packet_write( \"smudge=true\\n\" );\n+packet_txt_write(\"clean=true\");\n+packet_txt_write(\"smudge=true\");\n packet_flush();\n\n while (1) {\n- my ($command) = packet_read() =~ /^command=([^=]+)\\n$/;\n- my ($pathname) = packet_read() =~ /^pathname=([^=]+)\\n$/;\n+ my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n+ my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;\n\n- packet_read();\n+ packet_bin_read();\n\n my $input = \"\";\n {\n@@ -76,7 +88,7 @@ while (1) {\n my $buffer;\n my $done = 0;\n while ( !$done ) {\n- ( $done, $buffer ) = packet_read();\n+ ( $done, $buffer ) = packet_bin_read();\n $input .= $buffer;\n }\n }\n@@ -94,11 +106,11 @@ while (1) {\n die \"bad command '$command'\";\n }\n\n- packet_write(\"status=success\\n\");\n+ packet_txt_write(\"status=success\");\n packet_flush();\n while ( length($output) > 0 ) {\n my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n- packet_write($packet);\n+ packet_bin_write($packet);\n if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n }\n@@ -106,6 +118,6 @@ while (1) {\n $output = \"\";\n }\n }\n- packet_flush(); # flush content!\n- packet_flush(); # empty list!\n+ packet_flush(); # flush content!\n+ packet_flush(); # empty list!\n }\ndiff --git a/convert.c b/convert.c\nindex 0ed48ed..bd66257 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -472,16 +472,13 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le\n return 0; /* error was already reported */\n\n if (strbuf_read(&nbuf, async.out, len) < 0) {\n- error(\"read from external filter '%s' failed\", cmd);\n- err = -1;\n+ err = error(\"read from external filter '%s' failed\", cmd);\n }\n if (close(async.out)) {\n- error(\"read from external filter '%s' failed\", cmd);\n- err = -1;\n+ err = error(\"read from external filter '%s' failed\", cmd);\n }\n if (finish_async(&async)) {\n- error(\"external filter '%s' failed\", cmd);\n- err = -1;\n+ err = error(\"external filter '%s' failed\", cmd);\n }\n\n if (!err) {\n@@ -496,7 +493,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le\n\n struct cmd2process {\n struct hashmap_entry ent; /* must be the first member! */\n- int supported_capabilities;\n+ unsigned int supported_capabilities;\n const char *cmd;\n struct child_process process;\n };\n@@ -541,13 +538,12 @@ static int packet_write_list(int fd, const char *line, ...)\n va_list args;\n int err;\n va_start(args, line);\n- for (;;)\n- {\n+ for (;;) {\n if (!line)\n break;\n if (strlen(line) > PKTLINE_DATA_MAXLEN)\n return -1;\n- err = packet_write_fmt_gently(fd, \"%s\", line);\n+ err = packet_write_fmt_gently(fd, \"%s\\n\", line);\n if (err)\n return err;\n line = va_arg(args, const char*);\n@@ -601,8 +597,7 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n\n err = packet_write_list(process->in, \"clean=true\", \"smudge=true\", NULL);\n\n- for (;;)\n- {\n+ for (;;) {\n cap_buf = packet_read_line(process->out, NULL);\n if (!cap_buf)\n break;\n@@ -658,7 +653,7 @@ static void read_multi_file_filter_values(int fd, struct strbuf *status) {\n\n static int apply_multi_file_filter(const char *path, const char *src, size_t len,\n int fd, struct strbuf *dst, const char *cmd,\n- const int wanted_capability)\n+ const unsigned int wanted_capability)\n {\n int err;\n struct cmd2process *entry;\n@@ -703,17 +698,18 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n\n sigchain_push(SIGPIPE, SIG_IGN);\n\n-\n- err = (strlen(filter_type) > PKTLINE_DATA_MAXLEN);\n+ err = strlen(filter_type) > PKTLINE_DATA_MAXLEN;\n if (err)\n goto done;\n+\n err = packet_write_fmt_gently(process->in, \"command=%s\\n\", filter_type);\n if (err)\n goto done;\n\n- err = (strlen(path) > PKTLINE_DATA_MAXLEN);\n+ err = strlen(path) > PKTLINE_DATA_MAXLEN;\n if (err)\n goto done;\n+\n err = packet_write_fmt_gently(process->in, \"pathname=%s\\n\", path);\n if (err)\n goto done;\n@@ -780,9 +776,9 @@ static struct convert_driver {\n\n static int apply_filter(const char *path, const char *src, size_t len,\n int fd, struct strbuf *dst, struct convert_driver *drv,\n- const int wanted_capability)\n+ const unsigned int wanted_capability)\n {\n- const char* cmd = NULL;\n+ const char *cmd = NULL;\n\n if (!drv)\n return 0;\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 5001a07..a0a8543 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -96,8 +96,7 @@ int packet_flush_gently(int fd)\n packet_trace(\"0000\", 4, 1);\n if (write_in_full(fd, \"0000\", 4) == 4)\n return 0;\n- error(\"flush packet write failed\");\n- return -1;\n+ return error(\"flush packet write failed\");\n }\n\n void packet_buf_flush(struct strbuf *buf)\n@@ -146,19 +145,10 @@ static int packet_write_fmt_1(int fd, int gently,\n return 0;\n\n if (!gently) {\n- if (errno == EPIPE) {\n- if (in_async())\n- async_exit(141);\n-\n- signal(SIGPIPE, SIG_DFL);\n- raise(SIGPIPE);\n- /* Should never happen, but just in case... */\n- exit(141);\n- }\n- die_errno(\"packet write error\");\n+ check_pipe(errno);\n+ die_errno(\"packet write with format failed\");\n }\n- error(\"packet write failed\");\n- return -1;\n+ return error(\"packet write with format failed\");\n }\n\n void packet_write_fmt(int fd, const char *fmt, ...)\n@@ -181,13 +171,12 @@ int packet_write_fmt_gently(int fd, const char *fmt, ...)\n return status;\n }\n\n-int packet_write_gently(const int fd_out, const char *buf, size_t size)\n+static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n {\n static char packet_write_buffer[LARGE_PACKET_MAX];\n\n if (size > sizeof(packet_write_buffer) - 4) {\n- error(\"packet write failed\");\n- return -1;\n+ return error(\"packet write failed - data exceeds max packet size\");\n }\n packet_trace(buf, size, 1);\n size += 4;\n@@ -195,9 +184,7 @@ int packet_write_gently(const int fd_out, const char *buf, size_t size)\n memcpy(packet_write_buffer + 4, buf, size - 4);\n if (write_in_full(fd_out, packet_write_buffer, size) == size)\n return 0;\n-\n- error(\"packet write failed\");\n- return -1;\n+ return error(\"packet write failed\");\n }\n\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\ndiff --git a/run-command.c b/run-command.c\nindex 5a4dbb6..b72f6d1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -6,6 +6,19 @@\n #include \"thread-utils.h\"\n #include \"strbuf.h\"\n\n+void check_pipe(int err)\n+{\n+ if (err == EPIPE) {\n+ if (in_async())\n+ async_exit(141);\n+\n+ signal(SIGPIPE, SIG_DFL);\n+ raise(SIGPIPE);\n+ /* Should never happen, but just in case... */\n+ exit(141);\n+ }\n+}\n+\n void child_process_init(struct child_process *child)\n {\n memset(child, 0, sizeof(*child));\ndiff --git a/run-command.h b/run-command.h\nindex 5066649..e7c5f71 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -54,6 +54,8 @@ int finish_command(struct child_process *);\n int finish_command_in_signal(struct child_process *);\n int run_command(struct child_process *);\n\n+void check_pipe(int err);\n+\n /*\n * Returns the path to the hook file, or NULL if the hook is missing\n * or disabled. Note that this points to static storage that will be\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 1c98ac3..210c4f6 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -34,7 +34,7 @@ test_expect_success setup '\n git checkout -- test test.t test.i &&\n\n echo \"content-test2\" >test2.o &&\n- echo \"content-test3-subdir\" >test3-subdir.o\n+ echo \"content-test3 - subdir\" >\"test3 - subdir.o\"\n '\n\n script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n@@ -317,9 +317,9 @@ check_filter_no_call () {\n }\n\n check_rot13 () {\n- test_cmp $1 $2 &&\n- ./../rot13.sh <$1 >expected &&\n- git cat-file blob :$2 >actual &&\n+ test_cmp \"$1\" \"$2\" &&\n+ ./../rot13.sh <\"$1\" >expected &&\n+ git cat-file blob :\"$2\" >actual &&\n test_cmp expected actual\n }\n\n@@ -340,7 +340,7 @@ test_expect_success PERL 'required process filter should filter data' '\n cp ../test.o test.r &&\n cp ../test2.o test2.r &&\n mkdir testsubdir &&\n- cp ../test3-subdir.o testsubdir/test3-subdir.r &&\n+ cp \"../test3 - subdir.o\" \"testsubdir/test3 - subdir.r\" &&\n >test4-empty.r &&\n\n check_filter \\\n@@ -349,7 +349,7 @@ test_expect_success PERL 'required process filter should filter data' '\n 1 IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n 1 IN: clean test2.r 14 [OK] -- OUT: 14 . [OK]\n 1 IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]\n- 1 IN: clean testsubdir/test3-subdir.r 21 [OK] -- OUT: 21 . [OK]\n+ 1 IN: clean testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n 1 START\n 1 STOP\n 1 wrote filter header\n@@ -361,13 +361,13 @@ test_expect_success PERL 'required process filter should filter data' '\n x IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n x IN: clean test2.r 14 [OK] -- OUT: 14 . [OK]\n x IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]\n- x IN: clean testsubdir/test3-subdir.r 21 [OK] -- OUT: 21 . [OK]\n+ x IN: clean testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n 1 START\n 1 STOP\n 1 wrote filter header\n EOF\n\n- rm -f test?.r testsubdir/test3-subdir.r &&\n+ rm -f test?.r \"testsubdir/test3 - subdir.r\" &&\n\n check_filter_ignore_clean \\\n git checkout . \\\n@@ -375,7 +375,7 @@ test_expect_success PERL 'required process filter should filter data' '\n START\n wrote filter header\n IN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n- IN: smudge testsubdir/test3-subdir.r 21 [OK] -- OUT: 21 . [OK]\n+ IN: smudge testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n STOP\n EOF\n\n@@ -395,13 +395,13 @@ test_expect_success PERL 'required process filter should filter data' '\n IN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n IN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n IN: smudge test4-empty.r 0 [OK] -- OUT: 0 [OK]\n- IN: smudge testsubdir/test3-subdir.r 21 [OK] -- OUT: 21 . [OK]\n+ IN: smudge testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n STOP\n EOF\n\n check_rot13 ../test.o test.r &&\n check_rot13 ../test2.o test2.r &&\n- check_rot13 ../test3-subdir.o testsubdir/test3-subdir.r\n+ check_rot13 \"../test3 - subdir.o\" \"testsubdir/test3 - subdir.r\"\n )\n '\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 8e27877..8958f71 100755\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -34,7 +34,7 @@ sub rot13 {\n return $str;\n }\n\n-sub packet_read {\n+sub packet_bin_read {\n my $buffer;\n my $bytes_read = read STDIN, $buffer, 4;\n if ( $bytes_read == 0 ) {\n@@ -63,13 +63,25 @@ sub packet_read {\n }\n }\n\n-sub packet_write {\n+sub packet_txt_read {\n+ my ( $res, $buf ) = packet_bin_read();\n+ unless ( $buf =~ /\\n$/ ) {\n+ die \"A non-binary line SHOULD BE terminated by an LF.\";\n+ }\n+ return ( $res, substr( $buf, 0, -1 ) );\n+}\n+\n+sub packet_bin_write {\n my ($packet) = @_;\n print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n print STDOUT $packet;\n STDOUT->flush();\n }\n\n+sub packet_txt_write {\n+ packet_bin_write( $_[0] . \"\\n\" );\n+}\n+\n sub packet_flush {\n print STDOUT sprintf( \"%04x\", 0 );\n STDOUT->flush();\n@@ -78,35 +90,35 @@ sub packet_flush {\n print $debug \"START\\n\";\n $debug->flush();\n\n-( packet_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialization\";\n-( packet_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n-( packet_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n+( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n+( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n+( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n\n-packet_write(\"git-filter-server\\n\");\n-packet_write(\"version=2\\n\");\n+packet_txt_write(\"git-filter-server\");\n+packet_txt_write(\"version=2\");\n\n-( packet_read() eq ( 0, \"clean=true\" ) ) || die \"bad capability\";\n-( packet_read() eq ( 0, \"smudge=true\" ) ) || die \"bad capability\";\n-( packet_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n+( packet_txt_read() eq ( 0, \"clean=true\" ) ) || die \"bad capability\";\n+( packet_txt_read() eq ( 0, \"smudge=true\" ) ) || die \"bad capability\";\n+( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n\n foreach (@capabilities) {\n- packet_write( $_ . \"=true\\n\" );\n+ packet_txt_write( $_ . \"=true\" );\n }\n packet_flush();\n print $debug \"wrote filter header\\n\";\n $debug->flush();\n\n while (1) {\n- my ($command) = packet_read() =~ /^command=([^=]+)\\n$/;\n+ my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n print $debug \"IN: $command\";\n $debug->flush();\n\n- my ($pathname) = packet_read() =~ /^pathname=([^=]+)\\n$/;\n+ my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;\n print $debug \" $pathname\";\n $debug->flush();\n\n # Flush\n- packet_read();\n+ packet_bin_read();\n\n my $input = \"\";\n {\n@@ -114,7 +126,7 @@ while (1) {\n my $buffer;\n my $done = 0;\n while ( !$done ) {\n- ( $done, $buffer ) = packet_read();\n+ ( $done, $buffer ) = packet_bin_read();\n $input .= $buffer;\n }\n print $debug \" \" . length($input) . \" [OK] -- \";\n@@ -141,17 +153,17 @@ while (1) {\n if ( $pathname eq \"error.r\" ) {\n print $debug \"[ERROR]\\n\";\n $debug->flush();\n- packet_write(\"status=error\\n\");\n+ packet_txt_write(\"status=error\");\n packet_flush();\n }\n elsif ( $pathname eq \"abort.r\" ) {\n print $debug \"[ABORT]\\n\";\n $debug->flush();\n- packet_write(\"status=abort\\n\");\n+ packet_txt_write(\"status=abort\");\n packet_flush();\n }\n else {\n- packet_write(\"status=success\\n\");\n+ packet_txt_write(\"status=success\");\n packet_flush();\n\n if ( $pathname eq \"${command}-write-fail.r\" ) {\n@@ -162,7 +174,7 @@ while (1) {\n\n while ( length($output) > 0 ) {\n my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n- packet_write($packet);\n+ packet_bin_write($packet);\n print $debug \".\";\n if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f6798f8..11c37fb 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -10,7 +10,6 @@\n #include \"attr.h\"\n #include \"split-index.h\"\n #include \"dir.h\"\n-#include \"convert.h\"\n\n /*\n * Error messages expected by scripts out of plumbing commands such as\ndiff --git a/write_or_die.c b/write_or_die.c\nindex 0734432..eab8c8d 100644\n--- a/write_or_die.c\n+++ b/write_or_die.c\n@@ -1,19 +1,6 @@\n #include \"cache.h\"\n #include \"run-command.h\"\n\n-static void check_pipe(int err)\n-{\n- if (err == EPIPE) {\n- if (in_async())\n- async_exit(141);\n-\n- signal(SIGPIPE, SIG_DFL);\n- raise(SIGPIPE);\n- /* Should never happen, but just in case... */\n- exit(141);\n- }\n-}\n-\n /*\n * Some cases use stdio, but want to flush after the write\n * to get error handling (and to get better interactive\n\n\n\nLars Schneider (11):\n pkt-line: rename packet_write() to packet_write_fmt()\n pkt-line: extract set_packet_header()\n run-command: move check_pipe() from write_or_die to run_command\n pkt-line: add packet_write_fmt_gently()\n pkt-line: add packet_flush_gently()\n pkt-line: add packet_write_gently()\n pkt-line: add functions to read/write flush terminated packet streams\n convert: quote filter names in error messages\n convert: modernize tests\n convert: make apply_filter() adhere to standard Git error handling\n convert: add filter..process option\n\n Documentation/gitattributes.txt | 156 +++++++++++-\n builtin/archive.c | 4 +-\n builtin/receive-pack.c | 4 +-\n builtin/remote-ext.c | 4 +-\n builtin/upload-archive.c | 4 +-\n connect.c | 2 +-\n contrib/long-running-filter/example.pl | 123 ++++++++++\n convert.c | 369 ++++++++++++++++++++++++----\n daemon.c | 2 +-\n http-backend.c | 2 +-\n pkt-line.c | 147 +++++++++++-\n pkt-line.h | 12 +-\n run-command.c | 13 +\n run-command.h | 2 +\n shallow.c | 2 +-\n t/t0021-conversion.sh | 423 ++++++++++++++++++++++++++++++---\n t/t0021/rot13-filter.pl | 191 +++++++++++++++\n upload-pack.c | 30 +--\n write_or_die.c | 13 -\n 19 files changed, 1379 insertions(+), 124 deletions(-)\n create mode 100755 contrib/long-running-filter/example.pl\n create mode 100755 t/t0021/rot13-filter.pl\n\n--\n2.10.0\n\n"},{"id":"302233","messageId":"20160920190247.82189-2-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 01/11] pkt-line: rename packet_write() to packet_write_fmt()","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:37Z","receivedAt":"2016-09-20T19:03:03Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\npacket_write() should be called packet_write_fmt() as the string\nparameter can be formatted.\n\nSuggested-by: Junio C Hamano \nSigned-off-by: Lars Schneider \n---\n builtin/archive.c | 4 ++--\n builtin/receive-pack.c | 4 ++--\n builtin/remote-ext.c | 4 ++--\n builtin/upload-archive.c | 4 ++--\n connect.c | 2 +-\n daemon.c | 2 +-\n http-backend.c | 2 +-\n pkt-line.c | 2 +-\n pkt-line.h | 2 +-\n shallow.c | 2 +-\n upload-pack.c | 30 +++++++++++++++---------------\n 11 files changed, 29 insertions(+), 29 deletions(-)\n\ndiff --git a/builtin/archive.c b/builtin/archive.c\nindex a1e3b94..49f4914 100644\n--- a/builtin/archive.c\n+++ b/builtin/archive.c\n@@ -47,10 +47,10 @@ static int run_remote_archiver(int argc, const char **argv,\n \tif (name_hint) {\n \t\tconst char *format = archive_format_from_filename(name_hint);\n \t\tif (format)\n-\t\t\tpacket_write(fd[1], \"argument --format=%s\\n\", format);\n+\t\t\tpacket_write_fmt(fd[1], \"argument --format=%s\\n\", format);\n \t}\n \tfor (i = 1; i < argc; i++)\n-\t\tpacket_write(fd[1], \"argument %s\\n\", argv[i]);\n+\t\tpacket_write_fmt(fd[1], \"argument %s\\n\", argv[i]);\n \tpacket_flush(fd[1]);\n \n \tbuf = packet_read_line(fd[0], NULL);\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 011db00..1ce7682 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -218,7 +218,7 @@ static int receive_pack_config(const char *var, const char *value, void *cb)\n static void show_ref(const char *path, const unsigned char *sha1)\n {\n \tif (sent_capabilities) {\n-\t\tpacket_write(1, \"%s %s\\n\", sha1_to_hex(sha1), path);\n+\t\tpacket_write_fmt(1, \"%s %s\\n\", sha1_to_hex(sha1), path);\n \t} else {\n \t\tstruct strbuf cap = STRBUF_INIT;\n \n@@ -233,7 +233,7 @@ static void show_ref(const char *path, const unsigned char *sha1)\n \t\tif (advertise_push_options)\n \t\t\tstrbuf_addstr(&cap, \" push-options\");\n \t\tstrbuf_addf(&cap, \" agent=%s\", git_user_agent_sanitized());\n-\t\tpacket_write(1, \"%s %s%c%s\\n\",\n+\t\tpacket_write_fmt(1, \"%s %s%c%s\\n\",\n \t\t\t sha1_to_hex(sha1), path, 0, cap.buf);\n \t\tstrbuf_release(&cap);\n \t\tsent_capabilities = 1;\ndiff --git a/builtin/remote-ext.c b/builtin/remote-ext.c\nindex 88eb8f9..11b48bf 100644\n--- a/builtin/remote-ext.c\n+++ b/builtin/remote-ext.c\n@@ -128,9 +128,9 @@ static void send_git_request(int stdin_fd, const char *serv, const char *repo,\n \tconst char *vhost)\n {\n \tif (!vhost)\n-\t\tpacket_write(stdin_fd, \"%s %s%c\", serv, repo, 0);\n+\t\tpacket_write_fmt(stdin_fd, \"%s %s%c\", serv, repo, 0);\n \telse\n-\t\tpacket_write(stdin_fd, \"%s %s%chost=%s%c\", serv, repo, 0,\n+\t\tpacket_write_fmt(stdin_fd, \"%s %s%chost=%s%c\", serv, repo, 0,\n \t\t\t vhost, 0);\n }\n \ndiff --git a/builtin/upload-archive.c b/builtin/upload-archive.c\nindex 2caedf1..dc872f6 100644\n--- a/builtin/upload-archive.c\n+++ b/builtin/upload-archive.c\n@@ -88,11 +88,11 @@ int cmd_upload_archive(int argc, const char **argv, const char *prefix)\n \twriter.git_cmd = 1;\n \tif (start_command(&writer)) {\n \t\tint err = errno;\n-\t\tpacket_write(1, \"NACK unable to spawn subprocess\\n\");\n+\t\tpacket_write_fmt(1, \"NACK unable to spawn subprocess\\n\");\n \t\tdie(\"upload-archive: %s\", strerror(err));\n \t}\n \n-\tpacket_write(1, \"ACK\\n\");\n+\tpacket_write_fmt(1, \"ACK\\n\");\n \tpacket_flush(1);\n \n \twhile (1) {\ndiff --git a/connect.c b/connect.c\nindex 722dc3f..5330d9c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -730,7 +730,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t * Note: Do not add any other headers here! Doing so\n \t\t * will cause older git-daemon servers to crash.\n \t\t */\n-\t\tpacket_write(fd[1],\n+\t\tpacket_write_fmt(fd[1],\n \t\t\t \"%s %s%chost=%s%c\",\n \t\t\t prog, path, 0,\n \t\t\t target_host, 0);\ndiff --git a/daemon.c b/daemon.c\nindex 425aad0..afce1b9 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -281,7 +281,7 @@ static int daemon_error(const char *dir, const char *msg)\n {\n \tif (!informative_errors)\n \t\tmsg = \"access denied or repository not exported\";\n-\tpacket_write(1, \"ERR %s: %s\", msg, dir);\n+\tpacket_write_fmt(1, \"ERR %s: %s\", msg, dir);\n \treturn -1;\n }\n \ndiff --git a/http-backend.c b/http-backend.c\nindex adc8c8c..eef0a36 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -464,7 +464,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg)\n \t\thdr_str(hdr, content_type, buf.buf);\n \t\tend_headers(hdr);\n \n-\t\tpacket_write(1, \"# service=git-%s\\n\", svc->name);\n+\t\tpacket_write_fmt(1, \"# service=git-%s\\n\", svc->name);\n \t\tpacket_flush(1);\n \n \t\targv[0] = svc->name;\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 62fdb37..0a9b61c 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -118,7 +118,7 @@ static void format_packet(struct strbuf *out, const char *fmt, va_list args)\n \tpacket_trace(out->buf + orig_len + 4, n - 4, 1);\n }\n \n-void packet_write(int fd, const char *fmt, ...)\n+void packet_write_fmt(int fd, const char *fmt, ...)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n \tva_list args;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3cb9d91..1902fb3 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -20,7 +20,7 @@\n * side can't, we stay with pure read/write interfaces.\n */\n void packet_flush(int fd);\n-void packet_write(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n \ndiff --git a/shallow.c b/shallow.c\nindex 54e2db7..d666e24 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -260,7 +260,7 @@ static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *c\n {\n \tint fd = *(int *)cb;\n \tif (graft->nr_parent == -1)\n-\t\tpacket_write(fd, \"shallow %s\\n\", oid_to_hex(&graft->oid));\n+\t\tpacket_write_fmt(fd, \"shallow %s\\n\", oid_to_hex(&graft->oid));\n \treturn 0;\n }\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex ca7f941..cd47de6 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -393,13 +393,13 @@ static int get_common_commits(void)\n \t\t\tif (multi_ack == 2 && got_common\n \t\t\t && !got_other && ok_to_give_up()) {\n \t\t\t\tsent_ready = 1;\n-\t\t\t\tpacket_write(1, \"ACK %s ready\\n\", last_hex);\n+\t\t\t\tpacket_write_fmt(1, \"ACK %s ready\\n\", last_hex);\n \t\t\t}\n \t\t\tif (have_obj.nr == 0 || multi_ack)\n-\t\t\t\tpacket_write(1, \"NAK\\n\");\n+\t\t\t\tpacket_write_fmt(1, \"NAK\\n\");\n \n \t\t\tif (no_done && sent_ready) {\n-\t\t\t\tpacket_write(1, \"ACK %s\\n\", last_hex);\n+\t\t\t\tpacket_write_fmt(1, \"ACK %s\\n\", last_hex);\n \t\t\t\treturn 0;\n \t\t\t}\n \t\t\tif (stateless_rpc)\n@@ -416,20 +416,20 @@ static int get_common_commits(void)\n \t\t\t\t\tconst char *hex = sha1_to_hex(sha1);\n \t\t\t\t\tif (multi_ack == 2) {\n \t\t\t\t\t\tsent_ready = 1;\n-\t\t\t\t\t\tpacket_write(1, \"ACK %s ready\\n\", hex);\n+\t\t\t\t\t\tpacket_write_fmt(1, \"ACK %s ready\\n\", hex);\n \t\t\t\t\t} else\n-\t\t\t\t\t\tpacket_write(1, \"ACK %s continue\\n\", hex);\n+\t\t\t\t\t\tpacket_write_fmt(1, \"ACK %s continue\\n\", hex);\n \t\t\t\t}\n \t\t\t\tbreak;\n \t\t\tdefault:\n \t\t\t\tgot_common = 1;\n \t\t\t\tmemcpy(last_hex, sha1_to_hex(sha1), 41);\n \t\t\t\tif (multi_ack == 2)\n-\t\t\t\t\tpacket_write(1, \"ACK %s common\\n\", last_hex);\n+\t\t\t\t\tpacket_write_fmt(1, \"ACK %s common\\n\", last_hex);\n \t\t\t\telse if (multi_ack)\n-\t\t\t\t\tpacket_write(1, \"ACK %s continue\\n\", last_hex);\n+\t\t\t\t\tpacket_write_fmt(1, \"ACK %s continue\\n\", last_hex);\n \t\t\t\telse if (have_obj.nr == 1)\n-\t\t\t\t\tpacket_write(1, \"ACK %s\\n\", last_hex);\n+\t\t\t\t\tpacket_write_fmt(1, \"ACK %s\\n\", last_hex);\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tcontinue;\n@@ -437,10 +437,10 @@ static int get_common_commits(void)\n \t\tif (!strcmp(line, \"done\")) {\n \t\t\tif (have_obj.nr > 0) {\n \t\t\t\tif (multi_ack)\n-\t\t\t\t\tpacket_write(1, \"ACK %s\\n\", last_hex);\n+\t\t\t\t\tpacket_write_fmt(1, \"ACK %s\\n\", last_hex);\n \t\t\t\treturn 0;\n \t\t\t}\n-\t\t\tpacket_write(1, \"NAK\\n\");\n+\t\t\tpacket_write_fmt(1, \"NAK\\n\");\n \t\t\treturn -1;\n \t\t}\n \t\tdie(\"git upload-pack: expected SHA1 list, got '%s'\", line);\n@@ -650,7 +650,7 @@ static void receive_needs(void)\n \t\twhile (result) {\n \t\t\tstruct object *object = &result->item->object;\n \t\t\tif (!(object->flags & (CLIENT_SHALLOW|NOT_SHALLOW))) {\n-\t\t\t\tpacket_write(1, \"shallow %s\",\n+\t\t\t\tpacket_write_fmt(1, \"shallow %s\",\n \t\t\t\t\t\toid_to_hex(&object->oid));\n \t\t\t\tregister_shallow(object->oid.hash);\n \t\t\t\tshallow_nr++;\n@@ -662,7 +662,7 @@ static void receive_needs(void)\n \t\t\tstruct object *object = shallows.objects[i].item;\n \t\t\tif (object->flags & NOT_SHALLOW) {\n \t\t\t\tstruct commit_list *parents;\n-\t\t\t\tpacket_write(1, \"unshallow %s\",\n+\t\t\t\tpacket_write_fmt(1, \"unshallow %s\",\n \t\t\t\t\toid_to_hex(&object->oid));\n \t\t\t\tobject->flags &= ~CLIENT_SHALLOW;\n \t\t\t\t/* make sure the real parents are parsed */\n@@ -741,7 +741,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\tstruct strbuf symref_info = STRBUF_INIT;\n \n \t\tformat_symref_info(&symref_info, cb_data);\n-\t\tpacket_write(1, \"%s %s%c%s%s%s%s%s agent=%s\\n\",\n+\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s agent=%s\\n\",\n \t\t\t oid_to_hex(oid), refname_nons,\n \t\t\t 0, capabilities,\n \t\t\t (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?\n@@ -753,11 +753,11 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\t\t git_user_agent_sanitized());\n \t\tstrbuf_release(&symref_info);\n \t} else {\n-\t\tpacket_write(1, \"%s %s\\n\", oid_to_hex(oid), refname_nons);\n+\t\tpacket_write_fmt(1, \"%s %s\\n\", oid_to_hex(oid), refname_nons);\n \t}\n \tcapabilities = NULL;\n \tif (!peel_ref(refname, peeled.hash))\n-\t\tpacket_write(1, \"%s %s^{}\\n\", oid_to_hex(&peeled), refname_nons);\n+\t\tpacket_write_fmt(1, \"%s %s^{}\\n\", oid_to_hex(&peeled), refname_nons);\n \treturn 0;\n }\n \n-- \n2.10.0\n\n"},{"id":"302234","messageId":"20160920190247.82189-4-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 03/11] run-command: move check_pipe() from write_or_die to run_command","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:39Z","receivedAt":"2016-09-20T19:03:06Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nMove check_pipe() to run_command and make it public. This is necessary\nto call the function from pkt-line in a subsequent patch.\n\nSigned-off-by: Lars Schneider \n---\n run-command.c | 13 +++++++++++++\n run-command.h | 2 ++\n write_or_die.c | 13 -------------\n 3 files changed, 15 insertions(+), 13 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 5a4dbb6..b72f6d1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -6,6 +6,19 @@\n #include \"thread-utils.h\"\n #include \"strbuf.h\"\n \n+void check_pipe(int err)\n+{\n+\tif (err == EPIPE) {\n+\t\tif (in_async())\n+\t\t\tasync_exit(141);\n+\n+\t\tsignal(SIGPIPE, SIG_DFL);\n+\t\traise(SIGPIPE);\n+\t\t/* Should never happen, but just in case... */\n+\t\texit(141);\n+\t}\n+}\n+\n void child_process_init(struct child_process *child)\n {\n \tmemset(child, 0, sizeof(*child));\ndiff --git a/run-command.h b/run-command.h\nindex 5066649..e7c5f71 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -54,6 +54,8 @@ int finish_command(struct child_process *);\n int finish_command_in_signal(struct child_process *);\n int run_command(struct child_process *);\n \n+void check_pipe(int err);\n+\n /*\n * Returns the path to the hook file, or NULL if the hook is missing\n * or disabled. Note that this points to static storage that will be\ndiff --git a/write_or_die.c b/write_or_die.c\nindex 0734432..eab8c8d 100644\n--- a/write_or_die.c\n+++ b/write_or_die.c\n@@ -1,19 +1,6 @@\n #include \"cache.h\"\n #include \"run-command.h\"\n \n-static void check_pipe(int err)\n-{\n-\tif (err == EPIPE) {\n-\t\tif (in_async())\n-\t\t\tasync_exit(141);\n-\n-\t\tsignal(SIGPIPE, SIG_DFL);\n-\t\traise(SIGPIPE);\n-\t\t/* Should never happen, but just in case... */\n-\t\texit(141);\n-\t}\n-}\n-\n /*\n * Some cases use stdio, but want to flush after the write\n * to get error handling (and to get better interactive\n-- \n2.10.0\n\n"},{"id":"302235","messageId":"20160920190247.82189-3-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 02/11] pkt-line: extract set_packet_header()","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:38Z","receivedAt":"2016-09-20T19:03:07Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nset_packet_header() converts an integer to a 4 byte hex string. Make\nthis function locally available so that other pkt-line functions can\nuse it.\n\nSigned-off-by: Lars Schneider \n---\n pkt-line.c | 19 +++++++++++++------\n 1 file changed, 13 insertions(+), 6 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 0a9b61c..e8adc0f 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -97,10 +97,20 @@ void packet_buf_flush(struct strbuf *buf)\n \tstrbuf_add(buf, \"0000\", 4);\n }\n \n-#define hex(a) (hexchar[(a) & 15])\n-static void format_packet(struct strbuf *out, const char *fmt, va_list args)\n+static void set_packet_header(char *buf, const int size)\n {\n \tstatic char hexchar[] = \"0123456789abcdef\";\n+\n+\t#define hex(a) (hexchar[(a) & 15])\n+\tbuf[0] = hex(size >> 12);\n+\tbuf[1] = hex(size >> 8);\n+\tbuf[2] = hex(size >> 4);\n+\tbuf[3] = hex(size);\n+\t#undef hex\n+}\n+\n+static void format_packet(struct strbuf *out, const char *fmt, va_list args)\n+{\n \tsize_t orig_len, n;\n \n \torig_len = out->len;\n@@ -111,10 +121,7 @@ static void format_packet(struct strbuf *out, const char *fmt, va_list args)\n \tif (n > LARGE_PACKET_MAX)\n \t\tdie(\"protocol error: impossibly long line\");\n \n-\tout->buf[orig_len + 0] = hex(n >> 12);\n-\tout->buf[orig_len + 1] = hex(n >> 8);\n-\tout->buf[orig_len + 2] = hex(n >> 4);\n-\tout->buf[orig_len + 3] = hex(n);\n+\tset_packet_header(&out->buf[orig_len], n);\n \tpacket_trace(out->buf + orig_len + 4, n - 4, 1);\n }\n \n-- \n2.10.0\n\n"},{"id":"302236","messageId":"20160920190247.82189-6-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 05/11] pkt-line: add packet_flush_gently()","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:41Z","receivedAt":"2016-09-20T19:03:10Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\npacket_flush() would die in case of a write error even though for some\ncallers an error would be acceptable. Add packet_flush_gently() which\nwrites a pkt-line flush packet like packet_flush() but does not die in\ncase of an error. The function is used in a subsequent patch.\n\nSigned-off-by: Lars Schneider \n---\n pkt-line.c | 8 ++++++++\n pkt-line.h | 1 +\n 2 files changed, 9 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 3b465fd..19f0271 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -91,6 +91,14 @@ void packet_flush(int fd)\n \twrite_or_die(fd, \"0000\", 4);\n }\n \n+int packet_flush_gently(int fd)\n+{\n+\tpacket_trace(\"0000\", 4, 1);\n+\tif (write_in_full(fd, \"0000\", 4) == 4)\n+\t\treturn 0;\n+\treturn error(\"flush packet write failed\");\n+}\n+\n void packet_buf_flush(struct strbuf *buf)\n {\n \tpacket_trace(\"0000\", 4, 1);\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3caea77..3fa0899 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -23,6 +23,7 @@ void packet_flush(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n \n /*\n-- \n2.10.0\n\n"},{"id":"302237","messageId":"20160920190247.82189-5-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 04/11] pkt-line: add packet_write_fmt_gently()","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:40Z","receivedAt":"2016-09-20T19:03:14Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\npacket_write_fmt() would die in case of a write error even though for\nsome callers an error would be acceptable. Add packet_write_fmt_gently()\nwhich writes a formatted pkt-line like packet_write_fmt() but does not\ndie in case of an error. The function is used in a subsequent patch.\n\nSigned-off-by: Lars Schneider \n---\n pkt-line.c | 34 ++++++++++++++++++++++++++++++----\n pkt-line.h | 1 +\n 2 files changed, 31 insertions(+), 4 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex e8adc0f..3b465fd 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -125,16 +125,42 @@ static void format_packet(struct strbuf *out, const char *fmt, va_list args)\n \tpacket_trace(out->buf + orig_len + 4, n - 4, 1);\n }\n \n+static int packet_write_fmt_1(int fd, int gently,\n+ const char *fmt, va_list args)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t count;\n+\n+\tformat_packet(&buf, fmt, args);\n+\tcount = write_in_full(fd, buf.buf, buf.len);\n+\tif (count == buf.len)\n+\t\treturn 0;\n+\n+\tif (!gently) {\n+\t\tcheck_pipe(errno);\n+\t\tdie_errno(\"packet write with format failed\");\n+\t}\n+\treturn error(\"packet write with format failed\");\n+}\n+\n void packet_write_fmt(int fd, const char *fmt, ...)\n {\n-\tstatic struct strbuf buf = STRBUF_INIT;\n \tva_list args;\n \n-\tstrbuf_reset(&buf);\n \tva_start(args, fmt);\n-\tformat_packet(&buf, fmt, args);\n+\tpacket_write_fmt_1(fd, 0, fmt, args);\n+\tva_end(args);\n+}\n+\n+int packet_write_fmt_gently(int fd, const char *fmt, ...)\n+{\n+\tint status;\n+\tva_list args;\n+\n+\tva_start(args, fmt);\n+\tstatus = packet_write_fmt_1(fd, 1, fmt, args);\n \tva_end(args);\n-\twrite_or_die(fd, buf.buf, buf.len);\n+\treturn status;\n }\n \n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 1902fb3..3caea77 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -23,6 +23,7 @@ void packet_flush(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n \n /*\n * Read a packetized line into the buffer, which must be at least size bytes\n-- \n2.10.0\n\n"},{"id":"302238","messageId":"20160920190247.82189-11-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 10/11] convert: make apply_filter() adhere to standard Git error handling","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:46Z","receivedAt":"2016-09-20T19:03:17Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\napply_filter() returns a boolean that tells the caller if it\n\"did convert or did not convert\". The variable `ret` was used throughout\nthe function to track errors whereas `1` denoted success and `0`\nfailure. This is unusual for the Git source where `0` denotes success.\n\nRename the variable and flip its value to make the function easier\nreadable for Git developers.\n\nSigned-off-by: Lars Schneider \n---\n convert.c | 15 ++++++---------\n 1 file changed, 6 insertions(+), 9 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 986c239..597f561 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -451,7 +451,7 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \t *\n \t * (child --> cmd) --> us\n \t */\n-\tint ret = 1;\n+\tint err = 0;\n \tstruct strbuf nbuf = STRBUF_INIT;\n \tstruct async async;\n \tstruct filter_params params;\n@@ -477,23 +477,20 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \t\treturn 0;\t/* error was already reported */\n \n \tif (strbuf_read(&nbuf, async.out, len) < 0) {\n-\t\terror(\"read from external filter '%s' failed\", cmd);\n-\t\tret = 0;\n+\t\terr = error(\"read from external filter '%s' failed\", cmd);\n \t}\n \tif (close(async.out)) {\n-\t\terror(\"read from external filter '%s' failed\", cmd);\n-\t\tret = 0;\n+\t\terr = error(\"read from external filter '%s' failed\", cmd);\n \t}\n \tif (finish_async(&async)) {\n-\t\terror(\"external filter '%s' failed\", cmd);\n-\t\tret = 0;\n+\t\terr = error(\"external filter '%s' failed\", cmd);\n \t}\n \n-\tif (ret) {\n+\tif (!err) {\n \t\tstrbuf_swap(dst, &nbuf);\n \t}\n \tstrbuf_release(&nbuf);\n-\treturn ret;\n+\treturn !err;\n }\n \n static struct convert_driver {\n-- \n2.10.0\n\n"},{"id":"302239","messageId":"20160920190247.82189-7-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 06/11] pkt-line: add packet_write_gently()","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:42Z","receivedAt":"2016-09-20T19:03:18Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\npacket_write_fmt_gently() uses format_packet() which lets the caller\nonly send string data via \"%s\". That means it cannot be used for\narbitrary data that may contain NULs.\n\nAdd packet_write_gently() which writes arbitrary data and does not die\nin case of an error. The function is used by other pkt-line functions in\na subsequent patch.\n\nSigned-off-by: Lars Schneider \n---\n pkt-line.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 19f0271..fc0ac12 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -171,6 +171,22 @@ int packet_write_fmt_gently(int fd, const char *fmt, ...)\n \treturn status;\n }\n \n+static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n+{\n+\tstatic char packet_write_buffer[LARGE_PACKET_MAX];\n+\n+\tif (size > sizeof(packet_write_buffer) - 4) {\n+\t\treturn error(\"packet write failed - data exceeds max packet size\");\n+\t}\n+\tpacket_trace(buf, size, 1);\n+\tsize += 4;\n+\tset_packet_header(packet_write_buffer, size);\n+\tmemcpy(packet_write_buffer + 4, buf, size - 4);\n+\tif (write_in_full(fd_out, packet_write_buffer, size) == size)\n+\t\treturn 0;\n+\treturn error(\"packet write failed\");\n+}\n+\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n {\n \tva_list args;\n-- \n2.10.0\n\n"},{"id":"302240","messageId":"20160920190247.82189-10-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 09/11] convert: modernize tests","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:45Z","receivedAt":"2016-09-20T19:03:21Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nUse `test_config` to set the config, check that files are empty with\n`test_must_be_empty`, compare files with `test_cmp`, and remove spaces\nafter \">\" and \"<\".\n\nPlease note that the \"rot13\" filter configured in \"setup\" keeps using\n`git config` instead of `test_config` because subsequent tests might\ndepend on it.\n\nReviewed-by: Stefan Beller \nSigned-off-by: Lars Schneider \n---\n t/t0021-conversion.sh | 58 +++++++++++++++++++++++++--------------------------\n 1 file changed, 29 insertions(+), 29 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex e799e59..dc50938 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -38,8 +38,8 @@ script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n \n test_expect_success check '\n \n-\tcmp test.o test &&\n-\tcmp test.o test.t &&\n+\ttest_cmp test.o test &&\n+\ttest_cmp test.o test.t &&\n \n \t# ident should be stripped in the repository\n \tgit diff --raw --exit-code :test :test.i &&\n@@ -47,10 +47,10 @@ test_expect_success check '\n \tembedded=$(sed -ne \"$script\" test.i) &&\n \ttest \"z$id\" = \"z$embedded\" &&\n \n-\tgit cat-file blob :test.t > test.r &&\n+\tgit cat-file blob :test.t >test.r &&\n \n-\t./rot13.sh < test.o > test.t &&\n-\tcmp test.r test.t\n+\t./rot13.sh test.t &&\n+\ttest_cmp test.r test.t\n '\n \n # If an expanded ident ever gets into the repository, we want to make sure that\n@@ -130,7 +130,7 @@ test_expect_success 'filter shell-escaped filenames' '\n \n \t# delete the files and check them out again, using a smudge filter\n \t# that will count the args and echo the command-line back to us\n-\tgit config filter.argc.smudge \"sh ./argc.sh %f\" &&\n+\ttest_config filter.argc.smudge \"sh ./argc.sh %f\" &&\n \trm \"$normal\" \"$special\" &&\n \tgit checkout -- \"$normal\" \"$special\" &&\n \n@@ -141,7 +141,7 @@ test_expect_success 'filter shell-escaped filenames' '\n \ttest_cmp expect \"$special\" &&\n \n \t# do the same thing, but with more args in the filter expression\n-\tgit config filter.argc.smudge \"sh ./argc.sh %f --my-extra-arg\" &&\n+\ttest_config filter.argc.smudge \"sh ./argc.sh %f --my-extra-arg\" &&\n \trm \"$normal\" \"$special\" &&\n \tgit checkout -- \"$normal\" \"$special\" &&\n \n@@ -154,9 +154,9 @@ test_expect_success 'filter shell-escaped filenames' '\n '\n \n test_expect_success 'required filter should filter data' '\n-\tgit config filter.required.smudge ./rot13.sh &&\n-\tgit config filter.required.clean ./rot13.sh &&\n-\tgit config filter.required.required true &&\n+\ttest_config filter.required.smudge ./rot13.sh &&\n+\ttest_config filter.required.clean ./rot13.sh &&\n+\ttest_config filter.required.required true &&\n \n \techo \"*.r filter=required\" >.gitattributes &&\n \n@@ -165,17 +165,17 @@ test_expect_success 'required filter should filter data' '\n \n \trm -f test.r &&\n \tgit checkout -- test.r &&\n-\tcmp test.o test.r &&\n+\ttest_cmp test.o test.r &&\n \n \t./rot13.sh expected &&\n \tgit cat-file blob :test.r >actual &&\n-\tcmp expected actual\n+\ttest_cmp expected actual\n '\n \n test_expect_success 'required filter smudge failure' '\n-\tgit config filter.failsmudge.smudge false &&\n-\tgit config filter.failsmudge.clean cat &&\n-\tgit config filter.failsmudge.required true &&\n+\ttest_config filter.failsmudge.smudge false &&\n+\ttest_config filter.failsmudge.clean cat &&\n+\ttest_config filter.failsmudge.required true &&\n \n \techo \"*.fs filter=failsmudge\" >.gitattributes &&\n \n@@ -186,9 +186,9 @@ test_expect_success 'required filter smudge failure' '\n '\n \n test_expect_success 'required filter clean failure' '\n-\tgit config filter.failclean.smudge cat &&\n-\tgit config filter.failclean.clean false &&\n-\tgit config filter.failclean.required true &&\n+\ttest_config filter.failclean.smudge cat &&\n+\ttest_config filter.failclean.clean false &&\n+\ttest_config filter.failclean.required true &&\n \n \techo \"*.fc filter=failclean\" >.gitattributes &&\n \n@@ -197,8 +197,8 @@ test_expect_success 'required filter clean failure' '\n '\n \n test_expect_success 'filtering large input to small output should use little memory' '\n-\tgit config filter.devnull.clean \"cat >/dev/null\" &&\n-\tgit config filter.devnull.required true &&\n+\ttest_config filter.devnull.clean \"cat >/dev/null\" &&\n+\ttest_config filter.devnull.required true &&\n \tfor i in $(test_seq 1 30); do printf \"%1048576d\" 1; done >30MB &&\n \techo \"30MB filter=devnull\" >.gitattributes &&\n \tGIT_MMAP_LIMIT=1m GIT_ALLOC_LIMIT=1m git add 30MB\n@@ -207,7 +207,7 @@ test_expect_success 'filtering large input to small output should use little mem\n test_expect_success 'filter that does not read is fine' '\n \ttest-genrandom foo $((128 * 1024 + 1)) >big &&\n \techo \"big filter=epipe\" >.gitattributes &&\n-\tgit config filter.epipe.clean \"echo xyzzy\" &&\n+\ttest_config filter.epipe.clean \"echo xyzzy\" &&\n \tgit add big &&\n \tgit cat-file blob :big >actual &&\n \techo xyzzy >expect &&\n@@ -215,20 +215,20 @@ test_expect_success 'filter that does not read is fine' '\n '\n \n test_expect_success EXPENSIVE 'filter large file' '\n-\tgit config filter.largefile.smudge cat &&\n-\tgit config filter.largefile.clean cat &&\n+\ttest_config filter.largefile.smudge cat &&\n+\ttest_config filter.largefile.clean cat &&\n \tfor i in $(test_seq 1 2048); do printf \"%1048576d\" 1; done >2GB &&\n \techo \"2GB filter=largefile\" >.gitattributes &&\n \tgit add 2GB 2>err &&\n-\t! test -s err &&\n+\ttest_must_be_empty err &&\n \trm -f 2GB &&\n \tgit checkout -- 2GB 2>err &&\n-\t! test -s err\n+\ttest_must_be_empty err\n '\n \n test_expect_success \"filter: clean empty file\" '\n-\tgit config filter.in-repo-header.clean \"echo cleaned && cat\" &&\n-\tgit config filter.in-repo-header.smudge \"sed 1d\" &&\n+\ttest_config filter.in-repo-header.clean \"echo cleaned && cat\" &&\n+\ttest_config filter.in-repo-header.smudge \"sed 1d\" &&\n \n \techo \"empty-in-worktree filter=in-repo-header\" >>.gitattributes &&\n \t>empty-in-worktree &&\n@@ -240,8 +240,8 @@ test_expect_success \"filter: clean empty file\" '\n '\n \n test_expect_success \"filter: smudge empty file\" '\n-\tgit config filter.empty-in-repo.clean \"cat >/dev/null\" &&\n-\tgit config filter.empty-in-repo.smudge \"echo smudged && cat\" &&\n+\ttest_config filter.empty-in-repo.clean \"cat >/dev/null\" &&\n+\ttest_config filter.empty-in-repo.smudge \"echo smudged && cat\" &&\n \n \techo \"empty-in-repo filter=empty-in-repo\" >>.gitattributes &&\n \techo dead data walking >empty-in-repo &&\n-- \n2.10.0\n\n"},{"id":"302241","messageId":"20160920190247.82189-9-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 08/11] convert: quote filter names in error messages","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:44Z","receivedAt":"2016-09-20T19:03:26Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nGit filter driver commands with spaces (e.g. `filter.sh foo`) are hard\nto read in error messages. Quote them to improve the readability.\n\nSigned-off-by: Lars Schneider \n---\n convert.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 077f5e6..986c239 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -412,7 +412,7 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \tchild_process.out = out;\n \n \tif (start_command(&child_process))\n-\t\treturn error(\"cannot fork to run external filter %s\", params->cmd);\n+\t\treturn error(\"cannot fork to run external filter '%s'\", params->cmd);\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n@@ -430,13 +430,13 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \tif (close(child_process.in))\n \t\twrite_err = 1;\n \tif (write_err)\n-\t\terror(\"cannot feed the input to external filter %s\", params->cmd);\n+\t\terror(\"cannot feed the input to external filter '%s'\", params->cmd);\n \n \tsigchain_pop(SIGPIPE);\n \n \tstatus = finish_command(&child_process);\n \tif (status)\n-\t\terror(\"external filter %s failed %d\", params->cmd, status);\n+\t\terror(\"external filter '%s' failed %d\", params->cmd, status);\n \n \tstrbuf_release(&cmd);\n \treturn (write_err || status);\n@@ -477,15 +477,15 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \t\treturn 0;\t/* error was already reported */\n \n \tif (strbuf_read(&nbuf, async.out, len) < 0) {\n-\t\terror(\"read from external filter %s failed\", cmd);\n+\t\terror(\"read from external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \tif (close(async.out)) {\n-\t\terror(\"read from external filter %s failed\", cmd);\n+\t\terror(\"read from external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \tif (finish_async(&async)) {\n-\t\terror(\"external filter %s failed\", cmd);\n+\t\terror(\"external filter '%s' failed\", cmd);\n \t\tret = 0;\n \t}\n \n-- \n2.10.0\n\n"},{"id":"302242","messageId":"20160920190247.82189-8-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 07/11] pkt-line: add functions to read/write flush terminated packet streams","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:43Z","receivedAt":"2016-09-20T19:03:28Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nwrite_packetized_from_fd() and write_packetized_from_buf() write a\nstream of packets. All content packets use the maximal packet size\nexcept for the last one. After the last content packet a `flush` control\npacket is written.\n\nread_packetized_to_buf() reads arbitrary sized packets until it detects\na `flush` packet.\n\nSigned-off-by: Lars Schneider \n---\n pkt-line.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n pkt-line.h | 7 +++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex fc0ac12..a0a8543 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -196,6 +196,47 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n \tva_end(args);\n }\n \n+int write_packetized_from_fd(int fd_in, int fd_out)\n+{\n+\tstatic char buf[PKTLINE_DATA_MAXLEN];\n+\tint err = 0;\n+\tssize_t bytes_to_write;\n+\n+\twhile (!err) {\n+\t\tbytes_to_write = xread(fd_in, buf, sizeof(buf));\n+\t\tif (bytes_to_write < 0)\n+\t\t\treturn COPY_READ_ERROR;\n+\t\tif (bytes_to_write == 0)\n+\t\t\tbreak;\n+\t\terr = packet_write_gently(fd_out, buf, bytes_to_write);\n+\t}\n+\tif (!err)\n+\t\terr = packet_flush_gently(fd_out);\n+\treturn err;\n+}\n+\n+int write_packetized_from_buf(const char *src_in, size_t len, int fd_out)\n+{\n+\tstatic char buf[PKTLINE_DATA_MAXLEN];\n+\tint err = 0;\n+\tsize_t bytes_written = 0;\n+\tsize_t bytes_to_write;\n+\n+\twhile (!err) {\n+\t\tif ((len - bytes_written) > sizeof(buf))\n+\t\t\tbytes_to_write = sizeof(buf);\n+\t\telse\n+\t\t\tbytes_to_write = len - bytes_written;\n+\t\tif (bytes_to_write == 0)\n+\t\t\tbreak;\n+\t\terr = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);\n+\t\tbytes_written += bytes_to_write;\n+\t}\n+\tif (!err)\n+\t\terr = packet_flush_gently(fd_out);\n+\treturn err;\n+}\n+\n static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n \t\t\t void *dst, unsigned size, int options)\n {\n@@ -305,3 +346,30 @@ char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)\n {\n \treturn packet_read_line_generic(-1, src, src_len, dst_len);\n }\n+\n+ssize_t read_packetized_to_buf(int fd_in, struct strbuf *sb_out)\n+{\n+\tint paket_len;\n+\tint options = PACKET_READ_GENTLE_ON_EOF;\n+\n+\tsize_t oldlen = sb_out->len;\n+\tsize_t oldalloc = sb_out->alloc;\n+\n+\tfor (;;) {\n+\t\tstrbuf_grow(sb_out, PKTLINE_DATA_MAXLEN+1);\n+\t\tpaket_len = packet_read(fd_in, NULL, NULL,\n+\t\t\tsb_out->buf + sb_out->len, PKTLINE_DATA_MAXLEN+1, options);\n+\t\tif (paket_len <= 0)\n+\t\t\tbreak;\n+\t\tsb_out->len += paket_len;\n+\t}\n+\n+\tif (paket_len < 0) {\n+\t\tif (oldalloc == 0)\n+\t\t\tstrbuf_release(sb_out);\n+\t\telse\n+\t\t\tstrbuf_setlen(sb_out, oldlen);\n+\t\treturn paket_len;\n+\t}\n+\treturn sb_out->len - oldlen;\n+}\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3fa0899..6df8449 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -25,6 +25,8 @@ void packet_buf_flush(struct strbuf *buf);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+int write_packetized_from_fd(int fd_in, int fd_out);\n+int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n \n /*\n * Read a packetized line into the buffer, which must be at least size bytes\n@@ -77,6 +79,11 @@ char *packet_read_line(int fd, int *size);\n */\n char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n \n+/*\n+ * Reads a stream of variable sized packets until a flush packet is detected.\n+ */\n+ssize_t read_packetized_to_buf(int fd_in, struct strbuf *sb_out);\n+\n #define DEFAULT_PACKET_MAX 1000\n #define LARGE_PACKET_MAX 65520\n extern char packet_buffer[LARGE_PACKET_MAX];\n-- \n2.10.0\n\n"},{"id":"302243","messageId":"20160920190247.82189-12-larsxschneider@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"[PATCH v8 11/11] convert: add filter..process option","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-20T19:02:47Z","receivedAt":"2016-09-20T19:03:32Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nGit's clean/smudge mechanism invokes an external filter process for\nevery single blob that is affected by a filter. If Git filters a lot of\nblobs then the startup time of the external filter processes can become\na significant part of the overall Git execution time.\n\nIn a preliminary performance test this developer used a clean/smudge\nfilter written in golang to filter 12,000 files. This process took 364s\nwith the existing filter mechanism and 5s with the new mechanism. See\ndetails here: https://github.com/github/git-lfs/pull/1382\n\nThis patch adds the `filter..process` string option which, if\nused, keeps the external filter process running and processes all blobs\nwith the packet format (pkt-line) based protocol over standard input and\nstandard output. The full protocol is explained in detail in\n`Documentation/gitattributes.txt`.\n\nA few key decisions:\n\n* The long running filter process is referred to as filter protocol\n version 2 because the existing single shot filter invocation is\n considered version 1.\n* Git sends a welcome message and expects a response right after the\n external filter process has started. This ensures that Git will not\n hang if a version 1 filter is incorrectly used with the\n filter..process option for version 2 filters. In addition,\n Git can detect this kind of error and warn the user.\n* The status of a filter operation (e.g. \"success\" or \"error) is set\n before the actual response and (if necessary!) re-set after the\n response. The advantage of this two step status response is that if\n the filter detects an error early, then the filter can communicate\n this and Git does not even need to create structures to read the\n response.\n* All status responses are pkt-line lists terminated with a flush\n packet. This allows us to send other status fields with the same\n protocol in the future.\n\nHelped-by: Martin-Louis Bright \nReviewed-by: Jakub Narebski \nSigned-off-by: Lars Schneider \n---\n Documentation/gitattributes.txt | 156 +++++++++++++-\n contrib/long-running-filter/example.pl | 123 +++++++++++\n convert.c | 348 ++++++++++++++++++++++++++++---\n pkt-line.h | 1 +\n t/t0021-conversion.sh | 365 ++++++++++++++++++++++++++++++++-\n t/t0021/rot13-filter.pl | 191 +++++++++++++++++\n 6 files changed, 1153 insertions(+), 31 deletions(-)\n create mode 100755 contrib/long-running-filter/example.pl\n create mode 100755 t/t0021/rot13-filter.pl\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 7aff940..946dcad 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -293,7 +293,13 @@ checkout, when the `smudge` command is specified, the command is\n fed the blob object from its standard input, and its standard\n output is used to update the worktree file. Similarly, the\n `clean` command is used to convert the contents of worktree file\n-upon checkin.\n+upon checkin. By default these commands process only a single\n+blob and terminate. If a long running `process` filter is used\n+in place of `clean` and/or `smudge` filters, then Git can process\n+all blobs with a single filter command invocation for the entire\n+life of a single Git command, for example `git add --all`. See\n+section below for the description of the protocol used to\n+communicate with a `process` filter.\n \n One use of the content filtering is to massage the content into a shape\n that is more convenient for the platform, filesystem, and the user to use.\n@@ -373,6 +379,154 @@ not exist, or may have different contents. So, smudge and clean commands\n should not try to access the file on disk, but only act as filters on the\n content provided to them on standard input.\n \n+Long Running Filter Process\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+If the filter command (a string value) is defined via\n+`filter..process` then Git can process all blobs with a\n+single filter invocation for the entire life of a single Git\n+command. This is achieved by using a packet format (pkt-line,\n+see technical/protocol-common.txt) based protocol over standard\n+input and standard output as follows. All packets are considered\n+text and therefore are terminated by an LF. Exceptions are the\n+\"*CONTENT\" packets and the flush packet.\n+\n+Git starts the filter when it encounters the first file\n+that needs to be cleaned or smudged. After the filter started\n+Git sends a welcome message (\"git-filter-client\"), a list of\n+supported protocol version numbers, and a flush packet. Git expects\n+to read a welcome response message (\"git-filter-server\") and exactly\n+one protocol version number from the previously sent list. All further\n+communication will be based on the selected version. The remaining\n+protocol description below documents \"version=2\". Please note that\n+\"version=42\" in the example below does not exist and is only there\n+to illustrate how the protocol would look like with more than one\n+version.\n+\n+After the version negotiation Git sends a list of supported capabilities\n+and a flush packet. Git expects to read a list of desired capabilities,\n+which must be a subset of the supported capabilities list, and a flush\n+packet as response:\n+------------------------\n+packet: git> git-filter-client\n+packet: git> version=2\n+packet: git> version=42\n+packet: git> 0000\n+packet: git< git-filter-server\n+packet: git< version=2\n+packet: git> clean=true\n+packet: git> smudge=true\n+packet: git> not-yet-invented=true\n+packet: git> 0000\n+packet: git< clean=true\n+packet: git< smudge=true\n+packet: git< 0000\n+------------------------\n+Supported filter capabilities in version 2 are \"clean\" and\n+\"smudge\".\n+\n+Afterwards Git sends a list of \"key=value\" pairs terminated with\n+a flush packet. The list will contain at least the filter command\n+(based on the supported capabilities) and the pathname of the file\n+to filter relative to the repository root. Right after these packets\n+Git sends the content split in zero or more pkt-line packets and a\n+flush packet to terminate content.\n+------------------------\n+packet: git> command=smudge\n+packet: git> pathname=path/testfile.dat\n+packet: git> 0000\n+packet: git> CONTENT\n+packet: git> 0000\n+------------------------\n+\n+The filter is expected to respond with a list of \"key=value\" pairs\n+terminated with a flush packet. If the filter does not experience\n+problems then the list must contain a \"success\" status. Right after\n+these packets the filter is expected to send the content in zero\n+or more pkt-line packets and a flush packet at the end. Finally, a\n+second list of \"key=value\" pairs terminated with a flush packet\n+is expected. The filter can change the status in the second list.\n+------------------------\n+packet: git< status=success\n+packet: git< 0000\n+packet: git< SMUDGED_CONTENT\n+packet: git< 0000\n+packet: git< 0000 # empty list!\n+------------------------\n+\n+If the result content is empty then the filter is expected to respond\n+with a success status and an empty list.\n+------------------------\n+packet: git< status=success\n+packet: git< 0000\n+packet: git< 0000 # empty content!\n+packet: git< 0000 # empty list!\n+------------------------\n+\n+In case the filter cannot or does not want to process the content,\n+it is expected to respond with an \"error\" status. Depending on the\n+`filter..required` flag Git will interpret that as error\n+but it will not stop or restart the filter process.\n+------------------------\n+packet: git< status=error\n+packet: git< 0000\n+------------------------\n+\n+If the filter experiences an error during processing, then it can\n+send the status \"error\" after the content was (partially or\n+completely) sent. Depending on the `filter..required` flag\n+Git will interpret that as error but it will not stop or restart the\n+filter process.\n+------------------------\n+packet: git< status=success\n+packet: git< 0000\n+packet: git< HALF_WRITTEN_ERRONEOUS_CONTENT\n+packet: git< 0000\n+packet: git< status=error\n+packet: git< 0000\n+------------------------\n+\n+If the filter dies during the communication or does not adhere to\n+the protocol then Git will stop the filter process and restart it\n+with the next file that needs to be processed. Depending on the\n+`filter..required` flag Git will interpret that as error.\n+\n+The error handling for all cases above mimic the behavior of\n+the `filter..clean` / `filter..smudge` error\n+handling.\n+\n+In case the filter cannot or does not want to process the content\n+as well as any future content for the lifetime of the Git process,\n+it is expected to respond with an \"abort\" status. Depending on\n+the `filter..required` flag Git will interpret that as error\n+for the content as well as any future content for the lifetime of the\n+Git process but it will not stop or restart the filter process.\n+------------------------\n+packet: git< status=abort\n+packet: git< 0000\n+------------------------\n+\n+After the filter has processed a blob it is expected to wait for\n+the next \"key=value\" list containing a command. Git will close\n+the command pipe on exit. The filter is expected to detect EOF\n+and exit gracefully on its own.\n+\n+A long running filter demo implementation can be found in\n+`contrib/long-running-filter/example.pl` located in the Git\n+core repository. If you develop your own long running filter\n+process then the `GIT_TRACE_PACKET` environment variables can be\n+very helpful for debugging (see linkgit:git[1]).\n+\n+If a `filter..process` command is configured then it\n+always takes precedence over a configured `filter..clean`\n+or `filter..smudge` command.\n+\n+Please note that you cannot use an existing `filter..clean`\n+or `filter..smudge` command with `filter..process`\n+because the former two use a different inter process communication\n+protocol than the latter one.\n+\n+\n Interaction between checkin/checkout attributes\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \ndiff --git a/contrib/long-running-filter/example.pl b/contrib/long-running-filter/example.pl\nnew file mode 100755\nindex 0000000..c13a631\n--- /dev/null\n+++ b/contrib/long-running-filter/example.pl\n@@ -0,0 +1,123 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git filter protocol version 2\n+# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n+#\n+\n+use strict;\n+use warnings;\n+\n+my $MAX_PACKET_CONTENT_SIZE = 65516;\n+\n+sub packet_bin_read {\n+ my $buffer;\n+ my $bytes_read = read STDIN, $buffer, 4;\n+ if ( $bytes_read == 0 ) {\n+\n+ # EOF - Git stopped talking to us!\n+ exit();\n+ }\n+ elsif ( $bytes_read != 4 ) {\n+ die \"invalid packet size '$bytes_read' field\";\n+ }\n+ my $pkt_size = hex($buffer);\n+ if ( $pkt_size == 0 ) {\n+ return ( 1, \"\" );\n+ }\n+ elsif ( $pkt_size > 4 ) {\n+ my $content_size = $pkt_size - 4;\n+ $bytes_read = read STDIN, $buffer, $content_size;\n+ if ( $bytes_read != $content_size ) {\n+ die \"invalid packet ($content_size expected; $bytes_read read)\";\n+ }\n+ return ( 0, $buffer );\n+ }\n+ else {\n+ die \"invalid packet size\";\n+ }\n+}\n+\n+sub packet_txt_read {\n+ my ( $res, $buf ) = packet_bin_read();\n+ unless ( $buf =~ /\\n$/ ) {\n+ die \"A non-binary line SHOULD BE terminated by an LF.\";\n+ }\n+ return ( $res, substr( $buf, 0, -1 ) );\n+}\n+\n+sub packet_bin_write {\n+ my ($packet) = @_;\n+ print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n+ print STDOUT $packet;\n+ STDOUT->flush();\n+}\n+\n+sub packet_txt_write {\n+ packet_bin_write( $_[0] . \"\\n\" );\n+}\n+\n+sub packet_flush {\n+ print STDOUT sprintf( \"%04x\", 0 );\n+ STDOUT->flush();\n+}\n+\n+( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n+( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n+( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n+\n+packet_txt_write(\"git-filter-server\");\n+packet_txt_write(\"version=2\");\n+\n+( packet_txt_read() eq ( 0, \"clean=true\" ) ) || die \"bad capability\";\n+( packet_txt_read() eq ( 0, \"smudge=true\" ) ) || die \"bad capability\";\n+( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n+\n+packet_txt_write(\"clean=true\");\n+packet_txt_write(\"smudge=true\");\n+packet_flush();\n+\n+while (1) {\n+ my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n+ my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;\n+\n+ packet_bin_read();\n+\n+ my $input = \"\";\n+ {\n+ binmode(STDIN);\n+ my $buffer;\n+ my $done = 0;\n+ while ( !$done ) {\n+ ( $done, $buffer ) = packet_bin_read();\n+ $input .= $buffer;\n+ }\n+ }\n+\n+ my $output;\n+ if ( $command eq \"clean\" ) {\n+ ### Perform clean here ###\n+ $output = $input;\n+ }\n+ elsif ( $command eq \"smudge\" ) {\n+ ### Perform smudge here ###\n+ $output = $input;\n+ }\n+ else {\n+ die \"bad command '$command'\";\n+ }\n+\n+ packet_txt_write(\"status=success\");\n+ packet_flush();\n+ while ( length($output) > 0 ) {\n+ my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n+ packet_bin_write($packet);\n+ if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n+ $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n+ }\n+ else {\n+ $output = \"\";\n+ }\n+ }\n+ packet_flush(); # flush content!\n+ packet_flush(); # empty list!\n+}\ndiff --git a/convert.c b/convert.c\nindex 597f561..bd66257 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -3,6 +3,7 @@\n #include \"run-command.h\"\n #include \"quote.h\"\n #include \"sigchain.h\"\n+#include \"pkt-line.h\"\n \n /*\n * convert.c - convert a file when checking it out and checking it in.\n@@ -442,7 +443,7 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \treturn (write_err || status);\n }\n \n-static int apply_filter(const char *path, const char *src, size_t len, int fd,\n+static int apply_single_file_filter(const char *path, const char *src, size_t len, int fd,\n struct strbuf *dst, const char *cmd)\n {\n \t/*\n@@ -456,12 +457,6 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \tstruct async async;\n \tstruct filter_params params;\n \n-\tif (!cmd || !*cmd)\n-\t\treturn 0;\n-\n-\tif (!dst)\n-\t\treturn 1;\n-\n \tmemset(&async, 0, sizeof(async));\n \tasync.proc = filter_buffer_or_fd;\n \tasync.data = ¶ms;\n@@ -493,14 +488,317 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \treturn !err;\n }\n \n+#define CAP_CLEAN (1u<<0)\n+#define CAP_SMUDGE (1u<<1)\n+\n+struct cmd2process {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tunsigned int supported_capabilities;\n+\tconst char *cmd;\n+\tstruct child_process process;\n+};\n+\n+static int cmd_process_map_initialized;\n+static struct hashmap cmd_process_map;\n+\n+static int cmd2process_cmp(const struct cmd2process *e1,\n+ const struct cmd2process *e2,\n+ const void *unused)\n+{\n+\treturn strcmp(e1->cmd, e2->cmd);\n+}\n+\n+static struct cmd2process *find_multi_file_filter_entry(struct hashmap *hashmap, const char *cmd)\n+{\n+\tstruct cmd2process key;\n+\thashmap_entry_init(&key, strhash(cmd));\n+\tkey.cmd = cmd;\n+\treturn hashmap_get(hashmap, &key, NULL);\n+}\n+\n+static void kill_multi_file_filter(struct hashmap *hashmap, struct cmd2process *entry)\n+{\n+\tif (!entry)\n+\t\treturn;\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t/*\n+\t * We kill the filter most likely because an error happened already.\n+\t * That's why we are not interested in any error code here.\n+\t */\n+\tclose(entry->process.in);\n+\tclose(entry->process.out);\n+\tsigchain_pop(SIGPIPE);\n+\tfinish_command(&entry->process);\n+\thashmap_remove(hashmap, entry, NULL);\n+\tfree(entry);\n+}\n+\n+static int packet_write_list(int fd, const char *line, ...)\n+{\n+\tva_list args;\n+\tint err;\n+\tva_start(args, line);\n+\tfor (;;) {\n+\t\tif (!line)\n+\t\t\tbreak;\n+\t\tif (strlen(line) > PKTLINE_DATA_MAXLEN)\n+\t\t\treturn -1;\n+\t\terr = packet_write_fmt_gently(fd, \"%s\\n\", line);\n+\t\tif (err)\n+\t\t\treturn err;\n+\t\tline = va_arg(args, const char*);\n+\t}\n+\tva_end(args);\n+\treturn packet_flush_gently(fd);\n+}\n+\n+static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n+{\n+\tint err;\n+\tstruct cmd2process *entry;\n+\tstruct child_process *process;\n+\tconst char *argv[] = { cmd, NULL };\n+\tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n+\tchar *cap_buf;\n+\tconst char *cap_name;\n+\n+\tentry = xmalloc(sizeof(*entry));\n+\thashmap_entry_init(entry, strhash(cmd));\n+\tentry->cmd = cmd;\n+\tentry->supported_capabilities = 0;\n+\tprocess = &entry->process;\n+\n+\tchild_process_init(process);\n+\tprocess->argv = argv;\n+\tprocess->use_shell = 1;\n+\tprocess->in = -1;\n+\tprocess->out = -1;\n+\n+\tif (start_command(process)) {\n+\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n+\t\tkill_multi_file_filter(hashmap, entry);\n+\t\treturn NULL;\n+\t}\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = packet_write_list(process->in, \"git-filter-client\", \"version=2\", NULL);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = strcmp(packet_read_line(process->out, NULL), \"git-filter-server\");\n+\tif (err) {\n+\t\terror(\"external filter '%s' does not support long running filter protocol\", cmd);\n+\t\tgoto done;\n+\t}\n+\terr = strcmp(packet_read_line(process->out, NULL), \"version=2\");\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_write_list(process->in, \"clean=true\", \"smudge=true\", NULL);\n+\n+\tfor (;;) {\n+\t\tcap_buf = packet_read_line(process->out, NULL);\n+\t\tif (!cap_buf)\n+\t\t\tbreak;\n+\t\tstring_list_split_in_place(&cap_list, cap_buf, '=', 1);\n+\n+\t\tif (cap_list.nr != 2 || strcmp(cap_list.items[1].string, \"true\"))\n+\t\t\tcontinue;\n+\n+\t\tcap_name = cap_list.items[0].string;\n+\t\tif (!strcmp(cap_name, \"clean\")) {\n+\t\t\tentry->supported_capabilities |= CAP_CLEAN;\n+\t\t} else if (!strcmp(cap_name, \"smudge\")) {\n+\t\t\tentry->supported_capabilities |= CAP_SMUDGE;\n+\t\t} else {\n+\t\t\twarning(\n+\t\t\t\t\"external filter '%s' requested unsupported filter capability '%s'\",\n+\t\t\t\tcmd, cap_name\n+\t\t\t);\n+\t\t}\n+\n+\t\tstring_list_clear(&cap_list, 0);\n+\t}\n+\n+done:\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (err || errno == EPIPE) {\n+\t\terror(\"initialization for external filter '%s' failed\", cmd);\n+\t\tkill_multi_file_filter(hashmap, entry);\n+\t\treturn NULL;\n+\t}\n+\n+\thashmap_add(hashmap, entry);\n+\treturn entry;\n+}\n+\n+static void read_multi_file_filter_values(int fd, struct strbuf *status) {\n+\tstruct strbuf **pair;\n+\tchar *line;\n+\tfor (;;) {\n+\t\tline = packet_read_line(fd, NULL);\n+\t\tif (!line)\n+\t\t\tbreak;\n+\t\tpair = strbuf_split_str(line, '=', 2);\n+\t\tif (pair[0] && pair[0]->len && pair[1]) {\n+\t\t\tif (!strcmp(pair[0]->buf, \"status=\")) {\n+\t\t\t\tstrbuf_reset(status);\n+\t\t\t\tstrbuf_addbuf(status, pair[1]);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n+static int apply_multi_file_filter(const char *path, const char *src, size_t len,\n+ int fd, struct strbuf *dst, const char *cmd,\n+ const unsigned int wanted_capability)\n+{\n+\tint err;\n+\tstruct cmd2process *entry;\n+\tstruct child_process *process;\n+\tstruct stat file_stat;\n+\tstruct strbuf nbuf = STRBUF_INIT;\n+\tstruct strbuf filter_status = STRBUF_INIT;\n+\tchar *filter_type;\n+\n+\tif (!cmd_process_map_initialized) {\n+\t\tcmd_process_map_initialized = 1;\n+\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n+\t\tentry = NULL;\n+\t} else {\n+\t\tentry = find_multi_file_filter_entry(&cmd_process_map, cmd);\n+\t}\n+\n+\tfflush(NULL);\n+\n+\tif (!entry) {\n+\t\tentry = start_multi_file_filter(&cmd_process_map, cmd);\n+\t\tif (!entry)\n+\t\t\treturn 0;\n+\t}\n+\tprocess = &entry->process;\n+\n+\tif (!(wanted_capability & entry->supported_capabilities))\n+\t\treturn 0;\n+\n+\tif (CAP_CLEAN & wanted_capability)\n+\t\tfilter_type = \"clean\";\n+\telse if (CAP_SMUDGE & wanted_capability)\n+\t\tfilter_type = \"smudge\";\n+\telse\n+\t\tdie(\"unexpected filter type\");\n+\n+\tif (fd >= 0 && !src) {\n+\t\tif (fstat(fd, &file_stat) == -1)\n+\t\t\treturn 0;\n+\t\tlen = xsize_t(file_stat.st_size);\n+\t}\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = strlen(filter_type) > PKTLINE_DATA_MAXLEN;\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_write_fmt_gently(process->in, \"command=%s\\n\", filter_type);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = strlen(path) > PKTLINE_DATA_MAXLEN;\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_write_fmt_gently(process->in, \"pathname=%s\\n\", path);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (fd >= 0)\n+\t\terr = write_packetized_from_fd(fd, process->in);\n+\telse\n+\t\terr = write_packetized_from_buf(src, len, process->in);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tread_multi_file_filter_values(process->out, &filter_status);\n+\terr = strcmp(filter_status.buf, \"success\");\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = read_packetized_to_buf(process->out, &nbuf) < 0;\n+\tif (err)\n+\t\tgoto done;\n+\n+\tread_multi_file_filter_values(process->out, &filter_status);\n+\terr = strcmp(filter_status.buf, \"success\");\n+\n+done:\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (err || errno == EPIPE) {\n+\t\tif (!strcmp(filter_status.buf, \"error\")) {\n+\t\t\t/* The filter signaled a problem with the file. */\n+\t\t} else if (!strcmp(filter_status.buf, \"abort\")) {\n+\t\t\t/*\n+\t\t\t * The filter signaled a permanent problem. Don't try to filter\n+\t\t\t * files with the same command for the lifetime of the current\n+\t\t\t * Git process.\n+\t\t\t */\n+\t\t\t entry->supported_capabilities &= ~wanted_capability;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * Something went wrong with the protocol filter.\n+\t\t\t * Force shutdown and restart if another blob requires filtering!\n+\t\t\t */\n+\t\t\terror(\"external filter '%s' failed\", cmd);\n+\t\t\tkill_multi_file_filter(&cmd_process_map, entry);\n+\t\t}\n+\t} else {\n+\t\tstrbuf_swap(dst, &nbuf);\n+\t}\n+\tstrbuf_release(&nbuf);\n+\treturn !err;\n+}\n+\n static struct convert_driver {\n \tconst char *name;\n \tstruct convert_driver *next;\n \tconst char *smudge;\n \tconst char *clean;\n+\tconst char *process;\n \tint required;\n } *user_convert, **user_convert_tail;\n \n+static int apply_filter(const char *path, const char *src, size_t len,\n+ int fd, struct strbuf *dst, struct convert_driver *drv,\n+ const unsigned int wanted_capability)\n+{\n+\tconst char *cmd = NULL;\n+\n+\tif (!drv)\n+\t\treturn 0;\n+\n+\tif (!dst)\n+\t\treturn 1;\n+\n+\tif (!drv->process && (CAP_CLEAN & wanted_capability) && drv->clean)\n+\t\tcmd = drv->clean;\n+\telse if (!drv->process && (CAP_SMUDGE & wanted_capability) && drv->smudge)\n+\t\tcmd = drv->smudge;\n+\n+\tif (cmd && *cmd)\n+\t\treturn apply_single_file_filter(path, src, len, fd, dst, cmd);\n+\telse if (drv->process && *drv->process)\n+\t\treturn apply_multi_file_filter(path, src, len, fd, dst, drv->process, wanted_capability);\n+\n+\treturn 0;\n+}\n+\n static int read_convert_config(const char *var, const char *value, void *cb)\n {\n \tconst char *key, *name;\n@@ -538,6 +836,9 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(\"clean\", key))\n \t\treturn git_config_string(&drv->clean, var, value);\n \n+\tif (!strcmp(\"process\", key))\n+\t\treturn git_config_string(&drv->process, var, value);\n+\n \tif (!strcmp(\"required\", key)) {\n \t\tdrv->required = git_config_bool(var, value);\n \t\treturn 0;\n@@ -839,7 +1140,7 @@ int would_convert_to_git_filter_fd(const char *path)\n \tif (!ca.drv->required)\n \t\treturn 0;\n \n-\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n+\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN);\n }\n \n const char *get_convert_attr_ascii(const char *path)\n@@ -872,18 +1173,12 @@ int convert_to_git(const char *path, const char *src, size_t len,\n struct strbuf *dst, enum safe_crlf checksafe)\n {\n \tint ret = 0;\n-\tconst char *filter = NULL;\n-\tint required = 0;\n \tstruct conv_attrs ca;\n \n \tconvert_attrs(&ca, path);\n-\tif (ca.drv) {\n-\t\tfilter = ca.drv->clean;\n-\t\trequired = ca.drv->required;\n-\t}\n \n-\tret |= apply_filter(path, src, len, -1, dst, filter);\n-\tif (!ret && required)\n+\tret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN);\n+\tif (!ret && ca.drv && ca.drv->required)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n \tif (ret && dst) {\n@@ -905,9 +1200,9 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n \tconvert_attrs(&ca, path);\n \n \tassert(ca.drv);\n-\tassert(ca.drv->clean);\n+\tassert(ca.drv->clean || ca.drv->process);\n \n-\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))\n+\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n@@ -919,15 +1214,9 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t\t\t\t int normalizing)\n {\n \tint ret = 0, ret_filter = 0;\n-\tconst char *filter = NULL;\n-\tint required = 0;\n \tstruct conv_attrs ca;\n \n \tconvert_attrs(&ca, path);\n-\tif (ca.drv) {\n-\t\tfilter = ca.drv->smudge;\n-\t\trequired = ca.drv->required;\n-\t}\n \n \tret |= ident_to_worktree(path, src, len, dst, ca.ident);\n \tif (ret) {\n@@ -936,9 +1225,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t}\n \t/*\n \t * CRLF conversion can be skipped if normalizing, unless there\n-\t * is a smudge filter. The filter might expect CRLFs.\n+\t * is a smudge or process filter (even if the process filter doesn't\n+\t * support smudge). The filters might expect CRLFs.\n \t */\n-\tif (filter || !normalizing) {\n+\tif ((ca.drv && (ca.drv->smudge || ca.drv->process)) || !normalizing) {\n \t\tret |= crlf_to_worktree(path, src, len, dst, ca.crlf_action);\n \t\tif (ret) {\n \t\t\tsrc = dst->buf;\n@@ -946,8 +1236,8 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n-\tret_filter = apply_filter(path, src, len, -1, dst, filter);\n-\tif (!ret_filter && required)\n+\tret_filter = apply_filter(path, src, len, -1, dst, ca.drv, CAP_SMUDGE);\n+\tif (!ret_filter && ca.drv && ca.drv->required)\n \t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n \n \treturn ret | ret_filter;\n@@ -1399,7 +1689,7 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n \tstruct stream_filter *filter = NULL;\n \n \tconvert_attrs(&ca, path);\n-\tif (ca.drv && (ca.drv->smudge || ca.drv->clean))\n+\tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n \t\treturn NULL;\n \n \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 6df8449..3d873f3 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -86,6 +86,7 @@ ssize_t read_packetized_to_buf(int fd_in, struct strbuf *sb_out);\n \n #define DEFAULT_PACKET_MAX 1000\n #define LARGE_PACKET_MAX 65520\n+#define PKTLINE_DATA_MAXLEN (LARGE_PACKET_MAX - 4)\n extern char packet_buffer[LARGE_PACKET_MAX];\n \n #endif\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex dc50938..210c4f6 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -31,7 +31,10 @@ test_expect_success setup '\n \tcat test >test.i &&\n \tgit add test test.t test.i &&\n \trm -f test test.t test.i &&\n-\tgit checkout -- test test.t test.i\n+\tgit checkout -- test test.t test.i &&\n+\n+\techo \"content-test2\" >test2.o &&\n+\techo \"content-test3 - subdir\" >\"test3 - subdir.o\"\n '\n \n script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n@@ -279,4 +282,364 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '\n \ttest_line_count = 0 count\n '\n \n+check_filter () {\n+\trm -f rot13-filter.log actual.log &&\n+\t\"$@\" 2> git_stderr.log &&\n+\ttest_must_be_empty git_stderr.log &&\n+\tcat >expected.log &&\n+\tsort rot13-filter.log | uniq -c | sed \"s/^[ ]*//\" >actual.log &&\n+\ttest_cmp expected.log actual.log\n+}\n+\n+check_filter_count_clean () {\n+\trm -f rot13-filter.log actual.log &&\n+\t\"$@\" 2> git_stderr.log &&\n+\ttest_must_be_empty git_stderr.log &&\n+\tcat >expected.log &&\n+\tsort rot13-filter.log | uniq -c | sed \"s/^[ ]*//\" |\n+\t\tsed \"s/^\\([0-9]\\) IN: clean/x IN: clean/\" >actual.log &&\n+\ttest_cmp expected.log actual.log\n+}\n+\n+check_filter_ignore_clean () {\n+\trm -f rot13-filter.log actual.log &&\n+\t\"$@\" &&\n+\tcat >expected.log &&\n+\tgrep -v \"IN: clean\" rot13-filter.log >actual.log &&\n+\ttest_cmp expected.log actual.log\n+}\n+\n+check_filter_no_call () {\n+\trm -f rot13-filter.log &&\n+\t\"$@\" 2> git_stderr.log &&\n+\ttest_must_be_empty git_stderr.log &&\n+\ttest_must_be_empty rot13-filter.log\n+}\n+\n+check_rot13 () {\n+\ttest_cmp \"$1\" \"$2\" &&\n+\t./../rot13.sh <\"$1\" >expected &&\n+\tgit cat-file blob :\"$2\" >actual &&\n+\ttest_cmp expected actual\n+}\n+\n+test_expect_success PERL 'required process filter should filter data' '\n+\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\tgit branch empty &&\n+\n+\t\tcp ../test.o test.r &&\n+\t\tcp ../test2.o test2.r &&\n+\t\tmkdir testsubdir &&\n+\t\tcp \"../test3 - subdir.o\" \"testsubdir/test3 - subdir.r\" &&\n+\t\t>test4-empty.r &&\n+\n+\t\tcheck_filter \\\n+\t\t\tgit add . \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n+\t\t\t\t\t1 IN: clean test2.r 14 [OK] -- OUT: 14 . [OK]\n+\t\t\t\t\t1 IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]\n+\t\t\t\t\t1 IN: clean testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n+\t\t\t\t\t1 START\n+\t\t\t\t\t1 STOP\n+\t\t\t\t\t1 wrote filter header\n+\t\t\t\tEOF\n+\n+\t\tcheck_filter_count_clean \\\n+\t\t\tgit commit . -m \"test commit\" \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\tx IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n+\t\t\t\t\tx IN: clean test2.r 14 [OK] -- OUT: 14 . [OK]\n+\t\t\t\t\tx IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]\n+\t\t\t\t\tx IN: clean testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n+\t\t\t\t\t1 START\n+\t\t\t\t\t1 STOP\n+\t\t\t\t\t1 wrote filter header\n+\t\t\t\tEOF\n+\n+\t\trm -f test?.r \"testsubdir/test3 - subdir.r\" &&\n+\n+\t\tcheck_filter_ignore_clean \\\n+\t\t\tgit checkout . \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\tSTART\n+\t\t\t\t\twrote filter header\n+\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n+\t\t\t\t\tIN: smudge testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n+\t\t\t\t\tSTOP\n+\t\t\t\tEOF\n+\n+\t\tcheck_filter_ignore_clean \\\n+\t\t\tgit checkout empty \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\tSTART\n+\t\t\t\t\twrote filter header\n+\t\t\t\t\tSTOP\n+\t\t\t\tEOF\n+\n+\t\tcheck_filter_ignore_clean \\\n+\t\t\tgit checkout master \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\tSTART\n+\t\t\t\t\twrote filter header\n+\t\t\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n+\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n+\t\t\t\t\tIN: smudge test4-empty.r 0 [OK] -- OUT: 0 [OK]\n+\t\t\t\t\tIN: smudge testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n+\t\t\t\t\tSTOP\n+\t\t\t\tEOF\n+\n+\t\tcheck_rot13 ../test.o test.r &&\n+\t\tcheck_rot13 ../test2.o test2.r &&\n+\t\tcheck_rot13 \"../test3 - subdir.o\" \"testsubdir/test3 - subdir.r\"\n+\t)\n+'\n+\n+test_expect_success PERL 'required process filter should clean only and take precedence' '\n+\ttest_config_global filter.protocol.clean ./../rot13.sh &&\n+\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean\" &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\tgit branch empty &&\n+\n+\t\tcp ../test.o test.r &&\n+\n+\t\tcheck_filter \\\n+\t\t\tgit add . \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n+\t\t\t\t\t1 START\n+\t\t\t\t\t1 STOP\n+\t\t\t\t\t1 wrote filter header\n+\t\t\t\tEOF\n+\n+\t\tcheck_filter_count_clean \\\n+\t\t\tgit commit . -m \"test commit\" \\\n+\t\t\t\t<<-\\EOF\n+\t\t\t\t\tx IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n+\t\t\t\t\t1 START\n+\t\t\t\t\t1 STOP\n+\t\t\t\t\t1 wrote filter header\n+\t\t\t\tEOF\n+\t)\n+'\n+\n+generate_test_data () {\n+\tLEN=$1\n+\tNAME=$2\n+\ttest-genrandom end $LEN |\n+\t\tperl -pe \"s/./chr((ord($&) % 26) + 97)/sge\" >../$NAME.file &&\n+\tcp ../$NAME.file . &&\n+\t./../rot13.sh <../$NAME.file >../$NAME.file.rot13\n+}\n+\n+test_expect_success PERL 'required process filter should process multiple packets' '\n+\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.required true &&\n+\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\t# Generate data that requires 3 packets\n+\t\tPKTLINE_DATA_MAXLEN=65516 &&\n+\n+\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN )) 1pkt_1__ &&\n+\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN + 1)) 2pkt_1+1 &&\n+\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN * 2 - 1)) 2pkt_2-1 &&\n+\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN * 2 )) 2pkt_2__ &&\n+\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN * 2 + 1)) 3pkt_2+1 &&\n+\n+\t\techo \"*.file filter=protocol\" >.gitattributes &&\n+\t\tcheck_filter \\\n+\t\t\tgit add *.file .gitattributes \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\t1 IN: clean 1pkt_1__.file 65516 [OK] -- OUT: 65516 . [OK]\n+\t\t\t\t\t1 IN: clean 2pkt_1+1.file 65517 [OK] -- OUT: 65517 .. [OK]\n+\t\t\t\t\t1 IN: clean 2pkt_2-1.file 131031 [OK] -- OUT: 131031 .. [OK]\n+\t\t\t\t\t1 IN: clean 2pkt_2__.file 131032 [OK] -- OUT: 131032 .. [OK]\n+\t\t\t\t\t1 IN: clean 3pkt_2+1.file 131033 [OK] -- OUT: 131033 ... [OK]\n+\t\t\t\t\t1 START\n+\t\t\t\t\t1 STOP\n+\t\t\t\t\t1 wrote filter header\n+\t\t\t\tEOF\n+\t\tgit commit . -m \"test commit\" &&\n+\n+\t\trm -f *.file &&\n+\t\tgit checkout -- *.file &&\n+\n+\t\tfor f in *.file\n+\t\tdo\n+\t\t\tgit cat-file blob :$f >actual &&\n+\t\t\ttest_cmp ../$f.rot13 actual\n+\t\tdone\n+\t)\n+'\n+\n+test_expect_success PERL 'required process filter should with clean error should fail' '\n+\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcp ../test.o test.r &&\n+\t\techo \"this is going to fail\" >clean-write-fail.r &&\n+\t\techo \"content-test3-subdir\" >test3.r &&\n+\n+\t\t# Note: There are three clean paths in convert.c we just test one here.\n+\t\ttest_must_fail git add .\n+\t)\n+'\n+\n+test_expect_success PERL 'process filter should restart after unexpected write failure' '\n+\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcp ../test.o test.r &&\n+\t\tcp ../test2.o test2.r &&\n+\t\techo \"this is going to fail\" >smudge-write-fail.o &&\n+\t\tcat smudge-write-fail.o >smudge-write-fail.r &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\trm -f *.r &&\n+\n+\t\tcheck_filter_ignore_clean \\\n+\t\t\tgit checkout . \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\tSTART\n+\t\t\t\t\twrote filter header\n+\t\t\t\t\tIN: smudge smudge-write-fail.r 22 [OK] -- OUT: 22 [WRITE FAIL]\n+\t\t\t\t\tSTART\n+\t\t\t\t\twrote filter header\n+\t\t\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n+\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n+\t\t\t\t\tSTOP\n+\t\t\t\tEOF\n+\n+\t\tcheck_rot13 ../test.o test.r &&\n+\t\tcheck_rot13 ../test2.o test2.r &&\n+\n+\t\t! test_cmp smudge-write-fail.o smudge-write-fail.r && # Smudge failed!\n+\t\t./../rot13.sh expected &&\n+\t\tgit cat-file blob :smudge-write-fail.r >actual &&\n+\t\ttest_cmp expected actual\t\t\t\t\t\t\t # Clean worked!\n+\t)\n+'\n+\n+test_expect_success PERL 'process filter should not restart in case of an error' '\n+\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcp ../test.o test.r &&\n+\t\tcp ../test2.o test2.r &&\n+\t\techo \"this will cause an error\" >error.o &&\n+\t\tcp error.o error.r &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\trm -f *.r &&\n+\n+\t\tcheck_filter_ignore_clean \\\n+\t\t\tgit checkout . \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\tSTART\n+\t\t\t\t\twrote filter header\n+\t\t\t\t\tIN: smudge error.r 25 [OK] -- OUT: 0 [ERROR]\n+\t\t\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n+\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n+\t\t\t\t\tSTOP\n+\t\t\t\tEOF\n+\n+\t\tcheck_rot13 ../test.o test.r &&\n+\t\tcheck_rot13 ../test2.o test2.r &&\n+\t\ttest_cmp error.o error.r\n+\t)\n+'\n+\n+test_expect_success PERL 'process filter should be able to signal an error for all future files' '\n+\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcp ../test.o test.r &&\n+\t\tcp ../test2.o test2.r &&\n+\t\techo \"error this blob and all future blobs\" >abort.o &&\n+\t\tcp abort.o abort.r &&\n+\t\tgit add . &&\n+\t\tgit commit . -m \"test commit\" &&\n+\t\trm -f *.r &&\n+\n+\t\tcheck_filter_ignore_clean \\\n+\t\t\tgit checkout . \\\n+\t\t\t\t<<-\\EOF &&\n+\t\t\t\t\tSTART\n+\t\t\t\t\twrote filter header\n+\t\t\t\t\tIN: smudge abort.r 37 [OK] -- OUT: 0 [ABORT]\n+\t\t\t\t\tSTOP\n+\t\t\t\tEOF\n+\n+\t\ttest_cmp ../test.o test.r &&\n+\t\ttest_cmp ../test2.o test2.r &&\n+\t\ttest_cmp abort.o abort.r\n+\t)\n+'\n+\n+test_expect_success PERL 'invalid process filter must fail (and not hang!)' '\n+\ttest_config_global filter.protocol.process cat &&\n+\ttest_config_global filter.protocol.required true &&\n+\trm -rf repo &&\n+\tmkdir repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit init &&\n+\n+\t\techo \"*.r filter=protocol\" >.gitattributes &&\n+\n+\t\tcp ../test.o test.r &&\n+\t\ttest_must_fail git add . 2> git_stderr.log &&\n+\t\tgrep \"not support long running filter protocol\" git_stderr.log\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nnew file mode 100755\nindex 0000000..8958f71\n--- /dev/null\n+++ b/t/t0021/rot13-filter.pl\n@@ -0,0 +1,191 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git filter protocol version 2\n+# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n+#\n+# The script takes the list of supported protocol capabilities as\n+# arguments (\"clean\", \"smudge\", etc).\n+#\n+# This implementation supports special test cases:\n+# (1) If data with the pathname \"clean-write-fail.r\" is processed with\n+# a \"clean\" operation then the write operation will die.\n+# (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n+# a \"smudge\" operation then the write operation will die.\n+# (3) If data with the pathname \"error.r\" is processed with any\n+# operation then the filter signals that it cannot or does not want\n+# to process the file.\n+# (4) If data with the pathname \"abort.r\" is processed with any\n+# operation then the filter signals that it cannot or does not want\n+# to process the file and any file after that is processed with the\n+# same command.\n+#\n+\n+use strict;\n+use warnings;\n+\n+my $MAX_PACKET_CONTENT_SIZE = 65516;\n+my @capabilities = @ARGV;\n+\n+open my $debug, \">>\", \"rot13-filter.log\";\n+\n+sub rot13 {\n+ my ($str) = @_;\n+ $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n+ return $str;\n+}\n+\n+sub packet_bin_read {\n+ my $buffer;\n+ my $bytes_read = read STDIN, $buffer, 4;\n+ if ( $bytes_read == 0 ) {\n+\n+ # EOF - Git stopped talking to us!\n+ print $debug \"STOP\\n\";\n+ exit();\n+ }\n+ elsif ( $bytes_read != 4 ) {\n+ die \"invalid packet size '$bytes_read' field\";\n+ }\n+ my $pkt_size = hex($buffer);\n+ if ( $pkt_size == 0 ) {\n+ return ( 1, \"\" );\n+ }\n+ elsif ( $pkt_size > 4 ) {\n+ my $content_size = $pkt_size - 4;\n+ $bytes_read = read STDIN, $buffer, $content_size;\n+ if ( $bytes_read != $content_size ) {\n+ die \"invalid packet ($content_size expected; $bytes_read read)\";\n+ }\n+ return ( 0, $buffer );\n+ }\n+ else {\n+ die \"invalid packet size\";\n+ }\n+}\n+\n+sub packet_txt_read {\n+ my ( $res, $buf ) = packet_bin_read();\n+ unless ( $buf =~ /\\n$/ ) {\n+ die \"A non-binary line SHOULD BE terminated by an LF.\";\n+ }\n+ return ( $res, substr( $buf, 0, -1 ) );\n+}\n+\n+sub packet_bin_write {\n+ my ($packet) = @_;\n+ print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n+ print STDOUT $packet;\n+ STDOUT->flush();\n+}\n+\n+sub packet_txt_write {\n+ packet_bin_write( $_[0] . \"\\n\" );\n+}\n+\n+sub packet_flush {\n+ print STDOUT sprintf( \"%04x\", 0 );\n+ STDOUT->flush();\n+}\n+\n+print $debug \"START\\n\";\n+$debug->flush();\n+\n+( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n+( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n+( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n+\n+packet_txt_write(\"git-filter-server\");\n+packet_txt_write(\"version=2\");\n+\n+( packet_txt_read() eq ( 0, \"clean=true\" ) ) || die \"bad capability\";\n+( packet_txt_read() eq ( 0, \"smudge=true\" ) ) || die \"bad capability\";\n+( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n+\n+foreach (@capabilities) {\n+ packet_txt_write( $_ . \"=true\" );\n+}\n+packet_flush();\n+print $debug \"wrote filter header\\n\";\n+$debug->flush();\n+\n+while (1) {\n+ my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n+ print $debug \"IN: $command\";\n+ $debug->flush();\n+\n+ my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;\n+ print $debug \" $pathname\";\n+ $debug->flush();\n+\n+ # Flush\n+ packet_bin_read();\n+\n+ my $input = \"\";\n+ {\n+ binmode(STDIN);\n+ my $buffer;\n+ my $done = 0;\n+ while ( !$done ) {\n+ ( $done, $buffer ) = packet_bin_read();\n+ $input .= $buffer;\n+ }\n+ print $debug \" \" . length($input) . \" [OK] -- \";\n+ $debug->flush();\n+ }\n+\n+ my $output;\n+ if ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n+ $output = \"\";\n+ }\n+ elsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n+ $output = rot13($input);\n+ }\n+ elsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n+ $output = rot13($input);\n+ }\n+ else {\n+ die \"bad command '$command'\";\n+ }\n+\n+ print $debug \"OUT: \" . length($output) . \" \";\n+ $debug->flush();\n+\n+ if ( $pathname eq \"error.r\" ) {\n+ print $debug \"[ERROR]\\n\";\n+ $debug->flush();\n+ packet_txt_write(\"status=error\");\n+ packet_flush();\n+ }\n+ elsif ( $pathname eq \"abort.r\" ) {\n+ print $debug \"[ABORT]\\n\";\n+ $debug->flush();\n+ packet_txt_write(\"status=abort\");\n+ packet_flush();\n+ }\n+ else {\n+ packet_txt_write(\"status=success\");\n+ packet_flush();\n+\n+ if ( $pathname eq \"${command}-write-fail.r\" ) {\n+ print $debug \"[WRITE FAIL]\\n\";\n+ $debug->flush();\n+ die \"${command} write error\";\n+ }\n+\n+ while ( length($output) > 0 ) {\n+ my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n+ packet_bin_write($packet);\n+ print $debug \".\";\n+ if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n+ $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n+ }\n+ else {\n+ $output = \"\";\n+ }\n+ }\n+ packet_flush();\n+ print $debug \" [OK]\\n\";\n+ $debug->flush();\n+ packet_flush();\n+ }\n+}\n-- \n2.10.0\n\n"},{"id":"302488","messageId":"592e18b7-ac2e-ed23-0321-57eb418d5172@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-2-larsxschneider@gmail.com","subject":"Re: [PATCH v8 01/11] pkt-line: rename packet_write() to packet_write_fmt()","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-24T21:14:36Z","receivedAt":"2016-09-24T21:15:04Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Hello Lars,\n\nW dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n\n> From: Lars Schneider \n> \n> packet_write() should be called packet_write_fmt() as the string\n> parameter can be formatted.\n\nI would say:\n\n packet_write() should be called packet_write_fmt() because it\n is printf-like function where first parameter is format string.\n \nOr something like that. But such minor change might be not worth\nyet another reroll of this patch series.\n\nPerhaps it would be a good idea to explain the reasoning behind\nthis change:\n\n This is important distinction to know from the name if the\n function accepts arbitrary binary data and/or arbitrary\n strings to be written - packet_write[_fmt()] do not.\n\n> \n> Suggested-by: Junio C Hamano \n\nJust so nobody wonders later why this patch was needed/suggested.\n\n> Signed-off-by: Lars Schneider \n> ---\n> builtin/archive.c | 4 ++--\n> builtin/receive-pack.c | 4 ++--\n> builtin/remote-ext.c | 4 ++--\n> builtin/upload-archive.c | 4 ++--\n> connect.c | 2 +-\n> daemon.c | 2 +-\n> http-backend.c | 2 +-\n> pkt-line.c | 2 +-\n\nThe header of the renamed function looks now very nice:\n\n void packet_write_fmt(int fd, const char *fmt, ...)\n ^^^ ^^^\n\n> pkt-line.h | 2 +-\n> shallow.c | 2 +-\n> upload-pack.c | 30 +++++++++++++++---------------\n> 11 files changed, 29 insertions(+), 29 deletions(-)\n\nDiffstat looks correct. Was the patch generated by doing search\nand replace?\n\nBest,\n-- \nJakub Narębski\n"},{"id":"302489","messageId":"a5b9a46d-592e-089d-d156-36c4d61902c9@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-3-larsxschneider@gmail.com","subject":"Re: [PATCH v8 02/11] pkt-line: extract set_packet_header()","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-24T21:22:21Z","receivedAt":"2016-09-24T21:22:48Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n\n> From: Lars Schneider \n>\n> Subject: [PATCH v8 02/11] pkt-line: extract set_packet_header()\n> \n> set_packet_header() converts an integer to a 4 byte hex string. Make\n> this function locally available so that other pkt-line functions can\n> use it.\n\nAh. I have trouble understanding this commit message, as the\nset_packet_header() was not available before this patch, but it\nis good if one reads it together with commit summary / title.\n\nWriting\n\n Extracted set_packet_header() function converts...\n\nor\n\n New set_packet_header() function converts... \n\nwould make it more clear, but it is all right as it is now.\nPerhaps also\n\n ... could use it.\n\nas currently no other pkt-line function but the one set_packet_header()\nwas extracted from, namely format_packet(), uses it.\n\nBut that is just nitpicking; no need to change on that account.\n\n> \n> Signed-off-by: Lars Schneider \n\n"},{"id":"302490","messageId":"854ff387-57a4-4c27-4c27-b834f7797694@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-4-larsxschneider@gmail.com","subject":"Re: [PATCH v8 03/11] run-command: move check_pipe() from write_or_die to run_command","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-24T22:12:45Z","receivedAt":"2016-09-24T22:13:14Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> Move check_pipe() to run_command and make it public. This is necessary\n> to call the function from pkt-line in a subsequent patch.\n\nAll right.\n\n> \n> Signed-off-by: Lars Schneider \n> ---\n> run-command.c | 13 +++++++++++++\n> run-command.h | 2 ++\n> write_or_die.c | 13 -------------\n> 3 files changed, 15 insertions(+), 13 deletions(-)\n\nDiffstat looks correct.\n\nNot to add to your burden, but perhaps somebody could add to his/her\nTODO documenting check_pipe() in Documentation/technical/api-run-command.txt\nOr is it not worth it?\n\nBest regards,\n-- \nJakub Narębski\n\n"},{"id":"302491","messageId":"23f1ba98-c045-d514-ac99-5cbb2727bfb9@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-5-larsxschneider@gmail.com","subject":"Re: [PATCH v8 04/11] pkt-line: add packet_write_fmt_gently()","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-24T22:27:46Z","receivedAt":"2016-09-24T22:29:02Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> packet_write_fmt() would die in case of a write error even though for\n> some callers an error would be acceptable. Add packet_write_fmt_gently()\n> which writes a formatted pkt-line like packet_write_fmt() but does not\n> die in case of an error. The function is used in a subsequent patch.\n\nLooks good.\n\n> \n> Signed-off-by: Lars Schneider \n> ---\n> pkt-line.c | 34 ++++++++++++++++++++++++++++++----\n> pkt-line.h | 1 +\n> 2 files changed, 31 insertions(+), 4 deletions(-)\n\n"},{"id":"302496","messageId":"0763ff86-9593-ff58-6d9d-1ff920bf1071@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-6-larsxschneider@gmail.com","subject":"Re: [PATCH v8 05/11] pkt-line: add packet_flush_gently()","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-24T22:56:29Z","receivedAt":"2016-09-24T22:56:57Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> packet_flush() would die in case of a write error even though for some\n> callers an error would be acceptable. Add packet_flush_gently() which\n> writes a pkt-line flush packet like packet_flush() but does not die in\n> case of an error. The function is used in a subsequent patch.\n> \n> Signed-off-by: Lars Schneider \n\nLooks good.\n\nI guess the difference in treatment from packet_write_fmt_gently() in\nthe previous patch is that there isn't anything to extract to form\na common code function... I have skipped a few iterations of this series.\n\n> ---\n> pkt-line.c | 8 ++++++++\n> pkt-line.h | 1 +\n> 2 files changed, 9 insertions(+)\n> \n> diff --git a/pkt-line.c b/pkt-line.c\n> index 3b465fd..19f0271 100644\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -91,6 +91,14 @@ void packet_flush(int fd)\n> \twrite_or_die(fd, \"0000\", 4);\n> }\n> \n> +int packet_flush_gently(int fd)\n> +{\n> +\tpacket_trace(\"0000\", 4, 1);\n> +\tif (write_in_full(fd, \"0000\", 4) == 4)\n> +\t\treturn 0;\n> +\treturn error(\"flush packet write failed\");\n> +}\n> +\n> void packet_buf_flush(struct strbuf *buf)\n> {\n> \tpacket_trace(\"0000\", 4, 1);\n> diff --git a/pkt-line.h b/pkt-line.h\n> index 3caea77..3fa0899 100644\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -23,6 +23,7 @@ void packet_flush(int fd);\n> void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n> void packet_buf_flush(struct strbuf *buf);\n> void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n> +int packet_flush_gently(int fd);\n> int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n> \n> /*\n> \n\n"},{"id":"302513","messageId":"55c5a86e-4055-f36d-bbf1-e8bae919088a@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-7-larsxschneider@gmail.com","subject":"Re: [PATCH v8 06/11] pkt-line: add packet_write_gently()","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-25T11:26:42Z","receivedAt":"2016-09-25T11:26:52Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> packet_write_fmt_gently() uses format_packet() which lets the caller\n> only send string data via \"%s\". That means it cannot be used for\n> arbitrary data that may contain NULs.\n> \n> Add packet_write_gently() which writes arbitrary data and does not die\n> in case of an error. The function is used by other pkt-line functions in\n> a subsequent patch.\n\nNice; obviously needed for sending binary data.\n\nI wonder how send-pack / receive-pack handles sending binary files.\nThough this is outside of scope of this patch series, it is something\nto think about for later.\n\n> \n> Signed-off-by: Lars Schneider \n> ---\n> pkt-line.c | 16 ++++++++++++++++\n> 1 file changed, 16 insertions(+)\n> \n> diff --git a/pkt-line.c b/pkt-line.c\n> index 19f0271..fc0ac12 100644\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -171,6 +171,22 @@ int packet_write_fmt_gently(int fd, const char *fmt, ...)\n> \treturn status;\n> }\n> \n> +static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n\nI'm not sure what naming convention the rest of Git uses, but isn't\nit more like '*data' rather than '*buf' here?\n\n> +{\n> +\tstatic char packet_write_buffer[LARGE_PACKET_MAX];\n\nI think there should be warning (as a comment before function\ndeclaration, or before function definition), that packet_write_gently()\nis not thread-safe (nor reentrant, but the latter does not matter here,\nI think).\n\nThread-safe vs reentrant: http://stackoverflow.com/a/33445858/46058\n\nThis is not something terribly important; I guess git code has tons\nof functions not marked as thread-unsafe...\n\n> +\n> +\tif (size > sizeof(packet_write_buffer) - 4) {\n\nFirst, wouldn't the following be more readable:\n\n +\tif (size + 4 > LARGE_PACKET_MAX) {\n\n> +\t\treturn error(\"packet write failed - data exceeds max packet size\");\n> +\t}\n\nSecond, CodingGuidelines is against using braces (blocks) for one\nline conditionals: \"We avoid using braces unnecessarily.\"\n\nBut this is just me nitpicking.\n\n> +\tpacket_trace(buf, size, 1);\n> +\tsize += 4;\n> +\tset_packet_header(packet_write_buffer, size);\n> +\tmemcpy(packet_write_buffer + 4, buf, size - 4);\n> +\tif (write_in_full(fd_out, packet_write_buffer, size) == size)\n\nHmmm... in some places we use original size, in others (original) size + 4;\nperhaps it would be more readable to add a new local temporary variable\n\n\tsize_t full_size = size + 4;\n\nOr perhaps use 'data_size' and 'packet_size', where 'packet_size = data_size + 4'.\nBut that might be too chatty for variable names ;-)\n\n> +\t\treturn 0;\n> +\treturn error(\"packet write failed\");\n> +}\n\nCompared to previous iterations, where there were two versions\nof this function, IIRC sharing no common code: one taking buffer\nwhich had to be with place for packet size info, one with a separate\nlocal buffer for packet size only and using two writes. This\nversion uses static buffer (thus not thread-safe, I think; and\nnot reentrant), and memcpy.\n\nAnyway, if reentrant / thread-safe version would be required,\nor not doing memcpy turns out to be important with respect to\nperformance, we can provide with the *_r version:\n\n static int packet_write_gently_r(const int fd_out, const char *data, size_t size,\n char *restrict buf)\n\nWe can check if 'buf + 4 == data', and if it is, we can skip\nmemcpy() as an optimization.\n\nThis is something for the future, but not very important for\nhaving this patch series accepted.\n\n> +\n> void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n> {\n> \tva_list args;\n> \n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"302514","messageId":"507a29a1-99ff-c13e-cf1f-76631345a822@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-8-larsxschneider@gmail.com","subject":"Re: [PATCH v8 07/11] pkt-line: add functions to read/write flush terminated packet streams","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-25T13:46:44Z","receivedAt":"2016-09-25T13:46:56Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> write_packetized_from_fd() and write_packetized_from_buf() write a\n> stream of packets. All content packets use the maximal packet size\n> except for the last one. After the last content packet a `flush` control\n> packet is written.\n> \n> read_packetized_to_buf() reads arbitrary sized packets until it detects\n> a `flush` packet.\n\nI guess that read_packetized_to_fd(), for completeness, is not needed\nfor the filter protocol (though it might be useful for the receive\nside of send-pack / receive-pack).\n\nAlso, should it be read_packetized_to_strbuf()? I guess using strbuf\nto read is here because we might need more size to read in full, isn't\nit.\n\n> \n> Signed-off-by: Lars Schneider \n> ---\n> pkt-line.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> pkt-line.h | 7 +++++++\n> 2 files changed, 75 insertions(+)\n> \n> diff --git a/pkt-line.c b/pkt-line.c\n> index fc0ac12..a0a8543 100644\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -196,6 +196,47 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n> \tva_end(args);\n> }\n> \n> +int write_packetized_from_fd(int fd_in, int fd_out)\n\nI wonder if it would be worth it to name parameters in such way that\nit is known from the name which one is to be packetized, for example\nfd_out_pkt here...\n\nBut it might be not worth it; you can get it from the function name.\n\n> +{\n> +\tstatic char buf[PKTLINE_DATA_MAXLEN];\n\nStatic buffer means not thread-safe and not reentrant. It would be\nnice to have this information in a comment for this function, but\nit is not necessary.\n\nAlso, is using static variable better than using global variable\n`packet_buffer`? Well, scope for weird interactions is smaller...\n\nSidenote: we have LARGE_PACKET_MAX (used in previous patch), but\nPKTLINE_DATA_MAXLEN not LARGE_PACKET_DATA_MAX.\n\n> +\tint err = 0;\n> +\tssize_t bytes_to_write;\n> +\n> +\twhile (!err) {\n> +\t\tbytes_to_write = xread(fd_in, buf, sizeof(buf));\n> +\t\tif (bytes_to_write < 0)\n> +\t\t\treturn COPY_READ_ERROR;\n> +\t\tif (bytes_to_write == 0)\n> +\t\t\tbreak;\n> +\t\terr = packet_write_gently(fd_out, buf, bytes_to_write);\n> +\t}\n> +\tif (!err)\n> +\t\terr = packet_flush_gently(fd_out);\n> +\treturn err;\n> +}\n\nLooks good: clean and readable.\n\nSidenote (probably outside of scope of this patch): what are the\nerrors that we can get from this function, beside COPY_READ_ERROR\nof course?\n\n> +\n> +int write_packetized_from_buf(const char *src_in, size_t len, int fd_out)\n> +{\n> +\tstatic char buf[PKTLINE_DATA_MAXLEN];\n> +\tint err = 0;\n> +\tsize_t bytes_written = 0;\n> +\tsize_t bytes_to_write;\n\nThose two variables, instead of modifying the values of len and/or src_in,\nmake code very easy to read.\n\n> +\n> +\twhile (!err) {\n> +\t\tif ((len - bytes_written) > sizeof(buf))\n> +\t\t\tbytes_to_write = sizeof(buf);\n> +\t\telse\n> +\t\t\tbytes_to_write = len - bytes_written;\n> +\t\tif (bytes_to_write == 0)\n> +\t\t\tbreak;\n> +\t\terr = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);\n> +\t\tbytes_written += bytes_to_write;\n> +\t}\n> +\tif (!err)\n> +\t\terr = packet_flush_gently(fd_out);\n> +\treturn err;\n> +}\n\nLooks good: clean and readable.\n\n> +\n> static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n> \t\t\t void *dst, unsigned size, int options)\n> {\n> @@ -305,3 +346,30 @@ char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)\n> {\n> \treturn packet_read_line_generic(-1, src, src_len, dst_len);\n> }\n> +\n> +ssize_t read_packetized_to_buf(int fd_in, struct strbuf *sb_out)\n\nIt's a bit strange that the signature of write_packetized_from_buf() is\nthat different from read_packetized_to_buf(). This includes the return\nvalue: int vs ssize_t. As I have checked, write() and read() both\nuse ssize_t, while fread() and fwrite() both use size_t.\n\nPerhaps this function should be named read_packetized_to_strbuf()\n(err, I asked this already)?\n\n> +{\n> +\tint paket_len;\n\nPossible typo: shouldn't it be called packet_len?\nShouldn't it be initialized to 0?\n\n +\tint packet_len = 0;\n\n> +\tint options = PACKET_READ_GENTLE_ON_EOF;\n\nWhy is this even a local variable? It is never changed, and it is\nused only in one place; we can inline it.\n\nIf it would be needed in subsequent patches, then such information\nshould be included in the commit message.\n\n> +\n> +\tsize_t oldlen = sb_out->len;\n> +\tsize_t oldalloc = sb_out->alloc;\n\nJust a nitpick (feel free to ignore): doesn't this looks better:\n\n +\tsize_t old_len = sb_out->len;\n +\tsize_t old_alloc = sb_out->alloc;\n\nAlso perhaps s/old_/orig_/g.\n\n> +\n> +\tfor (;;) {\n\nI see that you used the more popular way of coding forever loop:\n\n $ git grep 'for (;;)' -- '*.c' | wc -l\n 120\n $ git grep 'while (1)' -- '*.c' | wc -l\n 86\n\n\n> +\t\tstrbuf_grow(sb_out, PKTLINE_DATA_MAXLEN+1);\n> +\t\tpaket_len = packet_read(fd_in, NULL, NULL,\n> +\t\t\tsb_out->buf + sb_out->len, PKTLINE_DATA_MAXLEN+1, options);\n\nA question (which perhaps was answered during the development of this\npatch series): why is this +1 in PKTLINE_DATA_MAXLEN+1 here?\n\n> +\t\tif (paket_len <= 0)\n> +\t\t\tbreak;\n> +\t\tsb_out->len += paket_len;\n> +\t}\n> +\n> +\tif (paket_len < 0) {\n> +\t\tif (oldalloc == 0)\n> +\t\t\tstrbuf_release(sb_out);\n> +\t\telse\n> +\t\t\tstrbuf_setlen(sb_out, oldlen);\n\nA question (maybe I don't understand strbufs): why there is a special\ncase for oldalloc == 0?\n\n> +\t\treturn paket_len;\n> +\t}\n> +\treturn sb_out->len - oldlen;\n> +}\n> diff --git a/pkt-line.h b/pkt-line.h\n> index 3fa0899..6df8449 100644\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -25,6 +25,8 @@ void packet_buf_flush(struct strbuf *buf);\n> void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n> int packet_flush_gently(int fd);\n> int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n> +int write_packetized_from_fd(int fd_in, int fd_out);\n> +int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n> \n> /*\n> * Read a packetized line into the buffer, which must be at least size bytes\n> @@ -77,6 +79,11 @@ char *packet_read_line(int fd, int *size);\n> */\n> char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n> \n> +/*\n> + * Reads a stream of variable sized packets until a flush packet is detected.\n> + */\n> +ssize_t read_packetized_to_buf(int fd_in, struct strbuf *sb_out);\n> +\n> #define DEFAULT_PACKET_MAX 1000\n> #define LARGE_PACKET_MAX 65520\n> extern char packet_buffer[LARGE_PACKET_MAX];\n> \n\n"},{"id":"302516","messageId":"6fc996fd-cdae-43c1-3ea9-2408e9ea47de@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-9-larsxschneider@gmail.com","subject":"Re: [PATCH v8 08/11] convert: quote filter names in error messages","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-25T14:03:59Z","receivedAt":"2016-09-25T14:04:08Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> Git filter driver commands with spaces (e.g. `filter.sh foo`) are hard\n> to read in error messages. Quote them to improve the readability.\n> \n> Signed-off-by: Lars Schneider \n> ---\n> convert.c | 12 ++++++------\n> 1 file changed, 6 insertions(+), 6 deletions(-)\n\nLooks good (those are all sites matching 'error.*%s' in convert.c).\n\n> -\t\treturn error(\"cannot fork to run external filter %s\", params->cmd);\n> +\t\treturn error(\"cannot fork to run external filter '%s'\", params->cmd);\n\n> -\t\terror(\"cannot feed the input to external filter %s\", params->cmd);\n> +\t\terror(\"cannot feed the input to external filter '%s'\", params->cmd);\n\n> -\t\terror(\"external filter %s failed %d\", params->cmd, status);\n> +\t\terror(\"external filter '%s' failed %d\", params->cmd, status);\n\n> -\t\terror(\"read from external filter %s failed\", cmd);\n> +\t\terror(\"read from external filter '%s' failed\", cmd);\n\n> -\t\terror(\"read from external filter %s failed\", cmd);\n> +\t\terror(\"read from external filter '%s' failed\", cmd);\n\n> -\t\terror(\"external filter %s failed\", cmd);\n> +\t\terror(\"external filter '%s' failed\", cmd);\n\n"},{"id":"302522","messageId":"1e2aac1a-b25d-d747-7bf6-6f842b3e438d@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-10-larsxschneider@gmail.com","subject":"Re: [PATCH v8 09/11] convert: modernize tests","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-25T14:43:08Z","receivedAt":"2016-09-25T14:43:18Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> Use `test_config` to set the config, check that files are empty with\n> `test_must_be_empty`, compare files with `test_cmp`, and remove spaces\n> after \">\" and \"<\".\n\nThat's good.\n\n> \n> Please note that the \"rot13\" filter configured in \"setup\" keeps using\n> `git config` instead of `test_config` because subsequent tests might\n> depend on it.\n\nThis is good information to have for doing review (which could include\n\"post-mortem\" review during bisect, so it should be in commit message\nproper).\n\n> \n> Reviewed-by: Stefan Beller \n\nI have not reviewed this patch in detail, but it looks good.\nA bit of nitpicking below.\n\n> Signed-off-by: Lars Schneider \n> ---\n> t/t0021-conversion.sh | 58 +++++++++++++++++++++++++--------------------------\n> 1 file changed, 29 insertions(+), 29 deletions(-)\n> \n> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n> index e799e59..dc50938 100755\n> --- a/t/t0021-conversion.sh\n> +++ b/t/t0021-conversion.sh\n> @@ -38,8 +38,8 @@ script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n> \n> test_expect_success check '\n\nThis patch is \"while at it\" already for this patch series, done\nI guess for new tests to both use modern style, and be consistent\nwith the rest of test...\n\n...that said, if you could modernize _naming_ of tests. The t0021\ntest is quite inconsistent, and uses:\n\n * standard short names, like 'setup', without quotes (once),\n which is I think all right\n * cryptic short names, like 'check', without quotes (once)\n * snake_case name, like 'expanded_in_repo', without quotes (once)\n \n> test_expect_success \"filter: clean empty file\" '\n> test_expect_success \"filter: smudge empty file\" '\n\n * double quoted names (twice, see above)\n * proper modern names, with single quotes (the rest),\n which is as almost all the rest should be using\n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"302523","messageId":"c305f3c8-cb01-c806-9b8d-fa9e805714cc@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-11-larsxschneider@gmail.com","subject":"Re: [PATCH v8 10/11] convert: make apply_filter() adhere to standard Git error handling","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-25T14:47:27Z","receivedAt":"2016-09-25T14:47:37Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> apply_filter() returns a boolean that tells the caller if it\n> \"did convert or did not convert\". The variable `ret` was used throughout\n> the function to track errors whereas `1` denoted success and `0`\n> failure. This is unusual for the Git source where `0` denotes success.\n> \n> Rename the variable and flip its value to make the function easier\n> readable for Git developers.\n\nThis also allow to use the 'err = error(\"\");' idiom,\nisn't it...\n\n> \n> Signed-off-by: Lars Schneider \n> ---\n> convert.c | 15 ++++++---------\n> 1 file changed, 6 insertions(+), 9 deletions(-)\n\n...which allows to delete some lines of code. Very nice.\n\n> -\tint ret = 1;\n> +\tint err = 0;\n\n> -\t\terror(\"read from external filter '%s' failed\", cmd);\n> -\t\tret = 0;\n> +\t\terr = error(\"read from external filter '%s' failed\", cmd);\n\n> -\tif (ret) {\n> +\tif (!err) {\n\n> -\treturn ret;\n> +\treturn !err;\n\nLooks good.\n\n"},{"id":"302580","messageId":"4B4255C0-4C0C-45F0-B37D-0C78C2AAFAE9@gmail.com","threadId":"44122","inReplyTo":"854ff387-57a4-4c27-4c27-b834f7797694@gmail.com","subject":"Re: [PATCH v8 03/11] run-command: move check_pipe() from write_or_die to run_command","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-26T16:13:35Z","receivedAt":"2016-09-26T16:14:03Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 25 Sep 2016, at 00:12, Jakub Narębski wrote:\n> \n> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n>> From: Lars Schneider \n>> \n>> Move check_pipe() to run_command and make it public. This is necessary\n>> to call the function from pkt-line in a subsequent patch.\n> \n> All right.\n\nDoes this mean I can add your \"Acked-by: Jakub Narebski \" ?\n\nThanks,\nLars"},{"id":"302582","messageId":"CANQwDwd7YMjoyO3Y=FqcX720_vcqa0L=z6veTgYO==LorhqqFQ@mail.gmail.com","threadId":"44122","inReplyTo":"4B4255C0-4C0C-45F0-B37D-0C78C2AAFAE9@gmail.com","subject":"Re: [PATCH v8 03/11] run-command: move check_pipe() from write_or_die to run_command","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-26T16:21:47Z","receivedAt":"2016-09-26T16:22:34Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On 26 September 2016 at 18:13, Lars Schneider wrote:\n>> On 25 Sep 2016, at 00:12, Jakub Narębski wrote:\n>> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n>>> From: Lars Schneider \n>>>\n>>> Move check_pipe() to run_command and make it public. This is necessary\n>>> to call the function from pkt-line in a subsequent patch.\n>>\n>> All right.\n>\n> Does this mean I can add your \"Acked-by: Jakub Narebski \" ?\n\nWell, Acked-by makes sense if it is from subsystem maintainer. I can only\nclaim gitweb subsystem where my ACKs might make sense.\n\nThis \"All right\" is here to note that I have read this patch (and not\nskipped it),\nand I have't found anything to complain about or nitpick ;-P\n\nBest,\n-- \nJakub Narębski\n"},{"id":"302613","messageId":"13E00B0F-41FC-4FBE-9ABD-2E9B11743E2B@gmail.com","threadId":"44122","inReplyTo":"592e18b7-ac2e-ed23-0321-57eb418d5172@gmail.com","subject":"Re: [PATCH v8 01/11] pkt-line: rename packet_write() to packet_write_fmt()","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-26T18:49:51Z","receivedAt":"2016-09-26T18:50:01Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 24 Sep 2016, at 23:14, Jakub Narębski wrote:\n\n> Hello Lars,\n> \n> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> \n>> From: Lars Schneider \n>> \n>> packet_write() should be called packet_write_fmt() as the string\n>> parameter can be formatted.\n> \n> I would say:\n> \n> packet_write() should be called packet_write_fmt() because it\n> is printf-like function where first parameter is format string.\n> \n> Or something like that. But such minor change might be not worth\n> yet another reroll of this patch series.\n> \n> Perhaps it would be a good idea to explain the reasoning behind\n> this change:\n> \n> This is important distinction to know from the name if the\n> function accepts arbitrary binary data and/or arbitrary\n> strings to be written - packet_write[_fmt()] do not.\n\npacket_write() should be called packet_write_fmt() because it is a\nprintf-like function that takes a format string as first parameter.\n\npacket_write_fmt() should be used for text strings only. Arbitrary\nbinary data should use a new packet_write() function that is introduced\nin a subsequent patch.\n\nBetter?\n\n\n>> pkt-line.h | 2 +-\n>> shallow.c | 2 +-\n>> upload-pack.c | 30 +++++++++++++++---------------\n>> 11 files changed, 29 insertions(+), 29 deletions(-)\n> \n> Diffstat looks correct. Was the patch generated by doing search\n> and replace?\n\nYes.\n\n- Lars"},{"id":"302614","messageId":"C797715E-97C3-4EEA-AD9F-C22FA27F0421@gmail.com","threadId":"44122","inReplyTo":"a5b9a46d-592e-089d-d156-36c4d61902c9@gmail.com","subject":"Re: [PATCH v8 02/11] pkt-line: extract set_packet_header()","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-26T18:53:20Z","receivedAt":"2016-09-26T18:53:28Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 24 Sep 2016, at 23:22, Jakub Narębski wrote:\n\n> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> \n>> From: Lars Schneider \n>> \n>> Subject: [PATCH v8 02/11] pkt-line: extract set_packet_header()\n>> \n>> set_packet_header() converts an integer to a 4 byte hex string. Make\n>> this function locally available so that other pkt-line functions can\n>> use it.\n> \n> Ah. I have trouble understanding this commit message, as the\n> set_packet_header() was not available before this patch, but it\n> is good if one reads it together with commit summary / title.\n> \n> Writing\n> \n> Extracted set_packet_header() function converts...\n> \n> or\n> \n> New set_packet_header() function converts... \n> \n> would make it more clear, but it is all right as it is now.\n> Perhaps also\n> \n> ... could use it.\n> \n> as currently no other pkt-line function but the one set_packet_header()\n> was extracted from, namely format_packet(), uses it.\n> \n> But that is just nitpicking; no need to change on that account.\n\nChanged it:\n\nExtracted set_packet_header() function converts an integer to a 4 byte \nhex string. Make this function locally available so that other pkt-line \nfunctions could use it.\n\nThanks,\nLars"},{"id":"302618","messageId":"4CF26590-7B88-4DB6-B6D4-A6CFE0FD6ABF@gmail.com","threadId":"44122","inReplyTo":"55c5a86e-4055-f36d-bbf1-e8bae919088a@gmail.com","subject":"Re: [PATCH v8 06/11] pkt-line: add packet_write_gently()","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-26T19:21:10Z","receivedAt":"2016-09-26T19:21:20Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 25 Sep 2016, at 13:26, Jakub Narębski wrote:\n\n> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n>> From: Lars Schneider \n>> ...\n>> \n>> +static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n> \n> I'm not sure what naming convention the rest of Git uses, but isn't\n> it more like '*data' rather than '*buf' here?\n\npkt-line seems to use 'buf' or 'buffer' for everything else.\n\n\n>> +{\n>> +\tstatic char packet_write_buffer[LARGE_PACKET_MAX];\n> \n> I think there should be warning (as a comment before function\n> declaration, or before function definition), that packet_write_gently()\n> is not thread-safe (nor reentrant, but the latter does not matter here,\n> I think).\n> \n> Thread-safe vs reentrant: http://stackoverflow.com/a/33445858/46058\n> \n> This is not something terribly important; I guess git code has tons\n> of functions not marked as thread-unsafe...\n\nI agree that the function is not thread-safe. However, I can't find \nan example in the Git source that marks a function as not thread-safe.\nUnless is it explicitly stated in the coding guidelines I would prefer\nnot to start way to mark functions.\n\n\n>> +\tif (size > sizeof(packet_write_buffer) - 4) {\n> \n> First, wouldn't the following be more readable:\n> \n> +\tif (size + 4 > LARGE_PACKET_MAX) {\n\nPeff suggested that here:\nhttp://public-inbox.org/git/20160810132814.gqnipsdwyzjmuqjy@sigill.intra.peff.net/\n\n\n>> +\t\treturn error(\"packet write failed - data exceeds max packet size\");\n>> +\t}\n> \n> Second, CodingGuidelines is against using braces (blocks) for one\n> line conditionals: \"We avoid using braces unnecessarily.\"\n> \n> But this is just me nitpicking.\n\nFixed.\n\n\n>> +\tpacket_trace(buf, size, 1);\n>> +\tsize += 4;\n>> +\tset_packet_header(packet_write_buffer, size);\n>> +\tmemcpy(packet_write_buffer + 4, buf, size - 4);\n>> +\tif (write_in_full(fd_out, packet_write_buffer, size) == size)\n> \n> Hmmm... in some places we use original size, in others (original) size + 4;\n> perhaps it would be more readable to add a new local temporary variable\n> \n> \tsize_t full_size = size + 4;\n\nAgreed. I introduced \"packet_size\".\n\nThanks,\nLars"},{"id":"302632","messageId":"77315FC2-47F3-433A-8D70-5497FB04CBBE@gmail.com","threadId":"44122","inReplyTo":"507a29a1-99ff-c13e-cf1f-76631345a822@gmail.com","subject":"Re: [PATCH v8 07/11] pkt-line: add functions to read/write flush terminated packet streams","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-26T20:23:32Z","receivedAt":"2016-09-26T20:23:42Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 25 Sep 2016, at 15:46, Jakub Narębski wrote:\n\n> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n>> From: Lars Schneider \n\n\n>> +{\n>> +\tstatic char buf[PKTLINE_DATA_MAXLEN];\n> \n> Sidenote: we have LARGE_PACKET_MAX (used in previous patch), but\n> PKTLINE_DATA_MAXLEN not LARGE_PACKET_DATA_MAX.\n\nAgreed, I will rename it.\n\n\n> \n>> +\tint err = 0;\n>> +\tssize_t bytes_to_write;\n>> +\n>> +\twhile (!err) {\n>> +\t\tbytes_to_write = xread(fd_in, buf, sizeof(buf));\n>> +\t\tif (bytes_to_write < 0)\n>> +\t\t\treturn COPY_READ_ERROR;\n>> +\t\tif (bytes_to_write == 0)\n>> +\t\t\tbreak;\n>> +\t\terr = packet_write_gently(fd_out, buf, bytes_to_write);\n>> +\t}\n>> +\tif (!err)\n>> +\t\terr = packet_flush_gently(fd_out);\n>> +\treturn err;\n>> +}\n> \n> Looks good: clean and readable.\n> \n> Sidenote (probably outside of scope of this patch): what are the\n> errors that we can get from this function, beside COPY_READ_ERROR\n> of course?\n\nEverything that is returned by \"read()\"\n\n\n>> +\n>> static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n>> \t\t\t void *dst, unsigned size, int options)\n>> {\n>> @@ -305,3 +346,30 @@ char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)\n>> {\n>> \treturn packet_read_line_generic(-1, src, src_len, dst_len);\n>> }\n>> +\n>> +ssize_t read_packetized_to_buf(int fd_in, struct strbuf *sb_out)\n> \n> It's a bit strange that the signature of write_packetized_from_buf() is\n> that different from read_packetized_to_buf(). This includes the return\n> value: int vs ssize_t. As I have checked, write() and read() both\n> use ssize_t, while fread() and fwrite() both use size_t.\n\nread_packetized_to_buf() returns the number of bytes read or a negative \nerror code.\n\nwrite_packetized_from_buf() returns 0 if the call was successful and an \nerror code if not.\n\nThat's the reason these two functions have a different signature\n\n\n> Perhaps this function should be named read_packetized_to_strbuf()\n> (err, I asked this already)?\n\nI agree with the rename as makes it distinct from \nwrite_packetized_from_buf().\n\n\n>> +{\n>> +\tint paket_len;\n> \n> Possible typo: shouldn't it be called packet_len?\n\nTrue!\n\n\n> Shouldn't it be initialized to 0?\n\nWell, it is set for sure later. That's why I think it is not necessary.\n\nPlus, Eric Wong thought me not to:\n\"Compilers complain about uninitialized variables.\"\nhttp://public-inbox.org/git/20160725072745.GB11634@starla/\n(Note: he was talking about pointers there :-)\n\n\n>> +\tint options = PACKET_READ_GENTLE_ON_EOF;\n> \n> Why is this even a local variable? It is never changed, and it is\n> used only in one place; we can inline it.\n\nRemoved.\n\n\n>> +\n>> +\tsize_t oldlen = sb_out->len;\n>> +\tsize_t oldalloc = sb_out->alloc;\n> \n> Just a nitpick (feel free to ignore): doesn't this looks better:\n> \n> +\tsize_t old_len = sb_out->len;\n> +\tsize_t old_alloc = sb_out->alloc;\n> \n> Also perhaps s/old_/orig_/g.\n\nAgreed. That matches the other variables better.\n\n\n>> +\t\tstrbuf_grow(sb_out, PKTLINE_DATA_MAXLEN+1);\n>> +\t\tpaket_len = packet_read(fd_in, NULL, NULL,\n>> +\t\t\tsb_out->buf + sb_out->len, PKTLINE_DATA_MAXLEN+1, options);\n> \n> A question (which perhaps was answered during the development of this\n> patch series): why is this +1 in PKTLINE_DATA_MAXLEN+1 here?\n\nNice catch. I think this is wrong:\nhttps://github.com/git/git/blob/6fe1b1407ed91823daa5d487abe457ff37463349/pkt-line.c#L196\n\nIt should be \"if (len > size)\" ... then we don't need the \"+1\" here.\n(but I need to think a bit more about this)\n\n> \n>> +\t\tif (paket_len <= 0)\n>> +\t\t\tbreak;\n>> +\t\tsb_out->len += paket_len;\n>> +\t}\n>> +\n>> +\tif (paket_len < 0) {\n>> +\t\tif (oldalloc == 0)\n>> +\t\t\tstrbuf_release(sb_out);\n>> +\t\telse\n>> +\t\t\tstrbuf_setlen(sb_out, oldlen);\n> \n> A question (maybe I don't understand strbufs): why there is a special\n> case for oldalloc == 0?\n\nI tried to mimic the behavior of strbuf_read() [1]. The error handling\nwas introduced in 2fc647 [2] to ease error handling:\n\n\"This allows for easier error handling, as callers only need to call\nstrbuf_release() if A) the command succeeded or B) if they would have had\nto do so anyway because they added something to the strbuf themselves.\"\n\n[1] https://github.com/git/git/blob/cda1bbd474805e653dda8a71d4ea3790e2a66cbb/strbuf.c#L377-L383\n[2] https://github.com/git/git/commit/2fc647004ac7016128372a85db8245581e493812\n\n\nThanks,\nLars"},{"id":"302648","messageId":"91589466-439e-7200-7256-b9288beae685@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-12-larsxschneider@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-26T22:41:54Z","receivedAt":"2016-09-26T22:42:13Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Part first of the review of 11/11.\n\nW dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> From: Lars Schneider \n> \n> Git's clean/smudge mechanism invokes an external filter process for\n> every single blob that is affected by a filter. If Git filters a lot of\n> blobs then the startup time of the external filter processes can become\n> a significant part of the overall Git execution time.\n> \n> In a preliminary performance test this developer used a clean/smudge\n> filter written in golang to filter 12,000 files. This process took 364s\n> with the existing filter mechanism and 5s with the new mechanism. See\n> details here: https://github.com/github/git-lfs/pull/1382\n> \n> This patch adds the `filter..process` string option which, if\n> used, keeps the external filter process running and processes all blobs\n> with the packet format (pkt-line) based protocol over standard input and\n> standard output. The full protocol is explained in detail in\n> `Documentation/gitattributes.txt`.\n\nThat is a good description. Enough detail to explain the new feature,\nall without duplicating information with (added) docs.\n\n> \n> A few key decisions:\n> \n> * The long running filter process is referred to as filter protocol\n> version 2 because the existing single shot filter invocation is\n> considered version 1.\n\nAll right.\n\n> * Git sends a welcome message and expects a response right after the\n> external filter process has started. This ensures that Git will not\n> hang if a version 1 filter is incorrectly used with the\n> filter..process option for version 2 filters. In addition,\n> Git can detect this kind of error and warn the user.\n\nOn one hand side, this involved handshake means that implementing\na filter process script is harder; you need to write quite a lot of\nboilerplate (though the example or examples would help).\n\nOn the other hand, this handshake is what allows good error detection,\neasy extendability of the protocol, and forward-compatibility. Which,\nas we agreed (AFAIU), is more important.\n\n> * The status of a filter operation (e.g. \"success\" or \"error) is set\n> before the actual response and (if necessary!) re-set after the\n> response. The advantage of this two step status response is that if\n> the filter detects an error early, then the filter can communicate\n> this and Git does not even need to create structures to read the\n> response.\n\nThat's nice (well, among others I have argued for this :-))\n\n> * All status responses are pkt-line lists terminated with a flush\n> packet. This allows us to send other status fields with the same\n> protocol in the future.\n\nGood.\n\nThis also makes protocol simple, easier to implement (on Git side),\nand easier to parse (on filter side).\n\n> \n> Helped-by: Martin-Louis Bright \n> Reviewed-by: Jakub Narebski \n> Signed-off-by: Lars Schneider \n> ---\n> Documentation/gitattributes.txt | 156 +++++++++++++-\n> contrib/long-running-filter/example.pl | 123 +++++++++++\n> convert.c | 348 ++++++++++++++++++++++++++++---\n> pkt-line.h | 1 +\n> t/t0021-conversion.sh | 365 ++++++++++++++++++++++++++++++++-\n> t/t0021/rot13-filter.pl | 191 +++++++++++++++++\n> 6 files changed, 1153 insertions(+), 31 deletions(-)\n\nThat's quite a large change. Large changes are harder to review.\nI was thinking about how one could split this change. I guess\nthat it is better to keep the new feature, its documentation, and\nits tests together. But perhaps the example in `contrib/`\n(which is newer, and thus less reviewed) would be better in\nseparate commit.\n\nThere is also another change that could be split off this patch\ninto purely preparatory commit, that is one that stands alone\nbut doesn't make much sense alone. I would write about this\nproposal (important only if there would be yet another iteration\nof this patch series) a bit later.\n\n> create mode 100755 contrib/long-running-filter/example.pl\n> create mode 100755 t/t0021/rot13-filter.pl\n> \n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index 7aff940..946dcad 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -293,7 +293,13 @@ checkout, when the `smudge` command is specified, the command is\n> fed the blob object from its standard input, and its standard\n> output is used to update the worktree file. Similarly, the\n> `clean` command is used to convert the contents of worktree file\n> -upon checkin.\n> +upon checkin. By default these commands process only a single\n> +blob and terminate. If a long running `process` filter is used\n ^^^^\n\nShould we use this terminology here? I have not read the preceding\npart of documentation, so I don't know if it talks about \"blobs\" or\nif it uses \"files\" and/or \"file contents\".\n\nThough this is very minor nitpick.\n\n> +in place of `clean` and/or `smudge` filters, then Git can process\n> +all blobs with a single filter command invocation for the entire\n> +life of a single Git command, for example `git add --all`. See\n> +section below for the description of the protocol used to\n> +communicate with a `process` filter.\n\nGood introduction of long lived filter feature (`process` filter).\n\n> \n> One use of the content filtering is to massage the content into a shape\n> that is more convenient for the platform, filesystem, and the user to use.\n> @@ -373,6 +379,154 @@ not exist, or may have different contents. So, smudge and clean commands\n> should not try to access the file on disk, but only act as filters on the\n> content provided to them on standard input.\n> \n> +Long Running Filter Process\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +If the filter command (a string value) is defined via\n> +`filter..process` then Git can process all blobs with a\n> +single filter invocation for the entire life of a single Git\n> +command. This is achieved by using a packet format (pkt-line,\n> +see technical/protocol-common.txt) based protocol over standard\n> +input and standard output as follows. All packets are considered\n> +text and therefore are terminated by an LF. Exceptions are the\n> +\"*CONTENT\" packets and the flush packet.\n\nI guess that reasoning here is that all but CONTENT packets are\nmetadata, and thus to aid debuggability of the protocol are \"text\",\nas considered by pkt-line.\n\nPerhaps a bit more readable would be the following (but current is\njust fine; I am nitpicking):\n\n All packets, except for the \"{star}CONTENT\" packets and the \"0000\"\n flush packer, are considered text and therefore are terminated by\n a LF.\n\nOr maybe:\n\n All metadata is considered text, and thus send as a text packet,\n that is terminated with the newline (LF) character. The file\n contents is send as binary data, as is, without appending LF.\n The flush packet is a packet composed of 4 bytes, represented\n in ASCII as \"0000\".\n\nAnyway, this is all right as it is now; we can always polish it\nlater.\n\n> +\n> +Git starts the filter when it encounters the first file\n> +that needs to be cleaned or smudged. After the filter started\n> +Git sends a welcome message (\"git-filter-client\"), a list of\n> +supported protocol version numbers, and a flush packet. Git expects\n\nI see that example below explains what is the format of sending\nthis list of process filter protocol version numbers supported by\nGit (I suppose it is to future proof adding new versions of protocol,\nand removing old ones if they are somehow buggy). It would be nice,\nI think, to explain it in more detail:\n\n a list of supported protocol version numbers, each version in\n a separate text packet using the \"version=\" format,\n\nBut with an example few paragraphs below it might be not necessary.\n\nI think it might be a good idea to describe what flush packet is\nsomewhere in this document; on the other hand referring (especially\nif hyperlinked) to pkt-line technical documentation might be good\nenough / better. I'm unsure, but I tend on the side that referring\nto technical documentation is better.\n\n> +to read a welcome response message (\"git-filter-server\") and exactly\n> +one protocol version number from the previously sent list. All further\n\nI guess that is to provide forward-compatibility, isn't it? Also,\n\"Git expects...\" probably means filter process MUST send, in the\nRFC2119 (https://tools.ietf.org/html/rfc2119) meaning.\n\n> +communication will be based on the selected version. The remaining\n> +protocol description below documents \"version=2\". Please note that\n> +\"version=42\" in the example below does not exist and is only there\n> +to illustrate how the protocol would look like with more than one\n> +version.\n\nAll right.\n\n> +\n> +After the version negotiation Git sends a list of supported capabilities\n> +and a flush packet.\n\nIs it that Git SHOULD send list of ALL supported capabilities, or is\nit that Git SHOULD NOT send capabilities it does not support, and that\nit MAY send only those capabilities it needs (so for example if command\nuses only `smudge`, it may not send `clean`, so that filter driver doesn't\nneed to initialize data it would not need).\n\nI guess with the example few lines below there is no need to explain\nthe format of capabilities (or use BNF / EBNF notation to define it).\n\nI wonder why it is \"=true\", and not \"capability=\".\nIs there a case where we would want to send \"=false\". Or\nis it to allow configurable / value based capabilities? Isn't it going\na bit too far: is there even a hind of an idea for parametrize-able\ncapability? YAGNI is a thing...\n\nA few new capabilities that we might want to support in the near future\nis \"size\", \"stream\", which are options describing how to communicate,\nand \"cleanFromFile\", \"smudgeToFile\", which are new types of operations...\nbut neither needs any parameter.\n\nI guess that adding new capabilities doesn't require having to come up\nwith the new version of the protocol, isn't it.\n\n> Git expects to read a list of desired capabilities,\n> +which must be a subset of the supported capabilities list, and a flush\n> +packet as response:\n\nAll right, with Git speaking first, having Git provide list of supported\ncapabilities first is quite natural.\n\n> +------------------------\n> +packet: git> git-filter-client\n\nI guess we assume that from the above description it is obvious that\nthis is\n\n +packet: git> git-filter-client\\n\n\nAll right.\n\n> +packet: git> version=2\n> +packet: git> version=42\n\n\"List\" means \"each in separate packet\", right. Here also\n\n +packet: git> version=2\\n\n\n> +packet: git> 0000\n\nAs I wrote, I hope everybody would understand that is a flush packet,\nthat is packet composed literally of 4 characters / bytes \"0000\",\nand not binary or text packet with \"0000\" as contents, that is\n\"00040000\" packet or \"00050000\\n\" packet.\n\nBut as it is consistent with other examples, and with GIT_TRACE_PACKET\noutput, I think both skipping trailing \\n for text packets, and\nwriting \"0000\" for flush packet is all right. Sorry for the noise.\n\nSidenote (you don't have to answer to): do we use \"0004\" packet as\na keep-alive packet anywhere?\n\n> +packet: git< git-filter-server\n> +packet: git< version=2\n> +packet: git> clean=true\n> +packet: git> smudge=true\n> +packet: git> not-yet-invented=true\n\nHmmm... should we hint at the use of kebab-case versus snake_case\nor camelCase for new capabilities?\n\n> +packet: git> 0000\n> +packet: git< clean=true\n> +packet: git< smudge=true\n> +packet: git< 0000\n> +------------------------\n> +Supported filter capabilities in version 2 are \"clean\" and\n> +\"smudge\".\n\nI think we can add new capabilities without increasing version number\nof the protocol. But then we can and should update this part of the\ndocumentation.\n\nAll right.\n\n> +\n> +Afterwards Git sends a list of \"key=value\" pairs terminated with\n> +a flush packet. The list will contain at least the filter command\n> +(based on the supported capabilities) and the pathname of the file\n> +to filter relative to the repository root. Right after these packets\n> +Git sends the content split in zero or more pkt-line packets and a\n> +flush packet to terminate content.\n\nAll right, the example below shows what are the names of 'variables;\nin those packets (that is \"command=\" ( \"clean\" | \"smudge\" ), and\n\"pathname=\" ).\n\n> +------------------------\n> +packet: git> command=smudge\n> +packet: git> pathname=path/testfile.dat\n> +packet: git> 0000\n> +packet: git> CONTENT\n> +packet: git> 0000\n> +------------------------\n\nI think it is important to mention that (at least with current\n`filter..process` implementation, that is absent future\n\"stream\" capability / option) the filter process needs to read\n*whole contents* at once, *before* writing anything. Otherwise\nit can lead to deadlock.\n\nThis is especially important in that it is different (!) from the\ncurrent behavior of `clean` and `smudge` filters, which can\nstream their response because Git invokes them async.\n\n> +\n> +The filter is expected to respond with a list of \"key=value\" pairs\n> +terminated with a flush packet. If the filter does not experience\n> +problems then the list must contain a \"success\" status.\n\nPerhaps \"status\" packet with value \"success\", or\n\n If the filter does not experience\n +problems then the list must contain a \"status=success\" line.\n\nPossibly s/line./packet./\n\nBut as I see with how it is explained further, 'a \"success\" status'\nworks too. No need to change, then.\n\n> Right after\n> +these packets the filter is expected to send the content in zero\n> +or more pkt-line packets and a flush packet at the end. Finally, a\n> +second list of \"key=value\" pairs terminated with a flush packet\n> +is expected. The filter can change the status in the second list.\n> +------------------------\n> +packet: git< status=success\n> +packet: git< 0000\n> +packet: git< SMUDGED_CONTENT\n> +packet: git< 0000\n> +packet: git< 0000 # empty list!\n> +------------------------\n\nAll right. Empty list with no change in status looks good.\n\nAn alternative would be to assume different meaning to the \"status\"\nbefore and after sending contents, e.g.\n\n packet: git< received=ok\n packet: git< 0000\n packet: git< SMUDGED_CONTENT\n packet: git< 0000\n packet: git< sent=ok\n packet: git< 0000\n\nBut I think current solution is good enough.\n\n> +\n> +If the result content is empty then the filter is expected to respond\n> +with a success status and an empty list.\n> +------------------------\n> +packet: git< status=success\n> +packet: git< 0000\n> +packet: git< 0000 # empty content!\n> +packet: git< 0000 # empty list!\n> +------------------------\n\nAll right. This follows from the definition, but it is nice to have\nit spelled in full.\n\n> +\n> +In case the filter cannot or does not want to process the content,\n> +it is expected to respond with an \"error\" status. Depending on the\n> +`filter..required` flag Git will interpret that as error\n> +but it will not stop or restart the filter process.\n\nRight, and Git would not try to read contents from the filter then.\n\n> +------------------------\n> +packet: git< status=error\n> +packet: git< 0000\n> +------------------------\n> +\n> +If the filter experiences an error during processing, then it can\n> +send the status \"error\" after the content was (partially or\n> +completely) sent. Depending on the `filter..required` flag\n> +Git will interpret that as error but it will not stop or restart the\n> +filter process.\n> +------------------------\n> +packet: git< status=success\n> +packet: git< 0000\n> +packet: git< HALF_WRITTEN_ERRONEOUS_CONTENT\n> +packet: git< 0000\n> +packet: git< status=error\n> +packet: git< 0000\n> +------------------------\n\nGood. A question is if the filter process can send \"status=abort\"\nafter partial contents, or does it need to wait for the next command?\n\n> +\n> +If the filter dies during the communication or does not adhere to\n> +the protocol then Git will stop the filter process and restart it\n> +with the next file that needs to be processed. Depending on the\n> +`filter..required` flag Git will interpret that as error.\n> +\n> +The error handling for all cases above mimic the behavior of\n> +the `filter..clean` / `filter..smudge` error\n> +handling.\n\nGood.\n\n> +\n> +In case the filter cannot or does not want to process the content\n> +as well as any future content for the lifetime of the Git process,\n> +it is expected to respond with an \"abort\" status. Depending on\n> +the `filter..required` flag Git will interpret that as error\n> +for the content as well as any future content for the lifetime of the\n> +Git process but it will not stop or restart the filter process.\n> +------------------------\n> +packet: git< status=abort\n> +packet: git< 0000\n> +------------------------\n\nI assume this is obvious that if filter process tells \"abort\", Git\nwould not try to send further files (regardless of the value of\n`filter..required`).\n\n> +\n> +After the filter has processed a blob it is expected to wait for\n> +the next \"key=value\" list containing a command. Git will close\n> +the command pipe on exit. The filter is expected to detect EOF\n> +and exit gracefully on its own.\n\nGood to have it documented. \n\nAnyway, as it is Git command that spawns the filter driver process,\nassuming that the filter process doesn't daemonize itself, wouldn't\nthe operating system reap it after its parent process, that is the\ngit command it invoked, dies? So detecting EOF is good, but not\nstrictly necessary for simple filter that do not need to free\nits resources, or can leave freeing resources to the operating\nsystem? But I may be wrong here.\n\n> +\n> +A long running filter demo implementation can be found in\n> +`contrib/long-running-filter/example.pl` located in the Git\n> +core repository. If you develop your own long running filter\n> +process then the `GIT_TRACE_PACKET` environment variables can be\n> +very helpful for debugging (see linkgit:git[1]).\n\nVery good... though I wonder if adding demo implementation should\nnot be left for a separate commit.\n\n> +\n> +If a `filter..process` command is configured then it\n> +always takes precedence over a configured `filter..clean`\n> +or `filter..smudge` command.\n\nThis is a change from what I remember of previous iterations of this\npatch series, but I see that it might be the best solution of the\nthree possible (that I can think of):\n\n* `filter..clean` or `filter..smudge` command\n always takes precedence over `filter..process`\n\n ADVANTAGES: \n - can convert only half of filter to process filter\n\n DISADVANTAGES: \n - uncommon \"older type wins\"\n - cannot provide fallback of one-shot filters for older Git\n\n* `filter..process` command always takes precedence\n over `filter..clean` or `filter..smudge` command\n (this is the one chosen)\n\n ADVANTAGES:\n - can provide `clean` and `smudge` filters as fallback for\n older Git (e.g. installed locally, and not always in PATH)\n\n DISADVANTAGES:\n - need to convert both `clean` and `smudge` part into `process`\n at once\n\n* `filter..process` command takes precedence over\n `filter..clean` if it supports \"clean\" capability, and\n similarly for `filter..smudge`\n\n ADVANTAGES:\n - can convert only half of filter to process filter\n - can provide `clean` and `smudge` filters as fallback for\n older Git (e.g. installed locally, and not always in PATH)\n \n DISADVANTAGES:\n - you need to see the filter implementation to know which\n one would be invoked\n - complicated to understand, reason about, and implement\n\n> +\n> +Please note that you cannot use an existing `filter..clean`\n> +or `filter..smudge` command with `filter..process`\n> +because the former two use a different inter process communication\n> +protocol than the latter one.\n\nGood.\n\n> +\n> +\n> Interaction between checkin/checkout attributes\n> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> \n> diff --git a/contrib/long-running-filter/example.pl b/contrib/long-running-filter/example.pl\n> new file mode 100755\n> index 0000000..c13a631\n> --- /dev/null\n> +++ b/contrib/long-running-filter/example.pl\n\nTo repeat myself, I think it would serve better as a separate patch.\n\nPerhaps even you could add the example filter in the Go language\nfrom https://github.com/github/git-lfs/pull/1382 (there is already\nat least one Go program in contrib/). In a separate patch, of course.\nThe commands/command_filter.go.\n\n> @@ -0,0 +1,123 @@\n> +#!/usr/bin/perl\n> +#\n> +# Example implementation for the Git filter protocol version 2\n> +# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n\nWe might add that it is a pass-thru filter, a skeleton to be\nmodified.\n\n> +#\n> +\n> +use strict;\n> +use warnings;\n> +\n> +my $MAX_PACKET_CONTENT_SIZE = 65516;\n\nAll right, constants (the built in / in core ones) are strange...\nVariables are easier to use.\n\n> +\n> +sub packet_bin_read {\n> + my $buffer;\n> + my $bytes_read = read STDIN, $buffer, 4;\n> + if ( $bytes_read == 0 ) {\n> +\n> + # EOF - Git stopped talking to us!\n> + exit();\n> + }\n> + elsif ( $bytes_read != 4 ) {\n\nThat is a different Perl coding convention that the one I am used to,\nbut it is no less valid. Consistent style is more important. This is\nthe contrib/ area anyway.\n\n> + die \"invalid packet size '$bytes_read' field\";\n\nThis would read \"invalid packet size '000' field\", for example.\nPerhaps the following would be (slightly) better:\n\n + die \"invalid packet size field: '$bytes_read'\";\n\n> + }\n> + my $pkt_size = hex($buffer);\n> + if ( $pkt_size == 0 ) {\n> + return ( 1, \"\" );\n\nIt feels a bit strange to me to return a list / a pair combining\nstatus / error condition with the actual result. I'm more used\nto packing such in hash reference. But I think it is all right.\n\n> + }\n> + elsif ( $pkt_size > 4 ) {\n\nIsn't a packet of $pkt_size == 4 a valid packet, a keep-alive\none? Or is it forbidden?\n\nWe can declare that Git should not use it for filter process anyway.\n\n> + my $content_size = $pkt_size - 4;\n> + $bytes_read = read STDIN, $buffer, $content_size;\n> + if ( $bytes_read != $content_size ) {\n> + die \"invalid packet ($content_size expected; $bytes_read read)\";\n\nThis error message would read \"invalid packet (12 expected; 10 read)\";\nI think it would be better to rephrase it as\n\n + die \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n\n> + }\n> + return ( 0, $buffer );\n> + }\n> + else {\n> + die \"invalid packet size\";\n\nI'm not sure if it is worth it (especially for the demo script),\nbut perhaps we could show what this invalid size was?\n\n + die \"invalid packet size value '$pkt_size'\";\n\n> + }\n> +}\n> +\n> +sub packet_txt_read {\n> + my ( $res, $buf ) = packet_bin_read();\n> + unless ( $buf =~ /\\n$/ ) {\n\nWouldn't\n\n + unless ( $buf =~ s/\\n$// ) {\n\nor (less so)\n\n + unless ( $buf =~ s/\\n$\\z// ) {\n\nbe more idiomatic (and not require use of 'substr')? Remember,\nthe s/// substitution quote-like operator returns number of\nsubstitutions in the scalar context.\n\n> + die \"A non-binary line SHOULD BE terminated by an LF.\";\n\nThis is SHOULD be, not MUST be, so perhaps 'warn' would be enough.\nNot that Git should send us such line.\n\n> + }\n> + return ( $res, substr( $buf, 0, -1 ) );\n\nThis would be not necessary if s/// instead of m// was used.\n\n> +}\n> +\n> +sub packet_bin_write {\n> + my ($packet) = @_;\n\nThis is equivalent to\n\n + my $packet = shift;\n\nwhich, I think, is more common for single-parameter subroutines.\n\nAlso, this is $data (or $buf), not $packet.\n\n> + print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n> + print STDOUT $packet;\n> + STDOUT->flush();\n> +}\n> +\n> +sub packet_txt_write {\n> + packet_bin_write( $_[0] . \"\\n\" );\n> +}\n\nNice.\n\n> +\n> +sub packet_flush {\n> + print STDOUT sprintf( \"%04x\", 0 );\n\nWe could use simply\n\n + print STDOUT \"0000\";\n\nbut this is more explicit. Good.\n\n> + STDOUT->flush();\n> +}\n> +\n\nPerhaps some comment that main begins here?\n\n> +( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n> +( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n> +( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n\nActually, it is overly strict. It should not fail if there\nare other \"version=3\", \"version=4\" etc. lines.\n\n> +\n> +packet_txt_write(\"git-filter-server\");\n> +packet_txt_write(\"version=2\");\n\nIt needs to do\n\n +packet_flush();\n\nhere.\n\n> +\n> +( packet_txt_read() eq ( 0, \"clean=true\" ) ) || die \"bad capability\";\n> +( packet_txt_read() eq ( 0, \"smudge=true\" ) ) || die \"bad capability\";\n> +( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n\nIt is also overly strict. The capabilities can be ordered in any\nway, and there can be additional capabilities which this script\ndo not understand. It is all right to have such capabilities.\n\nAll this makes it better to extract the handshake / metadata part\ninto a subroutine.\n\n> +\n> +packet_txt_write(\"clean=true\");\n> +packet_txt_write(\"smudge=true\");\n> +packet_flush();\n\nAll right.\n\n> +\n> +while (1) {\n> + my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n> + my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;\n\nDo we require this order? If it is, is that explained in the\ndocumentation?\n\n> +\n> + packet_bin_read();\n\nI think there can be other auxiliary data (like \"size=\") that\nfilter do not need to understand.\n\nAnyway, using packet_bin_read() is quite unreadable. What you\nmean is to wait until flush packet, or expect flush packet.\n\n \tmy $done = 0;\n \twhile ( !$done ) {\n \t\t( $done, undef ) = packet_bin_read();\n \t}\n\nOr\n\n \twait_for_flush();\n\nOr\n\n \tskip_until_flush();\n\n> +\n> + my $input = \"\";\n> + {\n> + binmode(STDIN);\n> + my $buffer;\n> + my $done = 0;\n> + while ( !$done ) {\n> + ( $done, $buffer ) = packet_bin_read();\n> + $input .= $buffer;\n> + }\n> + }\n\nAll right.\n\n> +\n> + my $output;\n> + if ( $command eq \"clean\" ) {\n> + ### Perform clean here ###\n> + $output = $input;\n\nPerhaps we should also mention here how to handle errors\n(the \"status=error\" and \"status=abort\" both for upfront\nand partial contents case).\n\n> + }\n> + elsif ( $command eq \"smudge\" ) {\n> + ### Perform smudge here ###\n> + $output = $input;\n\nSame as above.\n\n> + }\n> + else {\n> + die \"bad command '$command'\";\n> + }\n\nAll right.\n\n> +\n> + packet_txt_write(\"status=success\");\n> + packet_flush();\n> + while ( length($output) > 0 ) {\n> + my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n> + packet_bin_write($packet);\n> + if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n> + $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n> + }\n> + else {\n> + $output = \"\";\n> + }\n> + }\n> + packet_flush(); # flush content!\n\nAll right.\n\n> + packet_flush(); # empty list!\n\nThis is less \"empty list!\", and more keeping \"status=success\" unchanged.\n\n> +}\n> diff --git a/convert.c b/convert.c\n> index 597f561..bd66257 100644\n> --- a/convert.c\n> +++ b/convert.c\n\nI'll stop here, and I'll finish the review later.\n\nTo be continued,\n-- \nJakub Narębski\n\n"},{"id":"302687","messageId":"6D8B988C-8E54-4AA6-980C-A6BA40983D88@gmail.com","threadId":"44122","inReplyTo":"77315FC2-47F3-433A-8D70-5497FB04CBBE@gmail.com","subject":"Re: [PATCH v8 07/11] pkt-line: add functions to read/write flush terminated packet streams","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-27T08:14:16Z","receivedAt":"2016-09-27T08:14:28Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 26 Sep 2016, at 22:23, Lars Schneider wrote:\n\n> \n> On 25 Sep 2016, at 15:46, Jakub Narębski wrote:\n> \n>> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n>>> From: Lars Schneider \n> \n> \n>>> +\t\tstrbuf_grow(sb_out, PKTLINE_DATA_MAXLEN+1);\n>>> +\t\tpaket_len = packet_read(fd_in, NULL, NULL,\n>>> +\t\t\tsb_out->buf + sb_out->len, PKTLINE_DATA_MAXLEN+1, options);\n>> \n>> A question (which perhaps was answered during the development of this\n>> patch series): why is this +1 in PKTLINE_DATA_MAXLEN+1 here?\n> \n> Nice catch. I think this is wrong:\n> https://github.com/git/git/blob/6fe1b1407ed91823daa5d487abe457ff37463349/pkt-line.c#L196\n> \n> It should be \"if (len > size)\" ... then we don't need the \"+1\" here.\n> (but I need to think a bit more about this)\n\nAfter looking at it with fresh eyes I think the existing code is probably correct,\nbut maybe a bit confusing.\n\npacket_read() adds a '\\0' at the end of the destination buffer:\nhttps://github.com/git/git/blob/21f862b498925194f8f1ebe8203b7a7df756555b/pkt-line.c#L206\n\nThat is why the destination buffer needs to be one byte larger than the expected content.\n\nHowever, in this particular case that wouldn't be necessary because the destination\nbuffer is a 'strbuf' that allocates an extra byte for '\\0' at the end. But we are not\nsupposed to write to this extra byte:\nhttps://github.com/git/git/blob/21f862b498925194f8f1ebe8203b7a7df756555b/strbuf.h#L25-L31\n\n\nI see two options:\n\n\n(1) I leave the +1 as is and add a comment why the extra byte is necessary.\n\n Pro: No change in existing code necessary\n Con: The destination buffer has two '\\0' at the end.\n\n\n(2) I add an option PACKET_READ_DISABLE_NUL_TERMINATION. If the option is\n set then no '\\0' byte is added to the end.\n\n Pro: Correct solution, no byte wasted.\n Con: Change in existing code required.\n\n\nAny preference?\n\n\nThanks,\nLars"},{"id":"302689","messageId":"20160927083939.par2ea5nqjrtkkjs@sigill.intra.peff.net","threadId":"44122","inReplyTo":"4CF26590-7B88-4DB6-B6D4-A6CFE0FD6ABF@gmail.com","subject":"Re: [PATCH v8 06/11] pkt-line: add packet_write_gently()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-09-27T08:39:39Z","receivedAt":"2016-09-27T08:39:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 26, 2016 at 09:21:10PM +0200, Lars Schneider wrote:\n\n> On 25 Sep 2016, at 13:26, Jakub Narębski wrote:\n> \n> > W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> >> From: Lars Schneider \n> >> ...\n> >> \n> >> +static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n> > \n> > I'm not sure what naming convention the rest of Git uses, but isn't\n> > it more like '*data' rather than '*buf' here?\n> \n> pkt-line seems to use 'buf' or 'buffer' for everything else.\n\nI do not think we have definite rules, but I would generally expect to\nsee \"data\" as an opaque thing (e.g., passing \"void *data\" to callbacks).\n\"buf\" or \"buffer\" makes sense here, but I don't think it really matters\nthat much either way.\n\n> >> +\tstatic char packet_write_buffer[LARGE_PACKET_MAX];\n> > \n> > I think there should be warning (as a comment before function\n> > declaration, or before function definition), that packet_write_gently()\n> > is not thread-safe (nor reentrant, but the latter does not matter here,\n> > I think).\n> > \n> > Thread-safe vs reentrant: http://stackoverflow.com/a/33445858/46058\n> > \n> > This is not something terribly important; I guess git code has tons\n> > of functions not marked as thread-unsafe...\n> \n> I agree that the function is not thread-safe. However, I can't find \n> an example in the Git source that marks a function as not thread-safe.\n> Unless is it explicitly stated in the coding guidelines I would prefer\n> not to start way to mark functions.\n\nI'd agree. A large number of functions in git are not reentrant, and I\nwould not want to give the impression that those missing a warning are\nsafe to use.\n\n> >> +\tif (size > sizeof(packet_write_buffer) - 4) {\n> > \n> > First, wouldn't the following be more readable:\n> > \n> > +\tif (size + 4 > LARGE_PACKET_MAX) {\n> \n> Peff suggested that here:\n> http://public-inbox.org/git/20160810132814.gqnipsdwyzjmuqjy@sigill.intra.peff.net/\n\nThere is a good reason to do size checks as a subtraction from a known\nquantity: you can be sure that you are not introducing an overflow\n(e.g., Jakub's suggestion does the wrong thing when \"size\" is within 4\nbytes of its maximum value). That's unlikely in this case, but then so\nis the size exceeding LARGE_PACKET_MAX in the first place (arguably this\nshould be a die(\"BUG\"), because it is the caller's responsibility to\nsplit their packets.\n\n-Peff\n"},{"id":"302690","messageId":"20160927090031.jjb3dmhspbbnizja@sigill.intra.peff.net","threadId":"44122","inReplyTo":"6D8B988C-8E54-4AA6-980C-A6BA40983D88@gmail.com","subject":"Re: [PATCH v8 07/11] pkt-line: add functions to read/write flush terminated packet streams","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-09-27T09:00:32Z","receivedAt":"2016-09-27T09:00:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 27, 2016 at 10:14:16AM +0200, Lars Schneider wrote:\n\n> >>> +\t\tstrbuf_grow(sb_out, PKTLINE_DATA_MAXLEN+1);\n> >>> +\t\tpaket_len = packet_read(fd_in, NULL, NULL,\n> >>> +\t\t\tsb_out->buf + sb_out->len, PKTLINE_DATA_MAXLEN+1, options);\n> [...]\n> After looking at it with fresh eyes I think the existing code is probably correct,\n> but maybe a bit confusing.\n> \n> packet_read() adds a '\\0' at the end of the destination buffer:\n> https://github.com/git/git/blob/21f862b498925194f8f1ebe8203b7a7df756555b/pkt-line.c#L206\n> \n> That is why the destination buffer needs to be one byte larger than the expected content.\n> \n> However, in this particular case that wouldn't be necessary because the destination\n> buffer is a 'strbuf' that allocates an extra byte for '\\0' at the end. But we are not\n> supposed to write to this extra byte:\n> https://github.com/git/git/blob/21f862b498925194f8f1ebe8203b7a7df756555b/strbuf.h#L25-L31\n\nRight. The allocation happens as part of strbuf_grow(), but whatever\nfills the buffer is expected to write the actual NUL (either manually,\nor by calling strbuf_setlen().\n\nI see the bit you quoted warns not to touch the extra byte yourself,\nthough I wonder if that is a bit heavy-handed (I guess it would matter\nif we moved the extra 1-byte growth into strbuf_setlen(), but I find\nthat a rather unlikely change).\n\nThat being said, why don't you just use LARGE_PACKET_MAX here? It is\nalready the accepted size for feeding to packet_read(), and we know it\nhas enough space to hold a NUL terminator. Yes, we may over-allocate by\n4 bytes, but that isn't really relevant. Strbufs over-allocate anyway.\nSo just:\n\n for (;;) {\n strbuf_grow(sb_out, LARGE_PACKET_MAX);\n packet_len = packet_read(fd_in, NULL, NULL,\n\t sb_out->buf + sb_out->len, LARGE_PACKET_MAX,\n\t\t\t\t options);\n if (packet_len <= 0)\n break;\n /*\n * no need for strbuf_setlen() here; packet_read always adds a\n * NUL terminator.\n */\n sb_out->len += packet_len;\n }\n\nYou _could_ make the final line of the loop use strbuf_setlen(); it\nwould just overwrite something we already know is a NUL (and we know\nthat no extra allocation is necessary).\n\nAlso, using LARGE_PACKET_MAX fixes the fact that this patch is using\nPKTLINE_DATA_MAXLEN before it is actually defined. :)\n\nYou might want to occasionally run:\n\n git rebase -x make\n\nto make sure all of your incremental steps are valid (or even \"make\ntest\" if you are extremely patient; I often do that once after a big\nround of complex interactive-rebase reordering).\n\n> I see two options:\n> \n> \n> (1) I leave the +1 as is and add a comment why the extra byte is necessary.\n> \n> Pro: No change in existing code necessary\n> Con: The destination buffer has two '\\0' at the end.\n> \n> \n> (2) I add an option PACKET_READ_DISABLE_NUL_TERMINATION. If the option is\n> set then no '\\0' byte is added to the end.\n> \n> Pro: Correct solution, no byte wasted.\n> Con: Change in existing code required.\n> \n> \n> Any preference?\n\nOf the two, I prefer (1), though I like what I suggested above even more\n(big surprise, I know).\n\n-Peff\n"},{"id":"302695","messageId":"AF15F1E5-4DA0-459F-84EE-EDCB5FA23EF7@gmail.com","threadId":"44122","inReplyTo":"20160927090031.jjb3dmhspbbnizja@sigill.intra.peff.net","subject":"Re: [PATCH v8 07/11] pkt-line: add functions to read/write flush terminated packet streams","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-27T12:10:50Z","receivedAt":"2016-09-27T12:11:12Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Sep 2016, at 11:00, Jeff King wrote:\n> \n> On Tue, Sep 27, 2016 at 10:14:16AM +0200, Lars Schneider wrote:\n> \n>>>>> +\t\tstrbuf_grow(sb_out, PKTLINE_DATA_MAXLEN+1);\n>>>>> +\t\tpaket_len = packet_read(fd_in, NULL, NULL,\n>>>>> +\t\t\tsb_out->buf + sb_out->len, PKTLINE_DATA_MAXLEN+1, options);\n>> [...]\n>> After looking at it with fresh eyes I think the existing code is probably correct,\n>> but maybe a bit confusing.\n>> \n>> packet_read() adds a '\\0' at the end of the destination buffer:\n>> https://github.com/git/git/blob/21f862b498925194f8f1ebe8203b7a7df756555b/pkt-line.c#L206\n>> \n>> That is why the destination buffer needs to be one byte larger than the expected content.\n>> \n>> However, in this particular case that wouldn't be necessary because the destination\n>> buffer is a 'strbuf' that allocates an extra byte for '\\0' at the end. But we are not\n>> supposed to write to this extra byte:\n>> https://github.com/git/git/blob/21f862b498925194f8f1ebe8203b7a7df756555b/strbuf.h#L25-L31\n> \n> Right. The allocation happens as part of strbuf_grow(), but whatever\n> fills the buffer is expected to write the actual NUL (either manually,\n> or by calling strbuf_setlen().\n> \n> I see the bit you quoted warns not to touch the extra byte yourself,\n> though I wonder if that is a bit heavy-handed (I guess it would matter\n> if we moved the extra 1-byte growth into strbuf_setlen(), but I find\n> that a rather unlikely change).\n> \n> That being said, why don't you just use LARGE_PACKET_MAX here? It is\n> already the accepted size for feeding to packet_read(), and we know it\n> has enough space to hold a NUL terminator. Yes, we may over-allocate by\n> 4 bytes, but that isn't really relevant. Strbufs over-allocate anyway.\n\nTBH in that case I would prefer the \"PKTLINE_DATA_MAXLEN+1\" solution with\nan additional comment explaining \"+1\".\n\nWould that be OK for you?\n\nI am not worried about the extra 4 bytes. I am worried that we make it harder\nto see what is going on if we use LARGE_PACKET_MAX.\n\n\n> So just:\n> \n> for (;;) {\n> strbuf_grow(sb_out, LARGE_PACKET_MAX);\n> packet_len = packet_read(fd_in, NULL, NULL,\n> \t sb_out->buf + sb_out->len, LARGE_PACKET_MAX,\n> \t\t\t\t options);\n> if (packet_len <= 0)\n> break;\n> /*\n> * no need for strbuf_setlen() here; packet_read always adds a\n> * NUL terminator.\n> */\n> sb_out->len += packet_len;\n> }\n> \n> You _could_ make the final line of the loop use strbuf_setlen(); it\n> would just overwrite something we already know is a NUL (and we know\n> that no extra allocation is necessary).\n> \n> Also, using LARGE_PACKET_MAX fixes the fact that this patch is using\n> PKTLINE_DATA_MAXLEN before it is actually defined. :)\n\nYeah, I noticed that too and fixed it in v9 :-) Thanks for the reminder!\n\n\n> You might want to occasionally run:\n> \n> git rebase -x make\n> \n> to make sure all of your incremental steps are valid (or even \"make\n> test\" if you are extremely patient; I often do that once after a big\n> round of complex interactive-rebase reordering).\n\nThat is a good suggestion. I'll add that to my \"tool-belt\" :-)\n\n\nThanks,\nLars\n\n"},{"id":"302696","messageId":"20160927121336.7ftrqw2242tetcf3@sigill.intra.peff.net","threadId":"44122","inReplyTo":"AF15F1E5-4DA0-459F-84EE-EDCB5FA23EF7@gmail.com","subject":"Re: [PATCH v8 07/11] pkt-line: add functions to read/write flush terminated packet streams","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-09-27T12:13:36Z","receivedAt":"2016-09-27T12:13:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 27, 2016 at 02:10:50PM +0200, Lars Schneider wrote:\n\n> > That being said, why don't you just use LARGE_PACKET_MAX here? It is\n> > already the accepted size for feeding to packet_read(), and we know it\n> > has enough space to hold a NUL terminator. Yes, we may over-allocate by\n> > 4 bytes, but that isn't really relevant. Strbufs over-allocate anyway.\n> \n> TBH in that case I would prefer the \"PKTLINE_DATA_MAXLEN+1\" solution with\n> an additional comment explaining \"+1\".\n> \n> Would that be OK for you?\n> \n> I am not worried about the extra 4 bytes. I am worried that we make it harder\n> to see what is going on if we use LARGE_PACKET_MAX.\n\nI guess I don't feel to strongly either way. My interest in\nLARGE_PACKET_MAX is mostly that this is how all the rest of the\npacket_read() callers behave.\n\n-Peff\n"},{"id":"302711","messageId":"2a604438-b6cd-876d-0ec2-90027dea99b9@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-12-larsxschneider@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-27T15:37:53Z","receivedAt":"2016-09-27T15:38:06Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Part second of the review of 11/11.\n\nW dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n\n> diff --git a/contrib/long-running-filter/example.pl b/contrib/long-running-filter/example.pl\n> new file mode 100755\n> index 0000000..c13a631\n> --- /dev/null\n> +++ b/contrib/long-running-filter/example.pl\n[...]\n> +( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n> +( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n> +( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n\nWhat I would like to see here is some kind of packet_read_list()\nor packet_txt_read_list() that reads until flush packet or EOF,\nand returns list of chomp-ed lines (without LF terminator).\n\nThen you can examine those lines:\n\n my @lines = packet_read_list();\n $lines[0] eq \"git-filter-client\" or die \"bad initialization: '$lines[0]'\";\n grep { $_ eq \"version=2\" } @lines or die \"bad version: version=2 not found\";\n\nNote: I have not checked that I got operator precedence right.\n\n> +\n> +packet_txt_write(\"git-filter-server\");\n> +packet_txt_write(\"version=2\");\n\nHere using packet_write_list() would help us to not forget to\nsend the flush packet:\n\n packet_write_list(\n \t\"git-filter-server\",\n \t\"version=2\"\n );\n\n[...]\n> diff --git a/convert.c b/convert.c\n> index 597f561..bd66257 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -3,6 +3,7 @@\n> #include \"run-command.h\"\n> #include \"quote.h\"\n> #include \"sigchain.h\"\n> +#include \"pkt-line.h\"\n> \n> /*\n> * convert.c - convert a file when checking it out and checking it in.\n> @@ -442,7 +443,7 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n> \treturn (write_err || status);\n> }\n> \n> -static int apply_filter(const char *path, const char *src, size_t len, int fd,\n> +static int apply_single_file_filter(const char *path, const char *src, size_t len, int fd,\n> struct strbuf *dst, const char *cmd)\n> {\n> \t/*\n> @@ -456,12 +457,6 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n> \tstruct async async;\n> \tstruct filter_params params;\n> \n> -\tif (!cmd || !*cmd)\n> -\t\treturn 0;\n> -\n> -\tif (!dst)\n> -\t\treturn 1;\n> -\n> \tmemset(&async, 0, sizeof(async));\n> \tasync.proc = filter_buffer_or_fd;\n> \tasync.data = ¶ms;\n\nI have reordered a few chunks of this patch to make it easier to\nsee what happens here, and to review this part of patch.\n\n[moved here from further in the patch]\n> +static int apply_filter(const char *path, const char *src, size_t len,\n> + int fd, struct strbuf *dst, struct convert_driver *drv,\n> + const unsigned int wanted_capability)\n> +{\n> +\tconst char *cmd = NULL;\n> +\n> +\tif (!drv)\n> +\t\treturn 0;\n> +\n> +\tif (!dst)\n> +\t\treturn 1;\n\nTo reduce the size of this patch (which is not yet of the size that\nwould make vger reject the email :-/), perhaps this split into \napply_single_file_filter() and apply_filter(), without yet adding\napply_multi_file_filter(). The apply_filter() would be semi-simple\nwrapper, with the same signature as above.\n\n> +\n> +\tif (!drv->process && (CAP_CLEAN & wanted_capability) && drv->clean)\n\nThis is just a very minor nitpicking, but wouldn't it be easier\nto read with those checks reordered?\n\n +\tif ((wanted_capability & CAP_CLEAN) && !drv->process && drv->clean)\n\nThat is: if we want `clean` capability, and `process` is not set,\nand there is `clean` filter.\n\nThough the following would also work...\n\n +\tif ((wanted_capability & CAP_CLEAN) && !drv->process)\n\n> +\t\tcmd = drv->clean;\n> +\telse if (!drv->process && (CAP_SMUDGE & wanted_capability) && drv->smudge)\n> +\t\tcmd = drv->smudge;\n> +\n> +\tif (cmd && *cmd)\n\n... thanks to this check (the 'cmd' part, which we need to check anyway).\n\n'cmd = drv->clean', then 'if (cmd)' is the same as 'if (drv->clean)',\nthen 'cmd = drv->clean', then 'if (cmd)', isn't it.\n\nNot sure if it would be more readable, or less readable.\n\nCAP_CLEAN and CAP_SMUDGE are, in theory, mutually exclusive. Note that\nthe above order prefers `smudge` to `clean` if both given, while in other\nplaces we prefer `clean` to `smudge` if both given.\n\n\n> +\t\treturn apply_single_file_filter(path, src, len, fd, dst, cmd);\n> +\telse if (drv->process && *drv->process)\n> +\t\treturn apply_multi_file_filter(path, src, len, fd, dst, drv->process, wanted_capability);\n> +\n> +\treturn 0;\n> +}\n\nNice and clean wrapper.\n\n> +\n\n[moved here from further in the patch]\n> @@ -839,7 +1140,7 @@ int would_convert_to_git_filter_fd(const char *path)\n> \tif (!ca.drv->required)\n> \t\treturn 0;\n> \n> -\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n> +\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN);\n> }\n> \n> const char *get_convert_attr_ascii(const char *path)\n\nThis would also be a part of move adding apply_single_file_filter()\nand converting apply_filter() to wrapper around single / multi file\nfilters.\n\n> @@ -493,14 +488,317 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n> \treturn !err;\n> }\n> \n> +#define CAP_CLEAN (1u<<0)\n> +#define CAP_SMUDGE (1u<<1)\n\nIt's a pity that ANSI C does not include binary constants, like e.g.\nmodern Perl, that is '0b0001u' and '0b0010u'; we could use hexadecimal\nconstants '0x01u' and '0x02u', but perhaps the above is more readable,\nand as performant.\n\n> +\n> +struct cmd2process {\n> +\tstruct hashmap_entry ent; /* must be the first member! */\n> +\tunsigned int supported_capabilities;\n> +\tconst char *cmd;\n> +\tstruct child_process process;\n> +};\n> +\n> +static int cmd_process_map_initialized;\n> +static struct hashmap cmd_process_map;\n> +\n> +static int cmd2process_cmp(const struct cmd2process *e1,\n> + const struct cmd2process *e2,\n> + const void *unused)\n> +{\n> +\treturn strcmp(e1->cmd, e2->cmd);\n> +}\n> +\n> +static struct cmd2process *find_multi_file_filter_entry(struct hashmap *hashmap, const char *cmd)\n> +{\n> +\tstruct cmd2process key;\n> +\thashmap_entry_init(&key, strhash(cmd));\n> +\tkey.cmd = cmd;\n> +\treturn hashmap_get(hashmap, &key, NULL);\n> +}\n\n\nAll right, basic hashmap for the list of command processes, here\nso that we can find correct driver for current file, and reuse it\nif it was started already.\n\nI see that git code does not use /* ............ */ etc to separate\nsubsections / subparts of a file, so I won't ask for it ;-(\n\n> +\n> +static void kill_multi_file_filter(struct hashmap *hashmap, struct cmd2process *entry)\n> +{\n> +\tif (!entry)\n> +\t\treturn;\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\t/*\n> +\t * We kill the filter most likely because an error happened already.\n> +\t * That's why we are not interested in any error code here.\n> +\t */\n\nGood explanation.\n\n> +\tclose(entry->process.in);\n> +\tclose(entry->process.out);\n> +\tsigchain_pop(SIGPIPE);\n> +\tfinish_command(&entry->process);\n> +\thashmap_remove(hashmap, entry, NULL);\n> +\tfree(entry);\n> +}\n\nThat's more 'kill_and_remove_...', but that would make too long\nfunction name.\n\nSmall and readable. Nice.\n\n> +\n> +static int packet_write_list(int fd, const char *line, ...)\n> +{\n> +\tva_list args;\n> +\tint err;\n> +\tva_start(args, line);\n> +\tfor (;;) {\n> +\t\tif (!line)\n> +\t\t\tbreak;\n> +\t\tif (strlen(line) > PKTLINE_DATA_MAXLEN)\n\nHere we see that having PKTLINE_DATA_MAXLEN (or LARGE_PACKET_DATA_MAX)\nconstant is quite useful.\n\n> +\t\t\treturn -1;\n> +\t\terr = packet_write_fmt_gently(fd, \"%s\\n\", line);\n\nI wonder if adding the fact that we are writing text packets\nto function name would be worth it. Nah. Also, it is file-local\n(static) function.\n\n> +\t\tif (err)\n> +\t\t\treturn err;\n> +\t\tline = va_arg(args, const char*);\n> +\t}\n> +\tva_end(args);\n> +\treturn packet_flush_gently(fd);\n> +}\n\nNice abstraction.\n\n> +\n> +static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n> +{\n> +\tint err;\n> +\tstruct cmd2process *entry;\n> +\tstruct child_process *process;\n> +\tconst char *argv[] = { cmd, NULL };\n> +\tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n> +\tchar *cap_buf;\n> +\tconst char *cap_name;\n> +\n> +\tentry = xmalloc(sizeof(*entry));\n> +\thashmap_entry_init(entry, strhash(cmd));\n> +\tentry->cmd = cmd;\n> +\tentry->supported_capabilities = 0;\n> +\tprocess = &entry->process;\n> +\n> +\tchild_process_init(process);\n> +\tprocess->argv = argv;\n> +\tprocess->use_shell = 1;\n> +\tprocess->in = -1;\n> +\tprocess->out = -1;\n> +\n> +\tif (start_command(process)) {\n> +\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n> +\t\tkill_multi_file_filter(hashmap, entry);\n> +\t\treturn NULL;\n> +\t}\n\nI guess there is a reason why we init hashmap entry, try to start\nexternal process, then kill entry of unable to start, instead of\ntrying to start external process, and adding hashmap entry when\nwe succeed?\n\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n\nI guess that this is here to handle errors writing to filter\nby ourself, isn't it?\n\n> +\n> +\terr = packet_write_list(process->in, \"git-filter-client\", \"version=2\", NULL);\n> +\tif (err)\n> +\t\tgoto done;\n\nUgh, error / exception handling in C.\n\n> +\n> +\terr = strcmp(packet_read_line(process->out, NULL), \"git-filter-server\");\n> +\tif (err) {\n> +\t\terror(\"external filter '%s' does not support long running filter protocol\", cmd);\n> +\t\tgoto done;\n> +\t}\n> +\terr = strcmp(packet_read_line(process->out, NULL), \"version=2\");\n> +\tif (err)\n\nWe could have described the error here better.\n\n +\t\terror(\"external filter '%s' does not support filter protocol version 2\", cmd);\n\nBut this is probably not necessary; it should be rare to find\nfilter process that support the protocol halfway right.\n\n> +\t\tgoto done;\n\nI guess this would get more complicated if/when there is need\nfor new version of the protocol.\n\nShouldn't we read flush packet here? Ah, sorry, we know that we\nshould get only two lines from the `process` filter driver, and\nnot variable number of lines, so there is no need to flush here.\n\nDisregard my comments about lack of flush packet in the example\nof long running filter script. Well, unless the protocol itself\nwould get adjusted to always use flush packet to terminate set\nof metadata lines, even if number of lines is fixed.\n\n> +\n> +\terr = packet_write_list(process->in, \"clean=true\", \"smudge=true\", NULL);\n\nSo I see that Git sends all capabilities it supports, not only\nthose that given git command needs (which might be hard to find\nout).\n\nIf it were possible at this point of code for Git to know, for\nexample, that it would only do `clean` operation, shouldn't it\nwrite \"clean=true\", \"smudge=false\"? ;-PPP\n\nNote that this \"=true\" is totally spurious. Maybe \"capability=clean\",\nor just \"clean\" would make a better protocol?\n\n> +\n> +\tfor (;;) {\n> +\t\tcap_buf = packet_read_line(process->out, NULL);\n> +\t\tif (!cap_buf)\n> +\t\t\tbreak;\n> +\t\tstring_list_split_in_place(&cap_list, cap_buf, '=', 1);\n> +\n> +\t\tif (cap_list.nr != 2 || strcmp(cap_list.items[1].string, \"true\"))\n> +\t\t\tcontinue;\n> +\n> +\t\tcap_name = cap_list.items[0].string;\n> +\t\tif (!strcmp(cap_name, \"clean\")) {\n> +\t\t\tentry->supported_capabilities |= CAP_CLEAN;\n> +\t\t} else if (!strcmp(cap_name, \"smudge\")) {\n> +\t\t\tentry->supported_capabilities |= CAP_SMUDGE;\n> +\t\t} else {\n> +\t\t\twarning(\n> +\t\t\t\t\"external filter '%s' requested unsupported filter capability '%s'\",\n> +\t\t\t\tcmd, cap_name\n> +\t\t\t);\n> +\t\t}\n> +\n> +\t\tstring_list_clear(&cap_list, 0);\n> +\t}\n\nI guess there is a reason why it was not extracted into helper\nfunction?\n\nWell, both because handling of variable-length response, where\nmultiple lines must be analyzed, happens only once, and also\nbecause returning list of variable-length strings in C is hard\n(alloca? string_list?).\n\n> +\n> +done:\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tif (err || errno == EPIPE) {\n> +\t\terror(\"initialization for external filter '%s' failed\", cmd);\n> +\t\tkill_multi_file_filter(hashmap, entry);\n> +\t\treturn NULL;\n> +\t}\n\nGood.\n\n> +\n> +\thashmap_add(hashmap, entry);\n> +\treturn entry;\n> +}\n> +\n> +static void read_multi_file_filter_values(int fd, struct strbuf *status) {\n\nThis is more\n\n +static void read_multi_file_filter_status(int fd, struct strbuf *status) {\n\nIt doesn't read arbitrary values, it examines 'metadata' from\nfilter for \"status=\" lines.\n\n> +\tstruct strbuf **pair;\n\nShouldn't it be initialized to NULL, like in strbuf_split_buf()\ncode?\n\n> +\tchar *line;\n> +\tfor (;;) {\n> +\t\tline = packet_read_line(fd, NULL);\n> +\t\tif (!line)\n> +\t\t\tbreak;\n> +\t\tpair = strbuf_split_str(line, '=', 2);\n\nWhy, oh why, there is no Documentation/technical/api-strbuf.txt?\nWell, strbuf.h is really well commented... but perhaps not enough.\n\n> +\t\tif (pair[0] && pair[0]->len && pair[1]) {\n> +\t\t\tif (!strcmp(pair[0]->buf, \"status=\")) {\n> +\t\t\t\tstrbuf_reset(status);\n> +\t\t\t\tstrbuf_addbuf(status, pair[1]);\n> +\t\t\t}\n\nSo it is last status= line wins behavior?\n\n> +\t\t}\n\nShouldn't we free 'struct strbuf **pair', maybe allocated by the\nstrbuf_split_str() function, and reset to NULL?\n\n> +\t}\n> +}\n> +\n> +static int apply_multi_file_filter(const char *path, const char *src, size_t len,\n> + int fd, struct strbuf *dst, const char *cmd,\n> + const unsigned int wanted_capability)\n> +{\n> +\tint err;\n> +\tstruct cmd2process *entry;\n> +\tstruct child_process *process;\n> +\tstruct stat file_stat;\n> +\tstruct strbuf nbuf = STRBUF_INIT;\n\nThis name doesn't tell us much, but I guess there is precedence?\n\n> +\tstruct strbuf filter_status = STRBUF_INIT;\n> +\tchar *filter_type;\n> +\n> +\tif (!cmd_process_map_initialized) {\n> +\t\tcmd_process_map_initialized = 1;\n> +\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n> +\t\tentry = NULL;\n> +\t} else {\n> +\t\tentry = find_multi_file_filter_entry(&cmd_process_map, cmd);\n> +\t}\n> +\n> +\tfflush(NULL);\n\nWhy this fflush(NULL) is needed here?\n\n> +\n> +\tif (!entry) {\n> +\t\tentry = start_multi_file_filter(&cmd_process_map, cmd);\n> +\t\tif (!entry)\n> +\t\t\treturn 0;\n> +\t}\n> +\tprocess = &entry->process;\n\nAll right, we start process filter, or get existing instance.\n\n> +\n> +\tif (!(wanted_capability & entry->supported_capabilities))\n> +\t\treturn 0;\n\nIf filter doesn't support wanted capability, then Git just\nwouldn't filter. Looks good to me.\n\n> +\n> +\tif (CAP_CLEAN & wanted_capability)\n> +\t\tfilter_type = \"clean\";\n> +\telse if (CAP_SMUDGE & wanted_capability)\n> +\t\tfilter_type = \"smudge\";\n> +\telse\n> +\t\tdie(\"unexpected filter type\");\n\nThis should never happen; we should always request one of those\ncapabilities, and only one.\n\n> +\n> +\tif (fd >= 0 && !src) {\n> +\t\tif (fstat(fd, &file_stat) == -1)\n> +\t\t\treturn 0;\n> +\t\tlen = xsize_t(file_stat.st_size);\n> +\t}\n\nErrr... is it necessary? The protocol no longer provides size=\nhint, and neither uses such hint if provided.\n\n> +\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n\nRight, we want to handle errors ourself.\n\n> +\n> +\terr = strlen(filter_type) > PKTLINE_DATA_MAXLEN;\n> +\tif (err)\n> +\t\tgoto done;\n\nErrr... this should never happen. We control which capabilities\nwe pass, it can be only \"clean\" or \"smudge\", nothing else. Those\nwould always be shorter than PKTLINE_DATA_MAXLEN.\n\nNever mind that that is \"command=smudge\\n\" etc. that needs to\nbe shorter that PKTLINE_DATA_MAXLEN!\n\nSo, IMHO it should be at most assert, and needs to be corrected\nanyway.\n\n> +\n> +\terr = packet_write_fmt_gently(process->in, \"command=%s\\n\", filter_type);\n> +\tif (err)\n> +\t\tgoto done;\n> +\n> +\terr = strlen(path) > PKTLINE_DATA_MAXLEN;\n\nActually\n\n +\terr = strlen(path) > PKTLINE_DATA_MAXLEN - strlen(\"pathname=\\n\");\n\nThis version was chosen in the very unlikely case if\nstrlen(path) + strlen(\"pathname=\\n\") would overflow.\n\n +\terr = strlen(\"pathname=\") + strlen(path) + strlen(\"\\n\") > PKTLINE_DATA_MAXLEN;\n\n;-)\n\n> +\tif (err)\n> +\t\tgoto done;\n\nThis should never happen, PATH_MAX everywhere is much shorter\nthan PKTLINE_DATA_MAXLEN / LARGE_PACKET_MAX. Or is it?\n\nAnyway, we should probably explain or warn\n\n \t\terror(\"path name too long: '%s'\", path);\n\nThough if length of pathname is of the order of 2^16, I don't\nthink printing it would help :-)\n\n> +\n> +\terr = packet_write_fmt_gently(process->in, \"pathname=%s\\n\", path);\n> +\tif (err)\n> +\t\tgoto done;\n> +\n> +\terr = packet_flush_gently(process->in);\n> +\tif (err)\n> +\t\tgoto done;\n\nAll right, this list of values, currently composed of \"command=\"\nand \"pathname=\" - both of which are required, may be variable\nlength, so we need flush packet.\n\n> +\n> +\tif (fd >= 0)\n> +\t\terr = write_packetized_from_fd(fd, process->in);\n> +\telse\n> +\t\terr = write_packetized_from_buf(src, len, process->in);\n> +\tif (err)\n> +\t\tgoto done;\n\nLooks good, and I think it is better if the caller decided rather\nthat write_packetized(fd, src, len, process->in) deciding.\n\nNote for implementers: write in full, read in full, no streaming\nsupport (Git doesn't start to read filter output until it writes\nto filter in full). This is opposed to what `clean` and `smudge`\nfilters support.\n\n> +\n> +\tread_multi_file_filter_values(process->out, &filter_status);\n> +\terr = strcmp(filter_status.buf, \"success\");\n> +\tif (err)\n> +\t\tgoto done;\n> +\n> +\terr = read_packetized_to_buf(process->out, &nbuf) < 0;\n> +\tif (err)\n> +\t\tgoto done;\n> +\n> +\tread_multi_file_filter_values(process->out, &filter_status);\n> +\terr = strcmp(filter_status.buf, \"success\");\n\nLooks good to me (LGTM).\n\n> +\n> +done:\n> +\tsigchain_pop(SIGPIPE);\n> +\n> +\tif (err || errno == EPIPE) {\n> +\t\tif (!strcmp(filter_status.buf, \"error\")) {\n> +\t\t\t/* The filter signaled a problem with the file. */\n> +\t\t} else if (!strcmp(filter_status.buf, \"abort\")) {\n> +\t\t\t/*\n> +\t\t\t * The filter signaled a permanent problem. Don't try to filter\n> +\t\t\t * files with the same command for the lifetime of the current\n> +\t\t\t * Git process.\n> +\t\t\t */\n> +\t\t\t entry->supported_capabilities &= ~wanted_capability;\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * Something went wrong with the protocol filter.\n> +\t\t\t * Force shutdown and restart if another blob requires filtering!\n\nIs this exclamation mark '!' here necessary?\n\n> +\t\t\t */\n> +\t\t\terror(\"external filter '%s' failed\", cmd);\n> +\t\t\tkill_multi_file_filter(&cmd_process_map, entry);\n> +\t\t}\n\nLooks good. Three error conditions: resumable error from filter,\nfailure of filter (kill, would restart if necessary), and abort.\n\n> +\t} else {\n> +\t\tstrbuf_swap(dst, &nbuf);\n> +\t}\n> +\tstrbuf_release(&nbuf);\n> +\treturn !err;\n\nI guess this is for `filter..required` to handle correctly\nfilter error-ing out, or filter failing, while not aborting\nif filter simply doesn't support `clean` or `smudge` capability.\n\n> +}\n> +\n> static struct convert_driver {\n> \tconst char *name;\n> \tstruct convert_driver *next;\n> \tconst char *smudge;\n> \tconst char *clean;\n> +\tconst char *process;\n\nLGTM.\n\n> \tint required;\n> } *user_convert, **user_convert_tail;\n> \n\n[a section of chunk moved up]\n\n> static int read_convert_config(const char *var, const char *value, void *cb)\n> {\n> \tconst char *key, *name;\n> @@ -538,6 +836,9 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n> \tif (!strcmp(\"clean\", key))\n> \t\treturn git_config_string(&drv->clean, var, value);\n> \n> +\tif (!strcmp(\"process\", key))\n> +\t\treturn git_config_string(&drv->process, var, value);\n> +\n\nLGTM.\n\n> \tif (!strcmp(\"required\", key)) {\n> \t\tdrv->required = git_config_bool(var, value);\n> \t\treturn 0;\n\n[a chunk of diff moved up]\n\n> @@ -872,18 +1173,12 @@ int convert_to_git(const char *path, const char *src, size_t len,\n> struct strbuf *dst, enum safe_crlf checksafe)\n> {\n> \tint ret = 0;\n> -\tconst char *filter = NULL;\n\nAll right, this was (I think) moved into apply_filter()...\n\n> -\tint required = 0;\n\n...but this looks like just a removal of a temporary variable,\nwhich could have been done in a separate preparatory patch.\n\n> \tstruct conv_attrs ca;\n> \n> \tconvert_attrs(&ca, path);\n> -\tif (ca.drv) {\n> -\t\tfilter = ca.drv->clean;\n> -\t\trequired = ca.drv->required;\n> -\t}\n> \n> -\tret |= apply_filter(path, src, len, -1, dst, filter);\n> -\tif (!ret && required)\n> +\tret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN);\n> +\tif (!ret && ca.drv && ca.drv->required)\n> \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n\nLooks good. (And could be a part of patch adding apply_filter()\nas wrapper.)\n\n> \n> \tif (ret && dst) {\n> @@ -905,9 +1200,9 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n> \tconvert_attrs(&ca, path);\n> \n> \tassert(ca.drv);\n> -\tassert(ca.drv->clean);\n> +\tassert(ca.drv->clean || ca.drv->process);\n\nHmmm... asserts. Well, they were here.\n\n> \n> -\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))\n> +\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN))\n> \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n> \n\nLooks good. (And could be a part of patch adding apply_filter()\nas wrapper.)\n\n> \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n> @@ -919,15 +1214,9 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n> \t\t\t\t\t int normalizing)\n> {\n> \tint ret = 0, ret_filter = 0;\n> -\tconst char *filter = NULL;\n> -\tint required = 0;\n> \tstruct conv_attrs ca;\n> \n> \tconvert_attrs(&ca, path);\n> -\tif (ca.drv) {\n> -\t\tfilter = ca.drv->smudge;\n> -\t\trequired = ca.drv->required;\n> -\t}\n\nWell, this is the same change as a bit eaelier.\n\n> \n> \tret |= ident_to_worktree(path, src, len, dst, ca.ident);\n> \tif (ret) {\n> @@ -936,9 +1225,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n> \t}\n> \t/*\n> \t * CRLF conversion can be skipped if normalizing, unless there\n> -\t * is a smudge filter. The filter might expect CRLFs.\n> +\t * is a smudge or process filter (even if the process filter doesn't\n> +\t * support smudge). The filters might expect CRLFs.\n> \t */\n> -\tif (filter || !normalizing) {\n> +\tif ((ca.drv && (ca.drv->smudge || ca.drv->process)) || !normalizing) {\n> \t\tret |= crlf_to_worktree(path, src, len, dst, ca.crlf_action);\n> \t\tif (ret) {\n> \t\t\tsrc = dst->buf;\n> @@ -946,8 +1236,8 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n> \t\t}\n> \t}\n> \n> -\tret_filter = apply_filter(path, src, len, -1, dst, filter);\n> -\tif (!ret_filter && required)\n> +\tret_filter = apply_filter(path, src, len, -1, dst, ca.drv, CAP_SMUDGE);\n> +\tif (!ret_filter && ca.drv && ca.drv->required)\n> \t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n> \n> \treturn ret | ret_filter;\n\nLooks good to me. I understand ca.drv is checked so that ca.drv->required\nand/or ca.drv->smudge / ca.drv->clean / ca.drv->process can be safely\nchecked, isn't it.\n\n> @@ -1399,7 +1689,7 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n> \tstruct stream_filter *filter = NULL;\n> \n> \tconvert_attrs(&ca, path);\n> -\tif (ca.drv && (ca.drv->smudge || ca.drv->clean))\n> +\tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n\nLGTM, understandable change.\n\n> \t\treturn NULL;\n> \n> \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\n> diff --git a/pkt-line.h b/pkt-line.h\n> index 6df8449..3d873f3 100644\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -86,6 +86,7 @@ ssize_t read_packetized_to_buf(int fd_in, struct strbuf *sb_out);\n> \n> #define DEFAULT_PACKET_MAX 1000\n> #define LARGE_PACKET_MAX 65520\n> +#define PKTLINE_DATA_MAXLEN (LARGE_PACKET_MAX - 4)\n\nWhat the... didn't you use PKTLINE_DATA_MAXLEN in one of\nearlier patches in this series? How this even...?\n\n> extern char packet_buffer[LARGE_PACKET_MAX];\n> \n> #endif\n> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n> index dc50938..210c4f6 100755\n> --- a/t/t0021-conversion.sh\n> +++ b/t/t0021-conversion.sh\n\nI'll stop here, and I'll finish the review later.\n\nTo be continued,\n-- \nJakub Narębski\n\n"},{"id":"302748","messageId":"5e6a95a5-e744-1e9d-05c6-3700c595af36@gmail.com","threadId":"44122","inReplyTo":"20160927083939.par2ea5nqjrtkkjs@sigill.intra.peff.net","subject":"Re: [PATCH v8 06/11] pkt-line: add packet_write_gently()","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-27T19:33:32Z","receivedAt":"2016-09-27T19:33:59Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 27.09.2016 o 10:39, Jeff King pisze:\n> On Mon, Sep 26, 2016 at 09:21:10PM +0200, Lars Schneider wrote:\n> \n>> On 25 Sep 2016, at 13:26, Jakub Narębski wrote:\n>>\n>>> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n>>>> From: Lars Schneider \n>>>> ...\n>>>>\n>>>> +static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n>>>\n>>> I'm not sure what naming convention the rest of Git uses, but isn't\n>>> it more like '*data' rather than '*buf' here?\n>>\n>> pkt-line seems to use 'buf' or 'buffer' for everything else.\n> \n> I do not think we have definite rules, but I would generally expect to\n> see \"data\" as an opaque thing (e.g., passing \"void *data\" to callbacks).\n> \"buf\" or \"buffer\" makes sense here, but I don't think it really matters\n> that much either way.\n\nTrue.\n\n>>>> +\tstatic char packet_write_buffer[LARGE_PACKET_MAX];\n>>>\n>>> I think there should be warning (as a comment before function\n>>> declaration, or before function definition), that packet_write_gently()\n>>> is not thread-safe (nor reentrant, but the latter does not matter here,\n>>> I think).\n>>>\n>>> Thread-safe vs reentrant: http://stackoverflow.com/a/33445858/46058\n>>>\n>>> This is not something terribly important; I guess git code has tons\n>>> of functions not marked as thread-unsafe...\n>>\n>> I agree that the function is not thread-safe. However, I can't find \n>> an example in the Git source that marks a function as not thread-safe.\n>> Unless is it explicitly stated in the coding guidelines I would prefer\n>> not to start way to mark functions.\n\nThere is *one* example: \"fill_textconv is not remotely thread-safe;\"\ncomment in grep.c, but not in diff.{c,h} where it is declared/defined.\n\nAlso, it is static function; we should know if it is thread-safe\nor not.\n\nI am thinking about supporting streaming in the future, and perhaps\nalso running different filter drivers (for different files) in parallel.\nI guess that using \"static __thread char packet_write_buffer[...]\"\nis out of question (still not reentrant)?\n\n> \n> I'd agree. A large number of functions in git are not reentrant, and I\n> would not want to give the impression that those missing a warning are\n> safe to use.\n\nThe fact tha git code is undercommented and underdocumented does not\nmean that we should not add comments and documentation.\n\n> \n>>>> +\tif (size > sizeof(packet_write_buffer) - 4) {\n>>>\n>>> First, wouldn't the following be more readable:\n>>>\n>>> +\tif (size + 4 > LARGE_PACKET_MAX) {\n>>\n>> Peff suggested that here:\n>> http://public-inbox.org/git/20160810132814.gqnipsdwyzjmuqjy@sigill.intra.peff.net/\n> \n> There is a good reason to do size checks as a subtraction from a known\n> quantity: you can be sure that you are not introducing an overflow\n> (e.g., Jakub's suggestion does the wrong thing when \"size\" is within 4\n> bytes of its maximum value). That's unlikely in this case, but then so\n> is the size exceeding LARGE_PACKET_MAX in the first place (arguably this\n> should be a die(\"BUG\"), because it is the caller's responsibility to\n> split their packets.\n\nRight. I should train myself to watch for overflows.\n\n-- \nJakub Narębski\n\n"},{"id":"302841","messageId":"xmqq8tubitjs.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"20160920190247.82189-1-larsxschneider@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-28T21:49:27Z","receivedAt":"2016-09-28T21:49:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I suspect that you are preparing a reroll already, but the one that\nis sitting in 'pu' seems to be flaky in t/t0021 and I seem to see\noccasional failures from it.\n\nI didn't trace where the test goes wrong, but one easy mistake you\ncould make (I am not saying that is the reason of the failure) is to\nassume your filter will not be called under certain condition (like\nimmediately after you checked out from the index to the working\ntree), when the automated test goes fast enough and get you into a\n\"racy git\" situation---the filter may be asked to filter the\ncontents from the working tree again to re-validate what's there is\nstill what is in the index.\n\n\n"},{"id":"302852","messageId":"e8815048-212e-64c6-8811-9185f595aaf6@gmail.com","threadId":"44122","inReplyTo":"20160920190247.82189-12-larsxschneider@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-28T23:14:36Z","receivedAt":"2016-09-28T23:14:54Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Part third (and last) of the review of v8 11/11.\n\nW dniu 20.09.2016 o 21:02, larsxschneider@gmail.com napisał:\n[...]\n> diff --git a/contrib/long-running-filter/example.pl b/contrib/long-running-filter/example.pl\n> new file mode 100755\n> index 0000000..c13a631\n> --- /dev/null\n> +++ b/contrib/long-running-filter/example.pl\n[...]\n> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n> index dc50938..210c4f6 100755\n> --- a/t/t0021-conversion.sh\n> +++ b/t/t0021-conversion.sh\n\nOne thing that could have been done as yet another preparatory\npatch would be to modernize existing t/t0021-conversion.sh tests.\nFor example use here-doc instead of series of echo-s, use cp\nto copy files and not echo, etc.\n\n> @@ -31,7 +31,10 @@ test_expect_success setup '\n> \tcat test >test.i &&\n> \tgit add test test.t test.i &&\n> \trm -f test test.t test.i &&\n> -\tgit checkout -- test test.t test.i\n> +\tgit checkout -- test test.t test.i &&\n> +\n> +\techo \"content-test2\" >test2.o &&\n> +\techo \"content-test3 - subdir\" >\"test3 - subdir.o\"\n\nI see that you prepare here a few uncommitted files, but both\ntheir names and their contents leave much to be desired - you\ndon't know from the name and contents what they are for.\n\nAnd the '\"subdir\"' file which is not in subdirectory is\nespecially egregious.\n\n> '\n> \n> script='s/^\\$Id: \\([0-9a-f]*\\) \\$/\\1/p'\n> @@ -279,4 +282,364 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '\n> \ttest_line_count = 0 count\n> '\n> \n\nA small comment on parameters of this function would be nice.\n\n> +check_filter () {\n> +\trm -f rot13-filter.log actual.log &&\n> +\t\"$@\" 2> git_stderr.log &&\n> +\ttest_must_be_empty git_stderr.log &&\n> +\tcat >expected.log &&\n\nThis is too clever by half. Having a function that both tests\nthe behavior and prepares 'expected' file is too much.\n\nIn my opinion preparation of 'expected.log' file should be moved\nto another function or functions.\n\nAlso, if we are running sort on output, I think we should also\nrun sort on 'expected.log', so that what we write doesn't need to\nbe created sorted (so we don't have to sort expected lines by hand).\nOr maybe we should run the same transformation on rot13-filter.log\nand on the contents of expected.log.\n\n> +\tsort rot13-filter.log | uniq -c | sed \"s/^[ ]*//\" >actual.log &&\n> +\ttest_cmp expected.log actual.log\n> +}\n> +\n> +check_filter_count_clean () {\n> +\trm -f rot13-filter.log actual.log &&\n> +\t\"$@\" 2> git_stderr.log &&\n> +\ttest_must_be_empty git_stderr.log &&\n\nAll those functions (well, wait?) have common setup, which we can\nextract into separate shell function, I think. IMVHO.\n\n> +\tcat >expected.log &&\n> +\tsort rot13-filter.log | uniq -c | sed \"s/^[ ]*//\" |\n> +\t\tsed \"s/^\\([0-9]\\) IN: clean/x IN: clean/\" >actual.log &&\n> +\ttest_cmp expected.log actual.log\n> +}\n> +\n> +check_filter_ignore_clean () {\n> +\trm -f rot13-filter.log actual.log &&\n> +\t\"$@\" &&\n\nWhy we don't check for stderr here?\n\n> +\tcat >expected.log &&\n> +\tgrep -v \"IN: clean\" rot13-filter.log >actual.log &&\n> +\ttest_cmp expected.log actual.log\n> +}\n> +\n> +check_filter_no_call () {\n> +\trm -f rot13-filter.log &&\n> +\t\"$@\" 2> git_stderr.log &&\n> +\ttest_must_be_empty git_stderr.log &&\n> +\ttest_must_be_empty rot13-filter.log\n> +}\n> +\n\nA small comment on parameters of this function would be nice.\nAnd a comment what it does.\n\n> +check_rot13 () {\n> +\ttest_cmp \"$1\" \"$2\" &&\n> +\t./../rot13.sh <\"$1\" >expected &&\n\nWhy there is .. in this invocation?\n\n> +\tgit cat-file blob :\"$2\" >actual &&\n> +\ttest_cmp expected actual\n> +}\n> +\n> +test_expect_success PERL 'required process filter should filter data' '\n> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n> +\ttest_config_global filter.protocol.required true &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n\nDon't you think that creating a fresh test repository for each\nseparate test is a bit too much? I guess that you want for\neach and every test to be completely independent, but this setup\nand teardown is a bit excessive.\n\nOther tests in the same file (should we reuse the test, or use\nnew test file) do not use this method.\n\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\t\tgit add . &&\n> +\t\tgit commit . -m \"test commit\" &&\n> +\t\tgit branch empty &&\n\nErr... I think it would be better to name it 'empty-branch'\n(or 'almost-empty-branch', as it does include .gitattributes file).\nSee my mistake below (marked ...).\n\n> +\n> +\t\tcp ../test.o test.r &&\n> +\t\tcp ../test2.o test2.r &&\n\nWhat does this test2.o / test2.r file tests, that test.o / test.r\ndoesn't? The name doesn't tell us.\n\nWhy it is test.r, but test2.r? Why it isn't test1.r?\n\n> +\t\tmkdir testsubdir &&\n> +\t\tcp \"../test3 - subdir.o\" \"testsubdir/test3 - subdir.r\" &&\n\nWhy it needs to have different contents?\n\nI guess that you test two things here: file in a subdirectory,\nand file with spaces in names. Shouldn't it be better split\ninto two separate test files?\n\n> +\t\t>test4-empty.r &&\n\nYou test ordinary file, file in subdirectory, file with filename\ncontaining spaces, and an empty file.\n\nOther tests of single file `clean`/`smudge` filters use filename\nthat requires mangling; maybe we should use similar file?\n\n special=\"name with '\\''sq'\\'' and \\$x\" &&\n echo some test text >\"$special\" &&\n\nIn case of `process` filter, a special filename could look like\nthis:\n\n process_special=\"name=with equals and\\nembedded newlines\\n\" &&\n echo some test text >\"$process_special\" &&\n\n> +\n> +\t\tcheck_filter \\\n> +\t\t\tgit add . \\\n\nI assume that this kind of test is here also to check that\nwe are not regressing / backsliding, and we do not start to\nrun \"clean\" operation more than once per file for \"git add\",\nisn't it?\n\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n> +\t\t\t\t\t1 IN: clean test2.r 14 [OK] -- OUT: 14 . [OK]\n> +\t\t\t\t\t1 IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]\n> +\t\t\t\t\t1 IN: clean testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n> +\t\t\t\t\t1 START\n> +\t\t\t\t\t1 STOP\n> +\t\t\t\t\t1 wrote filter header\n> +\t\t\t\tEOF\n\nFirst, this indentation level confirms that the check_filter\nfunction is too clever by half, and that preparing expected.log\nfile should be a separate step.\n\nSecond, if we run \"sort\" on contents to be in expected.log, we\ncan write it in more natural, and less fragile way:\n\n +\t\tsort >expected.log <<-\\EOF &&\n +\t\t\t1 START\n +\t\t\t1 wrote filter header\n +\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n +\t\t\t1 IN: clean test2.r 14 [OK] -- OUT: 14 . [OK]\n +\t\t\t1 IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]\n +\t\t\t1 IN: clean testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n +\t\t\t1 STOP\n +\t\tEOF\n\nThird, why the filter even writes output size? It is no longer\npart of `process` filter driver protocol, and it makes test more\nfragile.\n\nIf we are to keep sizes, then to make test less fragile with\nrespect to changes in contents of tested files, we should use\nvariables containing file size:\n\n \t\ttest_r_size=$(wc -c test.r)\n \t\t...\n \t\tsort >expected.log <<-EOF &&\n \t\t...\n \t\t\t1 IN: clean test.r $test_r_size [OK] -- OUT: $test_r_size . [OK]\n\n> +\n> +\t\tcheck_filter_count_clean \\\n> +\t\t\tgit commit . -m \"test commit\" \\\n\nI guess that you use \"git commit .\" (not very visible this '.')\nin order to force cleaning of all files, isn't it?\n\nUse of *_count_clean function is here, from what I remember,\nbecause 'git commit .' sometimes call `clean` multiple times\nfor the same file (?), and sometimes it calls `smudge` (probably\nas part of some optimization?).\n\nI guess that fixing \"git commit\" so that calls clean operation\nat most once per file is left for a separate patch series; this\none is long enough and involved enough as it is.\n\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\tx IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n> +\t\t\t\t\tx IN: clean test2.r 14 [OK] -- OUT: 14 . [OK]\n> +\t\t\t\t\tx IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]\n> +\t\t\t\t\tx IN: clean testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n> +\t\t\t\t\t1 START\n> +\t\t\t\t\t1 STOP\n> +\t\t\t\t\t1 wrote filter header\n> +\t\t\t\tEOF\n> +\n> +\t\trm -f test?.r \"testsubdir/test3 - subdir.r\" &&\n\nWhy 'test?.r' when we are removing only 'test2.r'; why not be explicit?\n\n> +\n> +\t\tcheck_filter_ignore_clean \\\n> +\t\t\tgit checkout . \\\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\tSTART\n> +\t\t\t\t\twrote filter header\n> +\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n> +\t\t\t\t\tIN: smudge testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n\nAh, I see that there are no shenningans for `clean`\noperation, calling op multiple time for single file.\n\n> +\t\t\t\t\tSTOP\n> +\t\t\t\tEOF\n> +\n> +\t\tcheck_filter_ignore_clean \\\n> +\t\t\tgit checkout empty \\\n\n\nFirst, isn't it test4-empty.r? Trying to check out non-existent\nfile should not run filter, isn't it? How the heck this passed???\nThere is no branch 'empty'.\n\nSecond, the one-shot filter tests have empty-in-worktree and\nempty-in-repo files; why not reuse them?\n\n\nMy mistake, but the branch is named a bit strange.\n\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\tSTART\n> +\t\t\t\t\twrote filter header\n> +\t\t\t\t\tSTOP\n> +\t\t\t\tEOF\n\nWhy is even filter process invoked? If this is not expected, perhaps\nsimply ignore what checking out almost empty branch (one without any\nfiles marked for filtering) does.\n\nShouldn't we test_expect_failure no-call?\n\n> +\n> +\t\tcheck_filter_ignore_clean \\\n> +\t\t\tgit checkout master \\\n\nDoes this checks different code path than 'git checkout .'? For\nexample, does this test increase code coverage (e.g. as measured\nby gcov)? If not, then this test could be safely dropped.\n\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\tSTART\n> +\t\t\t\t\twrote filter header\n> +\t\t\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n> +\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n> +\t\t\t\t\tIN: smudge test4-empty.r 0 [OK] -- OUT: 0 [OK]\n> +\t\t\t\t\tIN: smudge testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n\nCan we assume that Git would pass files to filter in alphabetical\norder? This assumption might make the test unnecessary fragile.\n\n> +\t\t\t\t\tSTOP\n> +\t\t\t\tEOF\n> +\n> +\t\tcheck_rot13 ../test.o test.r &&\n> +\t\tcheck_rot13 ../test2.o test2.r &&\n> +\t\tcheck_rot13 \"../test3 - subdir.o\" \"testsubdir/test3 - subdir.r\"\n\nAll right.\n\n> +\t)\n> +'\n> +\n> +test_expect_success PERL 'required process filter should clean only and take precedence' '\n\nTrying to describe it better results in overly long description,\nwhich probably means that this test should be split into few\nsmaller ones:\n\n - `process` filter takes precedence over `clean` and/or `smudge`\n filters, regardless if it supports relevant (\"clean\" or \"smudge\")\n capability or not\n\n - `process` filter that includes only \"clean\" capability should\n clean only (be used only for 'clean' operation)\n\n - required process filter should do something (???)\n \n\n> +\ttest_config_global filter.protocol.clean ./../rot13.sh &&\n> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean\" &&\n> +\ttest_config_global filter.protocol.required true &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\t\tgit add . &&\n> +\t\tgit commit . -m \"test commit\" &&\n> +\t\tgit branch empty &&\n> +\n> +\t\tcp ../test.o test.r &&\n> +\n> +\t\tcheck_filter \\\n> +\t\t\tgit add . \\\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n> +\t\t\t\t\t1 START\n> +\t\t\t\t\t1 STOP\n> +\t\t\t\t\t1 wrote filter header\n> +\t\t\t\tEOF\n> +\n> +\t\tcheck_filter_count_clean \\\n> +\t\t\tgit commit . -m \"test commit\" \\\n\nIs this part really necessary? I think it duplicates what we\nhave tested earlier, and would not catch any new errors. Removing\nspurious/redundant tests results in faster testsuite, which is\nquite important.\n\n> +\t\t\t\t<<-\\EOF\n> +\t\t\t\t\tx IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n> +\t\t\t\t\t1 START\n> +\t\t\t\t\t1 STOP\n> +\t\t\t\t\t1 wrote filter header\n> +\t\t\t\tEOF\n\nAnd this test checks only the first one from the list.\nWell, actually the first part, without \"regardless if it\nsupports relevant ('clean' [...]) capability or not\".\n\n> +\t)\n> +'\n> +\n\nIn my opinion all functions should be placed at beginning,\nor even in separate file (if they are used in more than\none test).\n\n> +generate_test_data () {\n\nThe name is not good, it doesn't describe what kind of data\nwe want to generate.\n\n> +\tLEN=$1\n> +\tNAME=$2\n> +\ttest-genrandom end $LEN |\n\nWhy do you use 'end' as parameter to test-genrandom?\n\n> +\t\tperl -pe \"s/./chr((ord($&) % 26) + 97)/sge\" >../$NAME.file &&\n\nThose constants (26 and 97) are a bit cryptic; magical constants.\nI guess this is\n\n +\t\tperl -pe \"s/./chr((ord($&) % (ord('z') - ord('a') + 1) + ord('a'))/sge\" >../$NAME.file &&\n\nor\n\n +\t\tperl -pe \"s/./chr((ord($&) % 26 + ord('a'))/sge\" >../$NAME.file &&\n\nThat is, convert to a-z range (why not ASCII printable characters,\nthat is characters from ' ' / chr(32) to '~' / chr(126), which is\n95 characters instead of 26?)\n\nI guess this is so we can be sure that rot13 filter would work\n(note: the filter is defined for A-Za-z, not only a-z, never\nthe mind pass-through for other characters).\n\n> +\tcp ../$NAME.file . &&\n\nDo we re-generate this file each time?\n\n> +\t./../rot13.sh <../$NAME.file >../$NAME.file.rot13\n\nAnyway, I wonder if taking the last two lines out of the function\n(as they are not about _generating_ a file) would make it more\nreadable or not.\n\n> +}\n> +\n> +test_expect_success PERL 'required process filter should process multiple packets' '\n> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n> +\ttest_config_global filter.protocol.required true &&\n> +\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\t# Generate data that requires 3 packets\n> +\t\tPKTLINE_DATA_MAXLEN=65516 &&\n\nShouldn't this be set once per whole test?\n\n> +\n> +\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN )) 1pkt_1__ &&\n> +\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN + 1)) 2pkt_1+1 &&\n> +\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN * 2 - 1)) 2pkt_2-1 &&\n> +\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN * 2 )) 2pkt_2__ &&\n> +\t\tgenerate_test_data $(($PKTLINE_DATA_MAXLEN * 2 + 1)) 3pkt_2+1 &&\n\nLooks good to me.\n\n> +\n> +\t\techo \"*.file filter=protocol\" >.gitattributes &&\n> +\t\tcheck_filter \\\n> +\t\t\tgit add *.file .gitattributes \\\n\nShould it be shell expansion, or git expansion, that is\n\n \t\t\tgit add '*.file' .gitattributes\n\n\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\t1 IN: clean 1pkt_1__.file 65516 [OK] -- OUT: 65516 . [OK]\n> +\t\t\t\t\t1 IN: clean 2pkt_1+1.file 65517 [OK] -- OUT: 65517 .. [OK]\n> +\t\t\t\t\t1 IN: clean 2pkt_2-1.file 131031 [OK] -- OUT: 131031 .. [OK]\n> +\t\t\t\t\t1 IN: clean 2pkt_2__.file 131032 [OK] -- OUT: 131032 .. [OK]\n> +\t\t\t\t\t1 IN: clean 3pkt_2+1.file 131033 [OK] -- OUT: 131033 ... [OK]\n\nI think it would be better for those sizes to be calculated,\nnot entered by hand. Though in this case this doesn't matter\nmuch - it would always be this size.\n\n> +\t\t\t\t\t1 START\n> +\t\t\t\t\t1 STOP\n> +\t\t\t\t\t1 wrote filter header\n> +\t\t\t\tEOF\n> +\t\tgit commit . -m \"test commit\" &&\n\nIs this needed / necessary?\n\n> +\n> +\t\trm -f *.file &&\n> +\t\tgit checkout -- *.file &&\n\nIs this necessary? I guess this checks that it doesn't crash, but\nwe do not check that smudge operation works correctly, as we did\nfor clean.\n\n> +\n> +\t\tfor f in *.file\n> +\t\tdo\n> +\t\t\tgit cat-file blob :$f >actual &&\n> +\t\t\ttest_cmp ../$f.rot13 actual\n> +\t\tdone\n\nWasn't there helper function for this?\n\n> +\t)\n> +'\n> +\n> +test_expect_success PERL 'required process filter should with clean error should fail' '\n ^^^^^^ ^^^^^^\n\nErrr... what? You have 'should' twice here.\n\nAlso, does it matter that the error is during clean operation?\nWe don't test that error during smudge operation is handled in\nthe same way, do we?\n\n> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n\nDo we need to pass 'clean smudge', or does it provide both by\ndefault?\n\n> +\ttest_config_global filter.protocol.required true &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\n> +\t\tcp ../test.o test.r &&\n> +\t\techo \"this is going to fail\" >clean-write-fail.r &&\n> +\t\techo \"content-test3-subdir\" >test3.r &&\n> +\n> +\t\t# Note: There are three clean paths in convert.c we just test one here.\n\nWhat does this comment is about? What 'three clean paths'?\n\n> +\t\ttest_must_fail git add .\n> +\t)\n> +'\n> +\n> +test_expect_success PERL 'process filter should restart after unexpected write failure' '\n> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\n> +\t\tcp ../test.o test.r &&\n> +\t\tcp ../test2.o test2.r &&\n\nNote that the preparation step is almost the same, and we\nrepeat it over, and over, and over (no shell function for\nthis; and we always do full setup / teardown).\n\n> +\t\techo \"this is going to fail\" >smudge-write-fail.o &&\n> +\t\tcat smudge-write-fail.o >smudge-write-fail.r &&\n\nThis cat is cp.\n\n> +\t\tgit add . &&\n> +\t\tgit commit . -m \"test commit\" &&\n\nYou don't need to commit for 'git checkout ' (e.g. for .)\nor 'git cat-file -p :' to work.\n\n> +\t\trm -f *.r &&\n> +\n> +\t\tcheck_filter_ignore_clean \\\n> +\t\t\tgit checkout . \\\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\tSTART\n> +\t\t\t\t\twrote filter header\n> +\t\t\t\t\tIN: smudge smudge-write-fail.r 22 [OK] -- OUT: 22 [WRITE FAIL]\n> +\t\t\t\t\tSTART\n> +\t\t\t\t\twrote filter header\n> +\t\t\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n> +\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n> +\t\t\t\t\tSTOP\n> +\t\t\t\tEOF\n> +\n> +\t\tcheck_rot13 ../test.o test.r &&\n> +\t\tcheck_rot13 ../test2.o test2.r &&\n\nLooks good.\n\n> +\n> +\t\t! test_cmp smudge-write-fail.o smudge-write-fail.r && # Smudge failed!\n> +\t\t./../rot13.sh expected &&\n> +\t\tgit cat-file blob :smudge-write-fail.r >actual &&\n> +\t\ttest_cmp expected actual\t\t\t\t\t\t\t # Clean worked!\n\nThis is almost negation of check_rot13 - perhaps a helper function\nwould help here (check_not_rot13?).\n\nAlso, what this comment is about, and why so far to the right?\n\n> +\t)\n> +'\n> +\n> +test_expect_success PERL 'process filter should not restart in case of an error' '\n\nErrr... what? This description is not clear. Did you mean\nthat filter should not be restarted if it *signals* an error\nwith file (either before sending anything, or after sending\npartial contents)?\n\n> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\n> +\t\tcp ../test.o test.r &&\n> +\t\tcp ../test2.o test2.r &&\n> +\t\techo \"this will cause an error\" >error.o &&\n> +\t\tcp error.o error.r &&\n\nAnd here you (correctly) use cp, and not cat.\n\n> +\t\tgit add . &&\n> +\t\tgit commit . -m \"test commit\" &&\n> +\t\trm -f *.r &&\n> +\n> +\t\tcheck_filter_ignore_clean \\\n> +\t\t\tgit checkout . \\\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\tSTART\n> +\t\t\t\t\twrote filter header\n> +\t\t\t\t\tIN: smudge error.r 25 [OK] -- OUT: 0 [ERROR]\n> +\t\t\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n> +\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n> +\t\t\t\t\tSTOP\n> +\t\t\t\tEOF\n> +\n> +\t\tcheck_rot13 ../test.o test.r &&\n> +\t\tcheck_rot13 ../test2.o test2.r &&\n> +\t\ttest_cmp error.o error.r\n\nLooks good to me.\n\n> +\t)\n> +'\n> +\n> +test_expect_success PERL 'process filter should be able to signal an error for all future files' '\n\nDid you mean here that filter can abort processing of\nall future files?\n\n> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\n> +\t\tcp ../test.o test.r &&\n> +\t\tcp ../test2.o test2.r &&\n> +\t\techo \"error this blob and all future blobs\" >abort.o &&\n> +\t\tcp abort.o abort.r &&\n> +\t\tgit add . &&\n> +\t\tgit commit . -m \"test commit\" &&\n> +\t\trm -f *.r &&\n> +\n> +\t\tcheck_filter_ignore_clean \\\n> +\t\t\tgit checkout . \\\n> +\t\t\t\t<<-\\EOF &&\n> +\t\t\t\t\tSTART\n> +\t\t\t\t\twrote filter header\n> +\t\t\t\t\tIN: smudge abort.r 37 [OK] -- OUT: 0 [ABORT]\n> +\t\t\t\t\tSTOP\n\nHow can we know that 'abort' file is processed first?\nThough more resilent solution would be harder to create...\n\n> +\t\t\t\tEOF\n> +\n> +\t\ttest_cmp ../test.o test.r &&\n> +\t\ttest_cmp ../test2.o test2.r &&\n> +\t\ttest_cmp abort.o abort.r\n> +\t)\n> +'\n> +\n> +test_expect_success PERL 'invalid process filter must fail (and not hang!)' '\n> +\ttest_config_global filter.protocol.process cat &&\n\nWe could use rot13.sh, that is one-shot filter here.\n\n> +\ttest_config_global filter.protocol.required true &&\n\nAll right, filter is required to easier distinguish it not\nworking from not filtered.\n\n> +\trm -rf repo &&\n> +\tmkdir repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit init &&\n> +\n> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n> +\n> +\t\tcp ../test.o test.r &&\n> +\t\ttest_must_fail git add . 2> git_stderr.log &&\n> +\t\tgrep \"not support long running filter protocol\" git_stderr.log\n\nShouldn't this use gettext poison (or rather C locale)?\nThis error message could be translated in the future.\n\n> +\t)\n> +'\n> +\n> test_done\n\nI wonder how does the code coverage for the new v2 filter\ncode looks like...\n\nAnyway, I think it would be good idea to write at the beginning\nof new tests (be they in old test, or in new test) what we want\nto test:\n\n - that 'clean' and 'smudge' operations are invoked, for all\n possible combinations (covering all code paths), and that\n filter is invoked only once\n - that special types of files work:\n * empty file (in worktree, in index, in repo)\n * file in subdirectory\n * filename with special characters\n * large file (test marked as EXPENSIVE), multiple maximum\n packet size\n * perhaps binary file?\n - that 'process' overrides old-style 'clean' and 'smudge'\n filters, regardless of the former capabilities\n - that limiting capabilities works\n - that requiring filter works correctly (doubles number of\n tests, at least for a subset of them)\n - that filter is restarted if it fails on non-required,\n fails git command if required\n - that filter can error out out of filtering a file,\n upfront and after partial contents, without restart;\n fails git command if required\n - that filter can abort,\n fails git command if required (?)\n\n> diff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\n> new file mode 100755\n> index 0000000..8958f71\n> --- /dev/null\n> +++ b/t/t0021/rot13-filter.pl\n> @@ -0,0 +1,191 @@\n> +#!/usr/bin/perl\n> +#\n> +# Example implementation for the Git filter protocol version 2\n> +# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n> +#\n> +# The script takes the list of supported protocol capabilities as\n> +# arguments (\"clean\", \"smudge\", etc).\n> +#\n> +# This implementation supports special test cases:\n> +# (1) If data with the pathname \"clean-write-fail.r\" is processed with\n> +# a \"clean\" operation then the write operation will die.\n> +# (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n> +# a \"smudge\" operation then the write operation will die.\n> +# (3) If data with the pathname \"error.r\" is processed with any\n> +# operation then the filter signals that it cannot or does not want\n> +# to process the file.\n> +# (4) If data with the pathname \"abort.r\" is processed with any\n> +# operation then the filter signals that it cannot or does not want\n> +# to process the file and any file after that is processed with the\n> +# same command.\n\nNice to have this description.\n\nBTW. why write-fail is per operation (clean or smudge), but error and abort\nis not?\n\n\n> +#\n> +\n> +use strict;\n> +use warnings;\n\nI guess there is some duplication with the code in contrib, isn't it?\n\n> +\n> +my $MAX_PACKET_CONTENT_SIZE = 65516;\n> +my @capabilities = @ARGV;\n> +\n> +open my $debug, \">>\", \"rot13-filter.log\";\n\n \tor die \"cannot open file for appending: $!\";\n\nGood, three argument open. Bad (?), not error handling.\n\n> +\n> +sub rot13 {\n> + my ($str) = @_;\n ^^^^\n\nWhy 4 spaces, and not TAB character?\n\n\nI think\n\n \tmy $str = shift;\n\nis more idiomatic Perl.\n\n> + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n\nWhy not use tr/// version of this quote-like operation?\nOr do you follow prior art here?\n\n> + return $str;\n> +}\n> +\n> +sub packet_bin_read {\n> + my $buffer;\n> + my $bytes_read = read STDIN, $buffer, 4;\n> + if ( $bytes_read == 0 ) {\n> +\n> + # EOF - Git stopped talking to us!\n> + print $debug \"STOP\\n\";\n> + exit();\n> + }\n> + elsif ( $bytes_read != 4 ) {\n> + die \"invalid packet size '$bytes_read' field\";\n\nErrr, $bytes_read is not packet size field. It is $buffer.\nAlso, error message looks strange\n\n \t\tinvalid packet size '004' field\n\nShouldn't it be at end?\n\n> + }\n> + my $pkt_size = hex($buffer);\n\n$pkt_size greater than $MAX_PACKET_CONTENT_SIZE is also an error,\nas is sizes 1-3 (not that it matters much, at least here).\n\n> + if ( $pkt_size == 0 ) {\n> + return ( 1, \"\" );\n\nIt feels a bit strange to me to return list instead of hashref,\nbut this is a matter of opinion.\n\n> + }\n> + elsif ( $pkt_size > 4 ) {\n> + my $content_size = $pkt_size - 4;\n> + $bytes_read = read STDIN, $buffer, $content_size;\n> + if ( $bytes_read != $content_size ) {\n> + die \"invalid packet ($content_size expected; $bytes_read read)\";\n\nIt would read, strangely\n\n \t\t \"invalid packet (8 expected, 7 read)\"\n\nThe \"size\" or \"bytes\" is missing from this output.\n\n> + }\n> + return ( 0, $buffer );\n> + }\n> + else {\n> + die \"invalid packet size\";\n\nIs keep-alive packet valid (\"0004\")?\n\n> + }\n> +}\n> +\n> +sub packet_txt_read {\n> + my ( $res, $buf ) = packet_bin_read();\n> + unless ( $buf =~ /\\n$/ ) {\n> + die \"A non-binary line SHOULD BE terminated by an LF.\";\n\nFirst, if SHOULD BE, then perhaps 'warn' not 'die'... though for\ntests it is probably better to 'die'.\n\nSecond, we should probably print (a fragment of) this line.\n\n> + }\n> + return ( $res, substr( $buf, 0, -1 ) );\n\nSame comment as for example file in contrib/ - use s/// and no\nneed for substr stuff.\n\n> +}\n> +\n> +sub packet_bin_write {\n> + my ($packet) = @_;\n> + print STDOUT sprintf( \"%04x\", length($packet) + 4 );\n> + print STDOUT $packet;\n> + STDOUT->flush();\n> +}\n> +\n> +sub packet_txt_write {\n> + packet_bin_write( $_[0] . \"\\n\" );\n> +}\n> +\n> +sub packet_flush {\n> + print STDOUT sprintf( \"%04x\", 0 );\n> + STDOUT->flush();\n> +}\n\nLooks good to me (though same comments as to contrib/ file applies).\n\n> +\n> +print $debug \"START\\n\";\n> +$debug->flush();\n> +\n> +( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n> +( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n> +( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n> +\n> +packet_txt_write(\"git-filter-server\");\n> +packet_txt_write(\"version=2\");\n> +\n> +( packet_txt_read() eq ( 0, \"clean=true\" ) ) || die \"bad capability\";\n> +( packet_txt_read() eq ( 0, \"smudge=true\" ) ) || die \"bad capability\";\n> +( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n> +\n> +foreach (@capabilities) {\n> + packet_txt_write( $_ . \"=true\" );\n> +}\n> +packet_flush();\n> +print $debug \"wrote filter header\\n\";\n\nOr perhaps \"handshake end\"?\n\n> +$debug->flush();\n> +\n> +while (1) {\n> + my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n> + print $debug \"IN: $command\";\n> + $debug->flush();\n> +\n> + my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;\n\nAll right, here list context is necessary.\n\n> + print $debug \" $pathname\";\n\nNo \" pathname=$pathname\" ?\n\n> + $debug->flush();\n> +\n> + # Flush\n> + packet_bin_read();\n\nSame comment as earlier: read_flush, or read_varlist (till flush)\nto have would be better.\n\n> +\n> + my $input = \"\";\n> + {\n> + binmode(STDIN);\n> + my $buffer;\n> + my $done = 0;\n> + while ( !$done ) {\n> + ( $done, $buffer ) = packet_bin_read();\n> + $input .= $buffer;\n> + }\n> + print $debug \" \" . length($input) . \" [OK] -- \";\n> + $debug->flush();\n> + }\n> +\n> + my $output;\n> + if ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n> + $output = \"\";\n> + }\n> + elsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n> + $output = rot13($input);\n> + }\n> + elsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n> + $output = rot13($input);\n> + }\n> + else {\n> + die \"bad command '$command'\";\n> + }\n> +\n> + print $debug \"OUT: \" . length($output) . \" \";\n\nShouldn't we write the length of output only if we don't error out,\nabort, or fail?\n\n> + $debug->flush();\n> +\n> + if ( $pathname eq \"error.r\" ) {\n> + print $debug \"[ERROR]\\n\";\n> + $debug->flush();\n> + packet_txt_write(\"status=error\");\n> + packet_flush();\n> + }\n> + elsif ( $pathname eq \"abort.r\" ) {\n> + print $debug \"[ABORT]\\n\";\n> + $debug->flush();\n> + packet_txt_write(\"status=abort\");\n> + packet_flush();\n> + }\n\nLooks good, so this is upfront status=error or status-abort.\n\n> + else {\n> + packet_txt_write(\"status=success\");\n> + packet_flush();\n> +\n> + if ( $pathname eq \"${command}-write-fail.r\" ) {\n> + print $debug \"[WRITE FAIL]\\n\";\n> + $debug->flush();\n> + die \"${command} write error\";\n> + }\n> +\n> + while ( length($output) > 0 ) {\n> + my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n> + packet_bin_write($packet);\n> + print $debug \".\";\n\nAll right, so number of dots is the number of packets. This is\nsurprisingly opaque.\n\n> + if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n> + $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n> + }\n> + else {\n> + $output = \"\";\n> + }\n> + }\n> + packet_flush();\n> + print $debug \" [OK]\\n\";\n> + $debug->flush();\n> + packet_flush();\n\nShould we test partial contents case? Or failure during printing?\nWhat happens then - is file cleared by Git, or left partially converted?\n\n> + }\n> +}\n> \n\nKeep up good work. Looks quite good.\n-- \nJakub Narębski\n\n"},{"id":"302853","messageId":"62b5fd5d-51b7-36ef-fcc2-db1d53220abe@gmail.com","threadId":"44122","inReplyTo":"13E00B0F-41FC-4FBE-9ABD-2E9B11743E2B@gmail.com","subject":"Re: [PATCH v8 01/11] pkt-line: rename packet_write() to packet_write_fmt()","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-28T23:15:54Z","receivedAt":"2016-09-28T23:16:08Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 26.09.2016 o 20:49, Lars Schneider pisze: \n> On 24 Sep 2016, at 23:14, Jakub Narębski wrote:\n>> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n>>\n>>> From: Lars Schneider \n>>>\n>>> packet_write() should be called packet_write_fmt() as the string\n>>> parameter can be formatted.\n>>\n>> I would say:\n>>\n>> packet_write() should be called packet_write_fmt() because it\n>> is printf-like function where first parameter is format string.\n>>\n>> Or something like that. But such minor change might be not worth\n>> yet another reroll of this patch series.\n>>\n>> Perhaps it would be a good idea to explain the reasoning behind\n>> this change:\n>>\n>> This is important distinction to know from the name if the\n>> function accepts arbitrary binary data and/or arbitrary\n>> strings to be written - packet_write[_fmt()] do not.\n> \n> packet_write() should be called packet_write_fmt() because it is a\n> printf-like function that takes a format string as first parameter.\n> \n> packet_write_fmt() should be used for text strings only. Arbitrary\n> binary data should use a new packet_write() function that is introduced\n> in a subsequent patch.\n> \n> Better?\n\nBetter.\n\n> \n>>> pkt-line.h | 2 +-\n>>> shallow.c | 2 +-\n>>> upload-pack.c | 30 +++++++++++++++---------------\n>>> 11 files changed, 29 insertions(+), 29 deletions(-)\n>>\n>> Diffstat looks correct. Was the patch generated by doing search\n>> and replace?\n> \n> Yes.\n\nGood.\n\n-- \nJakub Narębski\n\n"},{"id":"302877","messageId":"C2C9761E-986F-473D-BFB7-CBEF900D9FA3@gmail.com","threadId":"44122","inReplyTo":"xmqq8tubitjs.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-29T10:28:23Z","receivedAt":"2016-09-29T10:39:54Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 28 Sep 2016, at 23:49, Junio C Hamano wrote:\n> \n> I suspect that you are preparing a reroll already, but the one that\n> is sitting in 'pu' seems to be flaky in t/t0021 and I seem to see\n> occasional failures from it.\n> \n> I didn't trace where the test goes wrong, but one easy mistake you\n> could make (I am not saying that is the reason of the failure) is to\n> assume your filter will not be called under certain condition (like\n> immediately after you checked out from the index to the working\n> tree), when the automated test goes fast enough and get you into a\n> \"racy git\" situation---the filter may be asked to filter the\n> contents from the working tree again to re-validate what's there is\n> still what is in the index.\n\nThanks for the heads-up! \n\nThis is what happens:\n\n1) Git exits\n2) The filter process receives EOF and prints \"STOP\" to the log\n3) t0021 checks the content of the log\n\nSometimes 3 happened before 2 which makes the test fail.\n(Example: https://travis-ci.org/git/git/jobs/162660563 )\n\nI added a this to wait until the filter process terminates:\n\n+wait_for_filter_termination () {\n+\twhile ps | grep -v grep | grep -F \"/t0021/rot13-filter.pl\" >/dev/null 2>&1\n+\tdo\n+\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n+\t\tsleep 1\n+\tdone\n+}\n\nDoes this look OK to you?\n\n- Lars\n"},{"id":"302880","messageId":"f7a4f828-bb1d-0ffa-e369-3b4fa476d9e5@web.de","threadId":"44122","inReplyTo":"C2C9761E-986F-473D-BFB7-CBEF900D9FA3@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-09-29T11:57:47Z","receivedAt":"2016-09-29T11:58:12Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\nOn 29/09/16 12:28, Lars Schneider wrote:\n>> On 28 Sep 2016, at 23:49, Junio C Hamano wrote:\n>>\n>> I suspect that you are preparing a reroll already, but the one that\n>> is sitting in 'pu' seems to be flaky in t/t0021 and I seem to see\n>> occasional failures from it.\n>>\n>> I didn't trace where the test goes wrong, but one easy mistake you\n>> could make (I am not saying that is the reason of the failure) is to\n>> assume your filter will not be called under certain condition (like\n>> immediately after you checked out from the index to the working\n>> tree), when the automated test goes fast enough and get you into a\n>> \"racy git\" situation---the filter may be asked to filter the\n>> contents from the working tree again to re-validate what's there is\n>> still what is in the index.\n> Thanks for the heads-up!\n>\n> This is what happens:\n>\n> 1) Git exits\n> 2) The filter process receives EOF and prints \"STOP\" to the log\n> 3) t0021 checks the content of the log\n>\n> Sometimes 3 happened before 2 which makes the test fail.\n> (Example: https://travis-ci.org/git/git/jobs/162660563 )\n>\n> I added a this to wait until the filter process terminates:\n>\n> +wait_for_filter_termination () {\n> +\twhile ps | grep -v grep | grep -F \"/t0021/rot13-filter.pl\" >/dev/null 2>&1\n> +\tdo\n> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n> +\t\tsleep 1\n> +\tdone\n> +}\n>\n> Does this look OK to you?\nDo we need the ps at all ?\nHow about this:\n\n+wait_for_filter_termination () {\n+\twhile ! grep \"STOP\" LOGFILENAME >/dev/null\n+\tdo\n+\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n+\t\tsleep 1\n+\tdone\n+}\n\n\n"},{"id":"302893","messageId":"xmqqk2duhcdm.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"f7a4f828-bb1d-0ffa-e369-3b4fa476d9e5@web.de","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-29T16:57:57Z","receivedAt":"2016-09-29T16:58:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n>> 1) Git exits\n>> 2) The filter process receives EOF and prints \"STOP\" to the log\n>> 3) t0021 checks the content of the log\n>>\n>> Sometimes 3 happened before 2 which makes the test fail.\n>> (Example: https://travis-ci.org/git/git/jobs/162660563 )\n>>\n>> I added a this to wait until the filter process terminates:\n>>\n>> +wait_for_filter_termination () {\n>> +\twhile ps | grep -v grep | grep -F \"/t0021/rot13-filter.pl\" >/dev/null 2>&1\n>> +\tdo\n>> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n>> +\t\tsleep 1\n>> +\tdone\n>> +}\n>>\n>> Does this look OK to you?\n> Do we need the ps at all ?\n> How about this:\n>\n> +wait_for_filter_termination () {\n> +\twhile ! grep \"STOP\" LOGFILENAME >/dev/null\n> +\tdo\n> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n> +\t\tsleep 1\n> +\tdone\n> +}\n\nRunning \"ps\" and grepping for a command is not suitable for script\nto reliably tell things, so it is out of question. Compared to\nthat, your version looks slightly better, but what if the machinery\nthat being tested, i.e. the part that drives the filter process, is\nbuggy or becomes buggy and causes the filter process that writes\n\"STOP\" to die before it actually writes that string?\n\nI have a feeling that the machinery being tested needs to be fixed\nso that the sequence is always be:\n\n 0) Git spawns the filter process, as it needs some contents to\n be filtered.\n\n 1) Git did everything it needed to do and decides that is time\n to go.\n\n 2) Filter process receives EOF and prints \"STOP\" to the log.\n\n 3) Git waits until the filter process finishes.\n\n 4) t0021, after Git finishes, checks the log.\n\nRepeated sleep combined with grep is probably just sweeping the real\nproblem under the rug. Do we have enough information to do the\nabove?\n\nAn inspiration may be in the way we centrally clean all tempfiles\nand lockfiles before exiting. We have a central registry of these\nfiles that need cleaning up and have a single atexit(3) handler to\nclean them up. Perhaps we need a registry that filter processes\nspawned by the mechanism Lars introduces in this series, and have an\natexit(3) handler that closes the pipe to them (which signals the\nfilters that it is time for them to go) and wait(2) on them, or\nsomething? I do not think we want any kill(2) to be involved in\nthis clean-up procedure, but I do think we should wait(2) on what we\nspawn, as long as these processes are meant to be shut down when the\nmain process of Git exits (this is different from things like\ncredential-cache daemon where they are expected to persist and meant\nto serve multiple Git processes).\n\n\n"},{"id":"302902","messageId":"1A8A9127-4DF9-44AD-9497-F8A630AB1193@gmail.com","threadId":"44122","inReplyTo":"xmqqk2duhcdm.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-29T17:57:43Z","receivedAt":"2016-09-29T17:58:08Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Sep 2016, at 18:57, Junio C Hamano wrote:\n> \n> Torsten Bögershausen writes:\n> \n>>> 1) Git exits\n>>> 2) The filter process receives EOF and prints \"STOP\" to the log\n>>> 3) t0021 checks the content of the log\n>>> \n>>> Sometimes 3 happened before 2 which makes the test fail.\n>>> (Example: https://travis-ci.org/git/git/jobs/162660563 )\n>>> \n>>> I added a this to wait until the filter process terminates:\n>>> \n>>> +wait_for_filter_termination () {\n>>> +\twhile ps | grep -v grep | grep -F \"/t0021/rot13-filter.pl\" >/dev/null 2>&1\n>>> +\tdo\n>>> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n>>> +\t\tsleep 1\n>>> +\tdone\n>>> +}\n>>> \n>>> Does this look OK to you?\n>> Do we need the ps at all ?\n>> How about this:\n>> \n>> +wait_for_filter_termination () {\n>> +\twhile ! grep \"STOP\" LOGFILENAME >/dev/null\n>> +\tdo\n>> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n>> +\t\tsleep 1\n>> +\tdone\n>> +}\n> \n> Running \"ps\" and grepping for a command is not suitable for script\n> to reliably tell things, so it is out of question. Compared to\n> that, your version looks slightly better, but what if the machinery\n> that being tested, i.e. the part that drives the filter process, is\n> buggy or becomes buggy and causes the filter process that writes\n> \"STOP\" to die before it actually writes that string?\n> \n> I have a feeling that the machinery being tested needs to be fixed\n> so that the sequence is always be:\n> \n> 0) Git spawns the filter process, as it needs some contents to\n> be filtered.\n> \n> 1) Git did everything it needed to do and decides that is time\n> to go.\n> \n> 2) Filter process receives EOF and prints \"STOP\" to the log.\n> \n> 3) Git waits until the filter process finishes.\n> \n> 4) t0021, after Git finishes, checks the log.\n> \n> Repeated sleep combined with grep is probably just sweeping the real\n> problem under the rug. Do we have enough information to do the\n> above?\n> \n> An inspiration may be in the way we centrally clean all tempfiles\n> and lockfiles before exiting. We have a central registry of these\n> files that need cleaning up and have a single atexit(3) handler to\n> clean them up. Perhaps we need a registry that filter processes\n> spawned by the mechanism Lars introduces in this series, and have an\n> atexit(3) handler that closes the pipe to them (which signals the\n> filters that it is time for them to go) and wait(2) on them, or\n> something? I do not think we want any kill(2) to be involved in\n> this clean-up procedure, but I do think we should wait(2) on what we\n> spawn, as long as these processes are meant to be shut down when the\n> main process of Git exits (this is different from things like\n> credential-cache daemon where they are expected to persist and meant\n> to serve multiple Git processes).\n\nWe discussed that issue in v4 and v6:\nhttp://public-inbox.org/git/20160803225313.pk3tfe5ovz4y3i7l@sigill.intra.peff.net/\nhttp://public-inbox.org/git/xmqqbn0a3wy3.fsf@gitster.mtv.corp.google.com/\n\nMy impression was that you don't want Git to wait for the filter process.\nIf Git waits for the filter process - how long should Git wait?\n\nThanks,\nLars"},{"id":"302903","messageId":"20160929180247.d4owhzwyawtanw3r@sigill.intra.peff.net","threadId":"44122","inReplyTo":"xmqqk2duhcdm.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-09-29T18:02:48Z","receivedAt":"2016-09-29T18:02:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Sep 29, 2016 at 09:57:57AM -0700, Junio C Hamano wrote:\n\n> > +wait_for_filter_termination () {\n> > +\twhile ! grep \"STOP\" LOGFILENAME >/dev/null\n> > +\tdo\n> > +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n> > +\t\tsleep 1\n> > +\tdone\n> > +}\n> \n> Running \"ps\" and grepping for a command is not suitable for script\n> to reliably tell things, so it is out of question. Compared to\n> that, your version looks slightly better, but what if the machinery\n> that being tested, i.e. the part that drives the filter process, is\n> buggy or becomes buggy and causes the filter process that writes\n> \"STOP\" to die before it actually writes that string?\n\nI'm of the opinion that any busy-waiting is a good sign that something\nis suboptimal. The right solution here seems like it should be signaling\nthe test script via a descriptor.\n\nI don't necessarily agree, though, that the timing of filter-process\ncleanup needs to be part of the public interface. So in your list:\n\n> 3) Git waits until the filter process finishes.\n\nThat seems simple and elegant, but I can think of reasons we might not\nwant to wait (e.g., if the filter has to do some maintenance task and\ndoes not the user to have to wait).\n\nOTOH, we already face this in git, and we solve it by explicitly\nbackgrounding the maintenance task (i.e., auto-gc). So one could argue\nthat it is the responsibility of the filter process to manage its own\nprocesses. It certainly makes the interaction with git simpler.\n\n-Peff\n"},{"id":"302907","messageId":"7f8ab626-ecdb-70a8-aa19-615c3c84148e@web.de","threadId":"44122","inReplyTo":"1A8A9127-4DF9-44AD-9497-F8A630AB1193@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-09-29T18:18:07Z","receivedAt":"2016-09-29T18:19:16Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\nOn 29/09/16 19:57, Lars Schneider wrote:\n>> On 29 Sep 2016, at 18:57, Junio C Hamano wrote:\n>>\n>> Torsten Bögershausen writes:\n>>\n>>>> 1) Git exits\n>>>> 2) The filter process receives EOF and prints \"STOP\" to the log\n>>>> 3) t0021 checks the content of the log\n>>>>\n>>>> Sometimes 3 happened before 2 which makes the test fail.\n>>>> (Example: https://travis-ci.org/git/git/jobs/162660563 )\n>>>>\n>>>> I added a this to wait until the filter process terminates:\n>>>>\n>>>> +wait_for_filter_termination () {\n>>>> +\twhile ps | grep -v grep | grep -F \"/t0021/rot13-filter.pl\" >/dev/null 2>&1\n>>>> +\tdo\n>>>> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n>>>> +\t\tsleep 1\n>>>> +\tdone\n>>>> +}\n>>>>\n>>>> Does this look OK to you?\n>>> Do we need the ps at all ?\n>>> How about this:\n>>>\n>>> +wait_for_filter_termination () {\n>>> +\twhile ! grep \"STOP\" LOGFILENAME >/dev/null\n>>> +\tdo\n>>> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n>>> +\t\tsleep 1\n>>> +\tdone\n>>> +}\n>> Running \"ps\" and grepping for a command is not suitable for script\n>> to reliably tell things, so it is out of question. Compared to\n>> that, your version looks slightly better, but what if the machinery\n>> that being tested, i.e. the part that drives the filter process, is\n>> buggy or becomes buggy and causes the filter process that writes\n>> \"STOP\" to die before it actually writes that string?\n>>\n>> I have a feeling that the machinery being tested needs to be fixed\n>> so that the sequence is always be:\n>>\n>> 0) Git spawns the filter process, as it needs some contents to\n>> be filtered.\n>>\n>> 1) Git did everything it needed to do and decides that is time\n>> to go.\n>>\n>> 2) Filter process receives EOF and prints \"STOP\" to the log.\n>>\n>> 3) Git waits until the filter process finishes.\n>>\n>> 4) t0021, after Git finishes, checks the log.\n>>\n>> Repeated sleep combined with grep is probably just sweeping the real\n>> problem under the rug. Do we have enough information to do the\n>> above?\n>>\n>> An inspiration may be in the way we centrally clean all tempfiles\n>> and lockfiles before exiting. We have a central registry of these\n>> files that need cleaning up and have a single atexit(3) handler to\n>> clean them up. Perhaps we need a registry that filter processes\n>> spawned by the mechanism Lars introduces in this series, and have an\n>> atexit(3) handler that closes the pipe to them (which signals the\n>> filters that it is time for them to go) and wait(2) on them, or\n>> something? I do not think we want any kill(2) to be involved in\n>> this clean-up procedure, but I do think we should wait(2) on what we\n>> spawn, as long as these processes are meant to be shut down when the\n>> main process of Git exits (this is different from things like\n>> credential-cache daemon where they are expected to persist and meant\n>> to serve multiple Git processes).\n> We discussed that issue in v4 and v6:\n> http://public-inbox.org/git/20160803225313.pk3tfe5ovz4y3i7l@sigill.intra.peff.net/\n> http://public-inbox.org/git/xmqqbn0a3wy3.fsf@gitster.mtv.corp.google.com/\n>\n> My impression was that you don't want Git to wait for the filter process.\n> If Git waits for the filter process - how long should Git wait?\n>\n> Thanks,\n> Lars\n\nHm,\nI would agree that Git should not wait for the filter.\nBut does the test suite need to wait for the filter ?\nMay be, in this case we test the filter and Git, which is good.\nAdding a 1 second delay, if, and only if, there is a racy condition,\nis not that bad (or do we have better ways to check for a process to\nbe terminated ?)\n\n\n"},{"id":"302912","messageId":"049fad9b-d625-a80b-bd6b-9b80c9701f7a@kdbg.org","threadId":"44122","inReplyTo":"7f8ab626-ecdb-70a8-aa19-615c3c84148e@web.de","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2016-09-29T18:38:02Z","receivedAt":"2016-09-29T18:38:27Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 29.09.2016 um 20:18 schrieb Torsten Bögershausen:\n> I would agree that Git should not wait for the filter.\n> But does the test suite need to wait for the filter ?\n\nWe have fixed a test case on Windows recently where a process hung \naround too long (5babb5bd). So, yes, the test suite has to wait for the \nfilter.\n\n-- Hannes\n\n"},{"id":"302931","messageId":"DADB0C80-1EDF-4498-8DAC-A1B09E596518@gmail.com","threadId":"44122","inReplyTo":"xmqqk2duhcdm.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-29T20:50:46Z","receivedAt":"2016-09-29T20:50:56Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Sep 2016, at 18:57, Junio C Hamano wrote:\n> \n> Torsten Bögershausen writes:\n> \n>>> 1) Git exits\n>>> 2) The filter process receives EOF and prints \"STOP\" to the log\n>>> 3) t0021 checks the content of the log\n>>> \n>>> Sometimes 3 happened before 2 which makes the test fail.\n>>> (Example: https://travis-ci.org/git/git/jobs/162660563 )\n>>> \n>>> I added a this to wait until the filter process terminates:\n>>> \n>>> +wait_for_filter_termination () {\n>>> +\twhile ps | grep -v grep | grep -F \"/t0021/rot13-filter.pl\" >/dev/null 2>&1\n>>> +\tdo\n>>> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n>>> +\t\tsleep 1\n>>> +\tdone\n>>> +}\n>>> \n>>> Does this look OK to you?\n>> Do we need the ps at all ?\n>> How about this:\n>> \n>> +wait_for_filter_termination () {\n>> +\twhile ! grep \"STOP\" LOGFILENAME >/dev/null\n>> +\tdo\n>> +\t\techo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n>> +\t\tsleep 1\n>> +\tdone\n>> +}\n> \n> Running \"ps\" and grepping for a command is not suitable for script\n> to reliably tell things, so it is out of question. Compared to\n> that, your version looks slightly better, but what if the machinery\n> that being tested, i.e. the part that drives the filter process, is\n> buggy or becomes buggy and causes the filter process that writes\n> \"STOP\" to die before it actually writes that string?\n> \n> I have a feeling that the machinery being tested needs to be fixed\n> so that the sequence is always be:\n> \n> 0) Git spawns the filter process, as it needs some contents to\n> be filtered.\n> \n> 1) Git did everything it needed to do and decides that is time\n> to go.\n> \n> 2) Filter process receives EOF and prints \"STOP\" to the log.\n> \n> 3) Git waits until the filter process finishes.\n> \n> 4) t0021, after Git finishes, checks the log.\n\n\nA pragmatic approach:\n\nI could drop the \"STOP\" message that the filter writes to the log\non exit and everything would work as is. We could argue that this \nis OK because Git doesn't care anyways if the filter process has \nstopped or not.\n\nWould that be OK for everyone?\n\n- Lars"},{"id":"302932","messageId":"f9a6dd02-34c0-d48b-3cbc-73202488920c@gmail.com","threadId":"44122","inReplyTo":"f7a4f828-bb1d-0ffa-e369-3b4fa476d9e5@web.de","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-09-29T20:59:54Z","receivedAt":"2016-09-29T21:00:10Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 29.09.2016 o 13:57, Torsten Bögershausen pisze: \n> On 29/09/16 12:28, Lars Schneider wrote:\n\n>> This is what happens:\n>>\n>> 1) Git exits\n>> 2) The filter process receives EOF and prints \"STOP\" to the log\n>> 3) t0021 checks the content of the log\n>>\n>> Sometimes 3 happened before 2 which makes the test fail.\n>> (Example: https://travis-ci.org/git/git/jobs/162660563 )\n>>\n>> I added a this to wait until the filter process terminates:\n>>\n>> +wait_for_filter_termination () {\n>> + while ps | grep -v grep | grep -F \"/t0021/rot13-filter.pl\" >/dev/null 2>&1\n>> + do\n>> + echo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n>> + sleep 1\n>> + done\n>> +}\n>>\n>> Does this look OK to you?\n> Do we need the ps at all ?\n> How about this:\n> \n> +wait_for_filter_termination () {\n> + while ! grep \"STOP\" LOGFILENAME >/dev/null\n> + do\n> + echo \"Waiting for /t0021/rot13-filter.pl to finish...\"\n> + sleep 1\n> + done\n> +}\n\nOr even better: make filter driver write its pid to pidfile, and then\n\"wait $(cat rot13-filter.pid)\". That's what we do in lib-git-daemon.sh\n(I think).\n\nIf the problem is exit status of \"wait\" builtin, then filter driver\ncan remove its pidfile after writing \"STOP\", just before ending.\n\n-- \nJakub Narębski\n\n"},{"id":"302945","messageId":"xmqqk2due7g2.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"DADB0C80-1EDF-4498-8DAC-A1B09E596518@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-29T21:12:45Z","receivedAt":"2016-09-29T21:13:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n> A pragmatic approach:\n>\n> I could drop the \"STOP\" message that the filter writes to the log\n> on exit and everything would work as is. We could argue that this \n> is OK because Git doesn't care anyways if the filter process has \n> stopped or not.\n\nThat would mean you can leave the process running while the test\nframework tries to remove the trash directory when we are done,\ncreating the same bug J6t mentioned in the thread, no?\n\n"},{"id":"302946","messageId":"xmqqfuoie78a.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"f9a6dd02-34c0-d48b-3cbc-73202488920c@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-29T21:17:25Z","receivedAt":"2016-09-29T21:17:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narębski writes:\n\n> Or even better: make filter driver write its pid to pidfile, and then\n> \"wait $(cat rot13-filter.pid)\". That's what we do in lib-git-daemon.sh\n> (I think).\n\nI am not sure if \"wait\"ing on a random process that is not a direct\nchild is a reasonable thing to do, but I like the direction.\n\nCommunicate with a pidfile and wait until \"kill -0 $that_pid\" fails,\nor something like that, would be clean enough.\n\n\n\n"},{"id":"302947","messageId":"xmqqa8eqe746.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"20160929180247.d4owhzwyawtanw3r@sigill.intra.peff.net","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-29T21:19:53Z","receivedAt":"2016-09-29T21:20:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I don't necessarily agree, though, that the timing of filter-process\n> cleanup needs to be part of the public interface. So in your list:\n>\n>> 3) Git waits until the filter process finishes.\n>\n> That seems simple and elegant, but I can think of reasons we might not\n> want to wait (e.g., if the filter has to do some maintenance task and\n> does not the user to have to wait).\n>\n> OTOH, we already face this in git, and we solve it by explicitly\n> backgrounding the maintenance task (i.e., auto-gc). So one could argue\n> that it is the responsibility of the filter process to manage its own\n> processes. It certainly makes the interaction with git simpler.\n\nYup, that summarizes my thinking a lot better than I managed to do\nin the previous message.\n\n"},{"id":"302949","messageId":"xmqq60pee6rp.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"1A8A9127-4DF9-44AD-9497-F8A630AB1193@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-29T21:27:22Z","receivedAt":"2016-09-29T21:27:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n> We discussed that issue in v4 and v6:\n> http://public-inbox.org/git/20160803225313.pk3tfe5ovz4y3i7l@sigill.intra.peff.net/\n> http://public-inbox.org/git/xmqqbn0a3wy3.fsf@gitster.mtv.corp.google.com/\n>\n> My impression was that you don't want Git to wait for the filter process.\n> If Git waits for the filter process - how long should Git wait?\n\nI am not sure where you got that impression. I did say that I do\nnot want Git to _KILL_ my filter process. That does not mean I want\nGit to go away without waiting for me.\n\nIf the filter process refuses to die forever when Git told it to\nshutdown (by closing the pipe to it, for example), that filter\nprocess is simply buggy. I think we want users to become aware of\nthat, instead of Git leaving it behind, which essentially is to\nsweep the problem under the rug.\n\nI agree with what Peff said elsewhere in the thread; if a filter\nprocess wants to take time to clean things up while letting Git\nproceed, it can do its own process management, but I think it is\nsensible for Git to wait the filter process it directly spawned.\n\n"},{"id":"303002","messageId":"4DE57A65-1020-4F17-81F2-9F319834BB2D@gmail.com","threadId":"44122","inReplyTo":"91589466-439e-7200-7256-b9288beae685@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-30T18:56:12Z","receivedAt":"2016-09-30T18:56:22Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Sep 2016, at 00:41, Jakub Narębski wrote:\n> \n> Part first of the review of 11/11.\n> \n> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n>> From: Lars Schneider \n>> \n>> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n>> index 7aff940..946dcad 100644\n>> --- a/Documentation/gitattributes.txt\n>> +++ b/Documentation/gitattributes.txt\n>> @@ -293,7 +293,13 @@ checkout, when the `smudge` command is specified, the command is\n>> fed the blob object from its standard input, and its standard\n>> output is used to update the worktree file. Similarly, the\n>> `clean` command is used to convert the contents of worktree file\n>> -upon checkin.\n>> +upon checkin. By default these commands process only a single\n>> +blob and terminate. If a long running `process` filter is used\n> ^^^^\n> \n> Should we use this terminology here? I have not read the preceding\n> part of documentation, so I don't know if it talks about \"blobs\" or\n> if it uses \"files\" and/or \"file contents\".\n\nI used that because it was used in the paragraph above already.\n\n\n>> +Long Running Filter Process\n>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>> +\n>> +If the filter command (a string value) is defined via\n>> +`filter..process` then Git can process all blobs with a\n>> +single filter invocation for the entire life of a single Git\n>> +command. This is achieved by using a packet format (pkt-line,\n>> +see technical/protocol-common.txt) based protocol over standard\n>> +input and standard output as follows. All packets are considered\n>> +text and therefore are terminated by an LF. Exceptions are the\n>> +\"*CONTENT\" packets and the flush packet.\n> \n> I guess that reasoning here is that all but CONTENT packets are\n> metadata, and thus to aid debuggability of the protocol are \"text\",\n> as considered by pkt-line.\n> \n> Perhaps a bit more readable would be the following (but current is\n> just fine; I am nitpicking):\n> \n> All packets, except for the \"{star}CONTENT\" packets and the \"0000\"\n> flush packer, are considered text and therefore are terminated by\n> a LF.\n\nOK, I use that!\n\n\n> I think it might be a good idea to describe what flush packet is\n> somewhere in this document; on the other hand referring (especially\n> if hyperlinked) to pkt-line technical documentation might be good\n> enough / better. I'm unsure, but I tend on the side that referring\n> to technical documentation is better.\n\nI have this line in the first paragraph of the Long Running Filter process:\n\"packet format (pkt-line, see technical/protocol-common.txt) based protocol\"\n\n> \n>> +to read a welcome response message (\"git-filter-server\") and exactly\n>> +one protocol version number from the previously sent list. All further\n> \n> I guess that is to provide forward-compatibility, isn't it? Also,\n> \"Git expects...\" probably means filter process MUST send, in the\n> RFC2119 (https://tools.ietf.org/html/rfc2119) meaning.\n\nTrue. I feel \"expects\" reads better but I am happy to change it if\nyou feel strong about it.\n\n\n>> +\n>> +After the version negotiation Git sends a list of supported capabilities\n>> +and a flush packet.\n> \n> Is it that Git SHOULD send list of ALL supported capabilities, or is\n> it that Git SHOULD NOT send capabilities it does not support, and that\n> it MAY send only those capabilities it needs (so for example if command\n> uses only `smudge`, it may not send `clean`, so that filter driver doesn't\n> need to initialize data it would not need).\n\n\"After the version negotiation Git sends a list of all capabilities that\nit supports and a flush packet.\"\n\nBetter?\n\n\n> I wonder why it is \"=true\", and not \"capability=\".\n> Is there a case where we would want to send \"=false\". Or\n> is it to allow configurable / value based capabilities? Isn't it going\n> a bit too far: is there even a hind of an idea for parametrize-able\n> capability? YAGNI is a thing...\n\nPeff suggested that format and I think it is OK:\nhttp://public-inbox.org/git/20160803224619.bwtbvmslhuicx2qi@sigill.intra.peff.net/\n\n\n> A few new capabilities that we might want to support in the near future\n> is \"size\", \"stream\", which are options describing how to communicate,\n> and \"cleanFromFile\", \"smudgeToFile\", which are new types of operations...\n> but neither needs any parameter.\n> \n> I guess that adding new capabilities doesn't require having to come up\n> with the new version of the protocol, isn't it.\n\nCorrect.\n\n\n>> +packet: git< git-filter-server\n>> +packet: git< version=2\n>> +packet: git> clean=true\n>> +packet: git> smudge=true\n>> +packet: git> not-yet-invented=true\n> \n> Hmmm... should we hint at the use of kebab-case versus snake_case\n> or camelCase for new capabilities?\n\nI personally prefer kebab-case but I think that is a discussion for\nfuture contributions ;-)\n\n\n>> +------------------------\n>> +packet: git> command=smudge\n>> +packet: git> pathname=path/testfile.dat\n>> +packet: git> 0000\n>> +packet: git> CONTENT\n>> +packet: git> 0000\n>> +------------------------\n> \n> I think it is important to mention that (at least with current\n> `filter..process` implementation, that is absent future\n> \"stream\" capability / option) the filter process needs to read\n> *whole contents* at once, *before* writing anything. Otherwise\n> it can lead to deadlock.\n> \n> This is especially important in that it is different (!) from the\n> current behavior of `clean` and `smudge` filters, which can\n> stream their response because Git invokes them async.\n\nI added this:\n\" Please note, that the filter\nmust not send any response before it received the content and the\nfinal flush packet. \"\n\n\n>> +\n>> +If the filter experiences an error during processing, then it can\n>> +send the status \"error\" after the content was (partially or\n>> +completely) sent. Depending on the `filter..required` flag\n>> +Git will interpret that as error but it will not stop or restart the\n>> +filter process.\n>> +------------------------\n>> +packet: git< status=success\n>> +packet: git< 0000\n>> +packet: git< HALF_WRITTEN_ERRONEOUS_CONTENT\n>> +packet: git< 0000\n>> +packet: git< status=error\n>> +packet: git< 0000\n>> +------------------------\n> \n> Good. A question is if the filter process can send \"status=abort\"\n> after partial contents, or does it need to wait for the next command?\n\nI added:\n\"expected to respond with an \"abort\" status at any point in\nthe protocol.\"\n\n\n>> +\n>> +After the filter has processed a blob it is expected to wait for\n>> +the next \"key=value\" list containing a command. Git will close\n>> +the command pipe on exit. The filter is expected to detect EOF\n>> +and exit gracefully on its own.\n> \n> Good to have it documented. \n> \n> Anyway, as it is Git command that spawns the filter driver process,\n> assuming that the filter process doesn't daemonize itself, wouldn't\n> the operating system reap it after its parent process, that is the\n> git command it invoked, dies? So detecting EOF is good, but not\n> strictly necessary for simple filter that do not need to free\n> its resources, or can leave freeing resources to the operating\n> system? But I may be wrong here.\n\nThe filter process runs independent of Git.\n\n\n>> +\n>> +\n>> Interaction between checkin/checkout attributes\n>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>> \n>> diff --git a/contrib/long-running-filter/example.pl b/contrib/long-running-filter/example.pl\n>> new file mode 100755\n>> index 0000000..c13a631\n>> --- /dev/null\n>> +++ b/contrib/long-running-filter/example.pl\n> \n> To repeat myself, I think it would serve better as a separate patch.\n\nOK\n\n\n>> + die \"invalid packet size '$bytes_read' field\";\n> \n> This would read \"invalid packet size '000' field\", for example.\n> Perhaps the following would be (slightly) better:\n> \n> + die \"invalid packet size field: '$bytes_read'\";\n\nOK\n\n\n>> + }\n>> + elsif ( $pkt_size > 4 ) {\n> \n> Isn't a packet of $pkt_size == 4 a valid packet, a keep-alive\n> one? Or is it forbidden?\n\n\"Implementations SHOULD NOT send an empty pkt-line (\"0004\").\"\nSource: Documentation/technical/protocol-common.txt\n\n\n>> + die \"invalid packet ($content_size expected; $bytes_read read)\";\n> \n> This error message would read \"invalid packet (12 expected; 10 read)\";\n> I think it would be better to rephrase it as\n> \n> + die \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n\nOK\n\n\n>> + die \"invalid packet size\";\n> \n> I'm not sure if it is worth it (especially for the demo script),\n> but perhaps we could show what this invalid size was?\n> \n> + die \"invalid packet size value '$pkt_size'\";\n\nOK\n\n\n>> +sub packet_txt_read {\n>> + my ( $res, $buf ) = packet_bin_read();\n>> + unless ( $buf =~ /\\n$/ ) {\n> \n> Wouldn't\n> \n> + unless ( $buf =~ s/\\n$// ) {\n> \n> or (less so)\n> \n> + unless ( $buf =~ s/\\n$\\z// ) {\n> \n> be more idiomatic (and not require use of 'substr')? Remember,\n> the s/// substitution quote-like operator returns number of\n> substitutions in the scalar context.\n\nOK.\n\n\n>> + die \"A non-binary line SHOULD BE terminated by an LF.\";\n> \n> This is SHOULD be, not MUST be, so perhaps 'warn' would be enough.\n> Not that Git should send us such line.\n\nActually it MUST per protocol definition. I'll change it to MUST.\n\n\n>> + my ($packet) = @_;\n> \n> This is equivalent to\n> \n> + my $packet = shift;\n> \n> which, I think, is more common for single-parameter subroutines.\n> \n> Also, this is $data (or $buf), not $packet.\n\nOK\n\n\n> Perhaps some comment that main begins here?\n> \n>> +( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n>> +( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n>> +( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n> \n> Actually, it is overly strict. It should not fail if there\n> are other \"version=3\", \"version=4\" etc. lines.\n\nTrue, but I think for an example this is OK. I'll add a note\nto the file header.\n\n\n>> +\n>> +while (1) {\n>> + my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n>> + my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;\n> \n> Do we require this order? If it is, is that explained in the\n> documentation?\n\nGit sends that order right now but the filter should not rely\non that order.\n\n\n>> + packet_flush(); # empty list!\n> \n> This is less \"empty list!\", and more keeping \"status=success\" unchanged.\n\nOK\n\n\nOK means, I agree and I added your suggestion to v9.\nThanks a lot for your review and the comments!\n\nCheers,\nLars"},{"id":"303013","messageId":"EF3723DE-B34A-4314-94C9-E3EB38EAB92A@gmail.com","threadId":"44122","inReplyTo":"2a604438-b6cd-876d-0ec2-90027dea99b9@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-09-30T19:38:31Z","receivedAt":"2016-09-30T19:39:41Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 27 Sep 2016, at 17:37, Jakub Narębski wrote:\n> \n> Part second of the review of 11/11.\n> \n> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com pisze:\n> \n>> +\n>> +\tif (!drv->process && (CAP_CLEAN & wanted_capability) && drv->clean)\n> \n> This is just a very minor nitpicking, but wouldn't it be easier\n> to read with those checks reordered?\n> \n> +\tif ((wanted_capability & CAP_CLEAN) && !drv->process && drv->clean)\n\nOK\n\n\n>> +\n>> +\tif (start_command(process)) {\n>> +\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n>> +\t\tkill_multi_file_filter(hashmap, entry);\n>> +\t\treturn NULL;\n>> +\t}\n> \n> I guess there is a reason why we init hashmap entry, try to start\n> external process, then kill entry of unable to start, instead of\n> trying to start external process, and adding hashmap entry when\n> we succeed?\n\nYes. This way I can reuse the kill_multi_file_filter() function.\n\n\n>> +\n>> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> \n> I guess that this is here to handle errors writing to filter\n> by ourself, isn't it?\n\nYes.\n\n\n>> +\t\terror(\"external filter '%s' does not support long running filter protocol\", cmd);\n> \n> We could have described the error here better.\n> \n> +\t\terror(\"external filter '%s' does not support filter protocol version 2\", cmd);\n\nOK\n\n\n>> +static void read_multi_file_filter_values(int fd, struct strbuf *status) {\n> \n> This is more\n> \n> +static void read_multi_file_filter_status(int fd, struct strbuf *status) {\n> \n> It doesn't read arbitrary values, it examines 'metadata' from\n> filter for \"status=\" lines.\n\nTrue!\n\n\n>> +\t\tif (pair[0] && pair[0]->len && pair[1]) {\n>> +\t\t\tif (!strcmp(pair[0]->buf, \"status=\")) {\n>> +\t\t\t\tstrbuf_reset(status);\n>> +\t\t\t\tstrbuf_addbuf(status, pair[1]);\n>> +\t\t\t}\n> \n> So it is last status= line wins behavior?\n\nCorrect.\n\n\n> \n>> +\t\t}\n> \n> Shouldn't we free 'struct strbuf **pair', maybe allocated by the\n> strbuf_split_str() function, and reset to NULL?\n\nTrue. strbuf_list_free() should be enough.\n\n\n>> \n>> +\tfflush(NULL);\n> \n> Why this fflush(NULL) is needed here?\n\nThis flushes all open output streams. The single filter does the same.\n\n\n>> \n>> +\tif (fd >= 0 && !src) {\n>> +\t\tif (fstat(fd, &file_stat) == -1)\n>> +\t\t\treturn 0;\n>> +\t\tlen = xsize_t(file_stat.st_size);\n>> +\t}\n> \n> Errr... is it necessary? The protocol no longer provides size=\n> hint, and neither uses such hint if provided.\n\nWe require the size in write_packetized_from_buf() later.\n\n\n>> +\n>> +\terr = strlen(filter_type) > PKTLINE_DATA_MAXLEN;\n>> +\tif (err)\n>> +\t\tgoto done;\n> \n> Errr... this should never happen. We control which capabilities\n> we pass, it can be only \"clean\" or \"smudge\", nothing else. Those\n> would always be shorter than PKTLINE_DATA_MAXLEN.\n> \n> Never mind that that is \"command=smudge\\n\" etc. that needs to\n> be shorter that PKTLINE_DATA_MAXLEN!\n> \n> So, IMHO it should be at most assert, and needs to be corrected\n> anyway.\n\nOK!\n\n\n> This should never happen, PATH_MAX everywhere is much shorter\n> than PKTLINE_DATA_MAXLEN / LARGE_PACKET_MAX. Or is it?\n> \n> Anyway, we should probably explain or warn\n> \n> \t\terror(\"path name too long: '%s'\", path);\n\nOK\n\n\n>> +\t\t\t/*\n>> +\t\t\t * Something went wrong with the protocol filter.\n>> +\t\t\t * Force shutdown and restart if another blob requires filtering!\n> \n> Is this exclamation mark '!' here necessary?\n> \n\nNo.\n\n\nThanks,\nLars\n\n"},{"id":"303044","messageId":"7C680AE2-7F5A-47E4-8E79-6C3F4AEBD39B@gmail.com","threadId":"44122","inReplyTo":"e8815048-212e-64c6-8811-9185f595aaf6@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-10-01T15:34:23Z","receivedAt":"2016-10-01T15:35:42Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Sep 2016, at 01:14, Jakub Narębski wrote:\n> \n> Part third (and last) of the review of v8 11/11.\n> \n> W dniu 20.09.2016 o 21:02, larsxschneider@gmail.com napisał:\n> \n> \n>> @@ -31,7 +31,10 @@ test_expect_success setup '\n>> \tcat test >test.i &&\n>> \tgit add test test.t test.i &&\n>> \trm -f test test.t test.i &&\n>> -\tgit checkout -- test test.t test.i\n>> +\tgit checkout -- test test.t test.i &&\n>> +\n>> +\techo \"content-test2\" >test2.o &&\n>> +\techo \"content-test3 - subdir\" >\"test3 - subdir.o\"\n> \n> I see that you prepare here a few uncommitted files, but both\n> their names and their contents leave much to be desired - you\n> don't know from the name and contents what they are for.\n> \n> And the '\"subdir\"' file which is not in subdirectory is\n> especially egregious.\n\nThese are 3 files with somewhat random test content. I renamed\n\"subdir\" to \"spaces\".\n\n\n>> +check_filter () {\n>> +\trm -f rot13-filter.log actual.log &&\n>> +\t\"$@\" 2> git_stderr.log &&\n>> +\ttest_must_be_empty git_stderr.log &&\n>> +\tcat >expected.log &&\n> \n> This is too clever by half. Having a function that both tests\n> the behavior and prepares 'expected' file is too much.\n> \n> In my opinion preparation of 'expected.log' file should be moved\n> to another function or functions.\n> \n> Also, if we are running sort on output, I think we should also\n> run sort on 'expected.log', so that what we write doesn't need to\n> be created sorted (so we don't have to sort expected lines by hand).\n> Or maybe we should run the same transformation on rot13-filter.log\n> and on the contents of expected.log.\n\nAgreed. Very good suggestion!\n\n\n>> +check_filter_ignore_clean () {\n>> +\trm -f rot13-filter.log actual.log &&\n>> +\t\"$@\" &&\n> \n> Why we don't check for stderr here?\n\nBecause this function is used by \"git checkout\" which writes all\nkinds of stuff to stderr. I added \"--quiet --no-progress\" to\ndisable this behavior. \n\n\n>> +check_rot13 () {\n>> +\ttest_cmp \"$1\" \"$2\" &&\n>> +\t./../rot13.sh <\"$1\" >expected &&\n> \n> Why there is .. in this invocation?\n\nBecause this script is located in the root of the current test directory.\n\n\n>> +\tgit cat-file blob :\"$2\" >actual &&\n>> +\ttest_cmp expected actual\n>> +}\n>> +\n>> +test_expect_success PERL 'required process filter should filter data' '\n>> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n>> +\ttest_config_global filter.protocol.required true &&\n>> +\trm -rf repo &&\n>> +\tmkdir repo &&\n>> +\t(\n>> +\t\tcd repo &&\n>> +\t\tgit init &&\n> \n> Don't you think that creating a fresh test repository for each\n> separate test is a bit too much? I guess that you want for\n> each and every test to be completely independent, but this setup\n> and teardown is a bit excessive.\n> \n> Other tests in the same file (should we reuse the test, or use\n> new test file) do not use this method.\n\nI see your point. However, I am always annoyed if Git tests are\nentangled because it makes working with them way way harder.\nThis test test runs in 4.5s on a slow Travis CI machine. I think\nthat is OK considering that we have tests running 3.5min (t3404).\n\n\n>> +\t\techo \"*.r filter=protocol\" >.gitattributes &&\n>> +\t\tgit add . &&\n>> +\t\tgit commit . -m \"test commit\" &&\n>> +\t\tgit branch empty &&\n> \n> Err... I think it would be better to name it 'empty-branch'\n> (or 'almost-empty-branch', as it does include .gitattributes file).\n> See my mistake below (marked ...).\n\n\"empty-branch\". OK\n\n\n>> +\n>> +\t\tcp ../test.o test.r &&\n>> +\t\tcp ../test2.o test2.r &&\n> \n> What does this test2.o / test2.r file tests, that test.o / test.r\n> doesn't? The name doesn't tell us.\n\nThis just tests multiple files with different content.\n\n\n> Why it is test.r, but test2.r? Why it isn't test1.r?\n\ntest.r already existed (created in setup test).\n\n\n>> +\t\tmkdir testsubdir &&\n>> +\t\tcp \"../test3 - subdir.o\" \"testsubdir/test3 - subdir.r\" &&\n> \n> Why it needs to have different contents?\n\nTo check that the filer does the right thing with multiple files\nand contents.\n\n\n\n>> +\t\t>test4-empty.r &&\n> \n> You test ordinary file, file in subdirectory, file with filename\n> containing spaces, and an empty file.\n> \n> Other tests of single file `clean`/`smudge` filters use filename\n> that requires mangling; maybe we should use similar file?\n> \n> special=\"name with '\\''sq'\\'' and \\$x\" &&\n> echo some test text >\"$special\" &&\n\nOK.\n\n\n> In case of `process` filter, a special filename could look like\n> this:\n> \n> process_special=\"name=with equals and\\nembedded newlines\\n\" &&\n> echo some test text >\"$process_special\" &&\n\nI think this test would create trouble on Windows. I'll stick to\nthe special characters used in the single shot filter.\n\n\n>> +\t\t\t\t<<-\\EOF &&\n>> +\t\t\t\t\t1 IN: clean test.r 57 [OK] -- OUT: 57 . [OK]\n>> +\t\t\t\t\t1 IN: clean test2.r 14 [OK] -- OUT: 14 . [OK]\n>> +\t\t\t\t\t1 IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]\n>> +\t\t\t\t\t1 IN: clean testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n>> +\t\t\t\t\t1 START\n>> +\t\t\t\t\t1 STOP\n>> +\t\t\t\t\t1 wrote filter header\n>> +\t\t\t\tEOF\n> \n> First, this indentation level confirms that the check_filter\n> function is too clever by half, and that preparing expected.log\n> file should be a separate step.\n\nAgreed.\n\n\n> Second, if we run \"sort\" on contents to be in expected.log, we\n> can write it in more natural, and less fragile way:\n\nAgreed.\n\n\n> Third, why the filter even writes output size? It is no longer\n> part of `process` filter driver protocol, and it makes test more\n> fragile.\n\nI would prefer to leave that in. I think it is good for the test to\ncheck that we are transmitting the amount of content that what we \nthink we transmit.\n\n\n> If we are to keep sizes, then to make test less fragile with\n> respect to changes in contents of tested files, we should use\n> variables containing file size:\n> \n> \t\ttest_r_size=$(wc -c test.r)\n> \t\t...\n> \t\tsort >expected.log <<-EOF &&\n> \t\t...\n> \t\t\t1 IN: clean test.r $test_r_size [OK] -- OUT: $test_r_size . [OK]\n\nAgreed.\n\n\n>> \n>> +\t\trm -f test?.r \"testsubdir/test3 - subdir.r\" &&\n> \n> Why 'test?.r' when we are removing only 'test2.r'; why not be explicit?\n\nTrue!\n\n\n>> +\t\t\t\t<<-\\EOF &&\n>> +\t\t\t\t\tSTART\n>> +\t\t\t\t\twrote filter header\n>> +\t\t\t\t\tSTOP\n>> +\t\t\t\tEOF\n> \n> Why is even filter process invoked? If this is not expected, perhaps\n> simply ignore what checking out almost empty branch (one without any\n> files marked for filtering) does.\n> \n> Shouldn't we test_expect_failure no-call?\n\nBecause a clean operation could happen. I added a clean operation to\nthe expected log in order to make this visible (expected log is stripped\nof clean operations in the same way as the actual log per your suggestion\nabove).\n\n\n>> +\n>> +\t\tcheck_filter_ignore_clean \\\n>> +\t\t\tgit checkout master \\\n> \n> Does this checks different code path than 'git checkout .'? For\n> example, does this test increase code coverage (e.g. as measured\n> by gcov)? If not, then this test could be safely dropped.\n\nWe checked out the \"empty-branch\" before. That's why we check here\nthat the smudge filter runs for all files (smudge filter did not run\nfor all files with `git checkout .`).\n\n\n>> +\t\t\t\t<<-\\EOF &&\n>> +\t\t\t\t\tSTART\n>> +\t\t\t\t\twrote filter header\n>> +\t\t\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n>> +\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n>> +\t\t\t\t\tIN: smudge test4-empty.r 0 [OK] -- OUT: 0 [OK]\n>> +\t\t\t\t\tIN: smudge testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n> \n> Can we assume that Git would pass files to filter in alphabetical\n> order? This assumption might make the test unnecessary fragile.\n\nI have never experienced another behavior. If we see fragility we could\nsort the result...\n\n\n>> \n>> +test_expect_success PERL 'required process filter should clean only and take precedence' '\n> \n> Trying to describe it better results in overly long description,\n> which probably means that this test should be split into few\n> smaller ones:\n> \n> - `process` filter takes precedence over `clean` and/or `smudge`\n> filters, regardless if it supports relevant (\"clean\" or \"smudge\")\n> capability or not\n> \n> - `process` filter that includes only \"clean\" capability should\n> clean only (be used only for 'clean' operation)\n\nAgreed!\n\n\n> In my opinion all functions should be placed at beginning,\n> or even in separate file (if they are used in more than\n> one test).\n\nOK\n\n\n>> +generate_test_data () {\n> \n> The name is not good, it doesn't describe what kind of data\n> we want to generate.\n\n\"generate_random_characters\" ok?!\n\n>> +\t\tperl -pe \"s/./chr((ord($&) % 26) + 97)/sge\" >../$NAME.file &&\n> \n> Those constants (26 and 97) are a bit cryptic; magical constants.\n> I guess this is\n> \n> +\t\tperl -pe \"s/./chr((ord($&) % (ord('z') - ord('a') + 1) + ord('a'))/sge\" >../$NAME.file &&\n> \n> or\n> \n> +\t\tperl -pe \"s/./chr((ord($&) % 26 + ord('a'))/sge\" >../$NAME.file &&\n\nOK!\n\n\n> Do we re-generate this file each time?\n> \n>> +\t./../rot13.sh <../$NAME.file >../$NAME.file.rot13\n> \n> Anyway, I wonder if taking the last two lines out of the function\n> (as they are not about _generating_ a file) would make it more\n> readable or not.\n\nAgreed.\n\n\n>> +\n>> +\t\techo \"*.file filter=protocol\" >.gitattributes &&\n>> +\t\tcheck_filter \\\n>> +\t\t\tgit add *.file .gitattributes \\\n> \n> Should it be shell expansion, or git expansion, that is\n> \n> \t\t\tgit add '*.file' .gitattributes\n\nBoth have the same output. Would the difference matter?\n\n\n>> +\t\t\t\t\t1 START\n>> +\t\t\t\t\t1 STOP\n>> +\t\t\t\t\t1 wrote filter header\n>> +\t\t\t\tEOF\n>> +\t\tgit commit . -m \"test commit\" &&\n> \n> Is this needed / necessary?\n\nYes, to test the smudge afterwards!\n\n> \n>> +\n>> +\t\trm -f *.file &&\n>> +\t\tgit checkout -- *.file &&\n> \n> Is this necessary? I guess this checks that it doesn't crash, but\n> we do not check that smudge operation works correctly, as we did\n> for clean.\n\nGood point. Smudge check added!\n\n\n>> +\t\tfor f in *.file\n>> +\t\tdo\n>> +\t\t\tgit cat-file blob :$f >actual &&\n>> +\t\t\ttest_cmp ../$f.rot13 actual\n>> +\t\tdone\n> \n> Wasn't there helper function for this?\n\nTrue :-)\n\n\n>> +test_expect_success PERL 'required process filter should with clean error should fail' '\n> ^^^^^^ ^^^^^^\n> \n> Errr... what? You have 'should' twice here.\n\nFixed\n\n\n> Also, does it matter that the error is during clean operation?\n> We don't test that error during smudge operation is handled in\n> the same way, do we?\n\nClean and smudge should hit the same code paths here. Therefore I think\nit is sufficient to test clean only.\n\n\n>> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n> \n> Do we need to pass 'clean smudge', or does it provide both by\n> default?\n\nWe need to pass them. Default is empty.\n\n\n>> +\t\tgit add . &&\n>> +\t\tgit commit . -m \"test commit\" &&\n> \n> You don't need to commit for 'git checkout ' (e.g. for .)\n> or 'git cat-file -p :' to work.\n\nTrue!\n\n\n>> +\t)\n>> +'\n>> +\n>> +test_expect_success PERL 'process filter should not restart in case of an error' '\n> \n> Errr... what? This description is not clear. Did you mean\n> that filter should not be restarted if it *signals* an error\n> with file (either before sending anything, or after sending\n> partial contents)?\n\nOK renamed to \"process filter should not be restarted if it signals an error\"\n\n\n>> +test_expect_success PERL 'process filter should be able to signal an error for all future files' '\n> \n> Did you mean here that filter can abort processing of\n> all future files?\n\n\"process filter signals abort once to abort processing of all future files\", better?\n\n\n>> +\n>> +\t\tcp ../test.o test.r &&\n>> +\t\ttest_must_fail git add . 2> git_stderr.log &&\n>> +\t\tgrep \"not support long running filter protocol\" git_stderr.log\n> \n> Shouldn't this use gettext poison (or rather C locale)?\n> This error message could be translated in the future.\n\nI would prefer to adjust that when we translate it.\n\n\n>> + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n> \n> Why not use tr/// version of this quote-like operation?\n> Or do you follow prior art here?\n\nI am not Perl expert. That worked for me :-)\n\n\n>> +sub packet_bin_read {\n>> + my $buffer;\n>> + my $bytes_read = read STDIN, $buffer, 4;\n>> + if ( $bytes_read == 0 ) {\n>> +\n>> + # EOF - Git stopped talking to us!\n>> + print $debug \"STOP\\n\";\n>> + exit();\n>> + }\n>> + elsif ( $bytes_read != 4 ) {\n>> + die \"invalid packet size '$bytes_read' field\";\n> \n> Errr, $bytes_read is not packet size field. It is $buffer.\n> Also, error message looks strange\n> \n> \t\tinvalid packet size '004' field\n> \n> Shouldn't it be at end?\n\nTrue. Fixed!\n\n\n>> + }\n>> + return ( 0, $buffer );\n>> + }\n>> + else {\n>> + die \"invalid packet size\";\n> \n> Is keep-alive packet valid (\"0004\")?\n\nNo.\n\n\n>> \n>> +packet_flush();\n>> +print $debug \"wrote filter header\\n\";\n> \n> Or perhaps \"handshake end\"?\n\n\"init handshake complete\", ok?\n\n\n>> + print $debug \" $pathname\";\n> \n> No \" pathname=$pathname\" ?\n\nYes, otherwise it gets too verbose in the tests.\n\n\n>> + while ( length($output) > 0 ) {\n>> + my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n>> + packet_bin_write($packet);\n>> + print $debug \".\";\n> \n> All right, so number of dots is the number of packets. This is\n> surprisingly opaque.\n\nI added a comment.\n\n\n> \n>> + if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n>> + $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n>> + }\n>> + else {\n>> + $output = \"\";\n>> + }\n>> + }\n>> + packet_flush();\n>> + print $debug \" [OK]\\n\";\n>> + $debug->flush();\n>> + packet_flush();\n> \n> Should we test partial contents case? Or failure during printing?\n> What happens then - is file cleared by Git, or left partially converted?\n\nGit will clear the file on any error (it doesn't matter when the error happens).\n\n---\n\nI am astonished how many valuable suggestion you were able to make\neven though I am working with this code for months now.\n\nThanks a lot for taking the time to review my code that thoroughly.\n\n- Lars"},{"id":"303050","messageId":"C53500E8-7352-4AAC-9F53-40CCFA7F1418@gmail.com","threadId":"44122","inReplyTo":"xmqq60pee6rp.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-10-01T18:59:48Z","receivedAt":"2016-10-01T19:00:50Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Sep 2016, at 23:27, Junio C Hamano wrote:\n> \n> Lars Schneider writes:\n> \n>> We discussed that issue in v4 and v6:\n>> http://public-inbox.org/git/20160803225313.pk3tfe5ovz4y3i7l@sigill.intra.peff.net/\n>> http://public-inbox.org/git/xmqqbn0a3wy3.fsf@gitster.mtv.corp.google.com/\n>> \n>> My impression was that you don't want Git to wait for the filter process.\n>> If Git waits for the filter process - how long should Git wait?\n> \n> I am not sure where you got that impression. I did say that I do\n> not want Git to _KILL_ my filter process. That does not mean I want\n> Git to go away without waiting for me.\n> \n> If the filter process refuses to die forever when Git told it to\n> shutdown (by closing the pipe to it, for example), that filter\n> process is simply buggy. I think we want users to become aware of\n> that, instead of Git leaving it behind, which essentially is to\n> sweep the problem under the rug.\n> \n> I agree with what Peff said elsewhere in the thread; if a filter\n> process wants to take time to clean things up while letting Git\n> proceed, it can do its own process management, but I think it is\n> sensible for Git to wait the filter process it directly spawned.\n\nTo realize the approach above I prototyped the run-command patch below:\n\nI added an \"exit_timeout\" variable to the \"child_process\" struct.\nOn exit, Git will close the pipe to the process and wait \"exit_timeout\" \nseconds until it kills the child process. If \"exit_timeout\" is negative\nthen Git will wait until the process is done.\n\nIf we use that in the long running filter process, then we could make\nthe timeout even configurable. E.g. with \"filter..process-timeout\".\n\nWhat do you think about this solution? \n\nThanks,\nLars\n\n\n\ndiff --git a/run-command.c b/run-command.c\nindex 3269362..a933066 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -21,6 +21,8 @@ void child_process_clear(struct child_process *child)\n \n struct child_to_clean {\n \tpid_t pid;\n+\tint stdin;\n+\tint timeout;\n \tstruct child_to_clean *next;\n };\n static struct child_to_clean *children_to_clean;\n@@ -28,9 +30,30 @@ static int installed_child_cleanup_handler;\n \n static void cleanup_children(int sig, int in_signal)\n {\n+\tint status;\n+\tstruct timeval tv;\n+\ttime_t secs;\n+\n \twhile (children_to_clean) {\n \t\tstruct child_to_clean *p = children_to_clean;\n \t\tchildren_to_clean = p->next;\n+\n+\t\tif (p->timeout != 0 && p->stdin > 0)\n+\t\t\tclose(p->stdin);\n+\n+\t\tif (p->timeout < 0) {\n+\t\t\t// Wait until the process finishes\n+\t\t\twhile ((waitpid(p->pid, &status, 0)) < 0 && errno == EINTR)\n+\t\t\t\t;\t/* nothing */\n+\t\t} else if (p->timeout != 0) {\n+\t\t\t// Wait until the process finishes or timeout\n+\t\t\tgettimeofday(&tv, NULL);\n+\t\t\tsecs = tv.tv_sec;\n+\t\t\twhile (getpgid(p->pid) >= 0 && tv.tv_sec - secs < p->timeout) {\n+\t\t\t\tgettimeofday(&tv, NULL);\n+\t\t\t}\n+\t\t}\n+\n \t\tkill(p->pid, sig);\n \t\tif (!in_signal)\n \t\t\tfree(p);\n@@ -49,10 +72,12 @@ static void cleanup_children_on_exit(void)\n \tcleanup_children(SIGTERM, 0);\n }\n \n-static void mark_child_for_cleanup(pid_t pid)\n+static void mark_child_for_cleanup(pid_t pid, int timeout, int stdin)\n {\n \tstruct child_to_clean *p = xmalloc(sizeof(*p));\n \tp->pid = pid;\n+\tp->stdin = stdin;\n+\tp->timeout = timeout;\n \tp->next = children_to_clean;\n \tchildren_to_clean = p;\n \n@@ -422,7 +447,7 @@ int start_command(struct child_process *cmd)\n \tif (cmd->pid < 0)\n \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n \telse if (cmd->clean_on_exit)\n-\t\tmark_child_for_cleanup(cmd->pid);\n+\t\tmark_child_for_cleanup(cmd->pid, cmd->exit_timeout, cmd->in);\n \n \t/*\n \t * Wait for child's execvp. If the execvp succeeds (or if fork()\n@@ -483,7 +508,7 @@ int start_command(struct child_process *cmd)\n \tif (cmd->pid < 0 && (!cmd->silent_exec_failure || errno != ENOENT))\n \t\terror_errno(\"cannot spawn %s\", cmd->argv[0]);\n \tif (cmd->clean_on_exit && cmd->pid >= 0)\n-\t\tmark_child_for_cleanup(cmd->pid);\n+\t\tmark_child_for_cleanup(cmd->pid, cmd->exit_timeout, cmd->in);\n \n \targv_array_clear(&nargv);\n \tcmd->argv = sargv;\n@@ -765,7 +790,7 @@ int start_async(struct async *async)\n \t\texit(!!async->proc(proc_in, proc_out, async->data));\n \t}\n \n-\tmark_child_for_cleanup(async->pid);\n+\tmark_child_for_cleanup(async->pid, 0, -1);\n \n \tif (need_in)\n \t\tclose(fdin[0]);\ndiff --git a/run-command.h b/run-command.h\nindex cf29a31..f2eca33 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -33,6 +33,7 @@ struct child_process {\n \tint in;\n \tint out;\n \tint err;\n+\tint exit_timeout;\n \tconst char *dir;\n \tconst char *const *env;\n \tunsigned no_stdin:1;\n\n\n"},{"id":"303054","messageId":"15ff438f-ec58-e649-b927-b1de4751cc45@gmail.com","threadId":"44122","inReplyTo":"C53500E8-7352-4AAC-9F53-40CCFA7F1418@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-10-01T20:48:34Z","receivedAt":"2016-10-01T20:52:22Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 01.10.2016 o 20:59, Lars Schneider pisze: \n> On 29 Sep 2016, at 23:27, Junio C Hamano wrote:\n>> Lars Schneider writes:\n>>\n>>> We discussed that issue in v4 and v6:\n>>> http://public-inbox.org/git/20160803225313.pk3tfe5ovz4y3i7l@sigill.intra.peff.net/\n>>> http://public-inbox.org/git/xmqqbn0a3wy3.fsf@gitster.mtv.corp.google.com/\n>>>\n>>> My impression was that you don't want Git to wait for the filter process.\n>>> If Git waits for the filter process - how long should Git wait?\n>>\n>> I am not sure where you got that impression. I did say that I do\n>> not want Git to _KILL_ my filter process. That does not mean I want\n>> Git to go away without waiting for me.\n>>\n>> If the filter process refuses to die forever when Git told it to\n>> shutdown (by closing the pipe to it, for example), that filter\n>> process is simply buggy. I think we want users to become aware of\n>> that, instead of Git leaving it behind, which essentially is to\n>> sweep the problem under the rug.\n\nWell, it would be good to tell users _why_ Git is hanging, see below.\n\n>>\n>> I agree with what Peff said elsewhere in the thread; if a filter\n>> process wants to take time to clean things up while letting Git\n>> proceed, it can do its own process management, but I think it is\n>> sensible for Git to wait the filter process it directly spawned.\n> \n> To realize the approach above I prototyped the run-command patch below:\n> \n> I added an \"exit_timeout\" variable to the \"child_process\" struct.\n> On exit, Git will close the pipe to the process and wait \"exit_timeout\" \n> seconds until it kills the child process. If \"exit_timeout\" is negative\n> then Git will wait until the process is done.\n\nThat might be good approach. Probably the default would be to wait.\n\n> \n> If we use that in the long running filter process, then we could make\n> the timeout even configurable. E.g. with \"filter..process-timeout\".\n\nSidenote: we prefer camelCase rather than kebab-case for config\nvariables, that is, \"filter..processTimeout\".\n\nAlso, how would one set default value of timeout for all process\nbased filters?\n\n> \n> What do you think about this solution?\n\nI think this addition be done after, assuming that we come up\nwith good default behavior (e.g. wait for filter processes\nto finish).\n\nAlso, we would probably want to add some progress information\nin a similar way to progress info for checkout, that is display\nit after a few seconds waiting.\n\nThis could be, for example:\n\n Waiting for filter '' to finish... done\n\nWith timeout it could look like this (where underlined part\nis interactive, that is changes every second):\n\n Waiting 10s for '' filter process to finish.\n ^^^\n\nAnd then either\n\n Filter '' killed\n\nor\n\n Filter '' finished\n\n\n> diff --git a/run-command.c b/run-command.c\n> index 3269362..a933066 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -21,6 +21,8 @@ void child_process_clear(struct child_process *child)\n> \n> struct child_to_clean {\n> \tpid_t pid;\n> +\tint stdin;\n> +\tint timeout;\n> \tstruct child_to_clean *next;\n> };\n> static struct child_to_clean *children_to_clean;\n> @@ -28,9 +30,30 @@ static int installed_child_cleanup_handler;\n> \n> static void cleanup_children(int sig, int in_signal)\n> {\n> +\tint status;\n> +\tstruct timeval tv;\n> +\ttime_t secs;\n> +\n> \twhile (children_to_clean) {\n> \t\tstruct child_to_clean *p = children_to_clean;\n> \t\tchildren_to_clean = p->next;\n> +\n> +\t\tif (p->timeout != 0 && p->stdin > 0)\n> +\t\t\tclose(p->stdin);\n\nWhy are you not closing stdin of filter driver process if timeout\nis zero? Is it maybe some kind of special value? If it is, this\nis not documented.\n\n> +\n> +\t\tif (p->timeout < 0) {\n> +\t\t\t// Wait until the process finishes\n> +\t\t\twhile ((waitpid(p->pid, &status, 0)) < 0 && errno == EINTR)\n> +\t\t\t\t;\t/* nothing */\n\nAh, this loop is here because waiting on waitpid() can be interrupted\nby the delivery of a signal to the calling process; though the result\nis -1, not just any < 0.\n\nLooks good (but we would want some progress information, probably).\n\n> +\t\t} else if (p->timeout != 0) {\n> +\t\t\t// Wait until the process finishes or timeout\n> +\t\t\tgettimeofday(&tv, NULL);\n> +\t\t\tsecs = tv.tv_sec;\n> +\t\t\twhile (getpgid(p->pid) >= 0 && tv.tv_sec - secs < p->timeout) {\n> +\t\t\t\tgettimeofday(&tv, NULL);\n> +\t\t\t}\n\nWTF? Busy wait loop???\n\nAlso, if we want to wait for child without blocking, then instead\nof cryptic getpgid(p->pid) maybe use waitpid(p->pid, &status, WNOHANG);\nit is more explicit.\n\nThere is also another complication: there can be more than one\nlong-running filter driver used. With this implementation we\nwait for each of one in sequence, e.g. 10s + 10s + 10s.\n\n> +\t\t}\n> +\n> \t\tkill(p->pid, sig);\n> \t\tif (!in_signal)\n> \t\t\tfree(p);\n> @@ -49,10 +72,12 @@ static void cleanup_children_on_exit(void)\n> \tcleanup_children(SIGTERM, 0);\n> }\n> \n> -static void mark_child_for_cleanup(pid_t pid)\n> +static void mark_child_for_cleanup(pid_t pid, int timeout, int stdin)\n\nHmmmm... three parameters is not too much, but we might want to\npass \"struct child_process *\" directly if this number grows.\n\n> {\n> \tstruct child_to_clean *p = xmalloc(sizeof(*p));\n> \tp->pid = pid;\n> +\tp->stdin = stdin;\n> +\tp->timeout = timeout;\n> \tp->next = children_to_clean;\n> \tchildren_to_clean = p;\n> \n> @@ -422,7 +447,7 @@ int start_command(struct child_process *cmd)\n> \tif (cmd->pid < 0)\n> \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n> \telse if (cmd->clean_on_exit)\n> -\t\tmark_child_for_cleanup(cmd->pid);\n> +\t\tmark_child_for_cleanup(cmd->pid, cmd->exit_timeout, cmd->in);\n> \n> \t/*\n> \t * Wait for child's execvp. If the execvp succeeds (or if fork()\n> @@ -483,7 +508,7 @@ int start_command(struct child_process *cmd)\n> \tif (cmd->pid < 0 && (!cmd->silent_exec_failure || errno != ENOENT))\n> \t\terror_errno(\"cannot spawn %s\", cmd->argv[0]);\n> \tif (cmd->clean_on_exit && cmd->pid >= 0)\n> -\t\tmark_child_for_cleanup(cmd->pid);\n> +\t\tmark_child_for_cleanup(cmd->pid, cmd->exit_timeout, cmd->in);\n> \n> \targv_array_clear(&nargv);\n> \tcmd->argv = sargv;\n> @@ -765,7 +790,7 @@ int start_async(struct async *async)\n> \t\texit(!!async->proc(proc_in, proc_out, async->data));\n> \t}\n> \n> -\tmark_child_for_cleanup(async->pid);\n> +\tmark_child_for_cleanup(async->pid, 0, -1);\n\nEh? A bit magic.\n\n> \n> \tif (need_in)\n> \t\tclose(fdin[0]);\n> diff --git a/run-command.h b/run-command.h\n> index cf29a31..f2eca33 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -33,6 +33,7 @@ struct child_process {\n> \tint in;\n> \tint out;\n> \tint err;\n> +\tint exit_timeout;\n\nI guess it is no problem adding new field to child_process struct;\nit is not constrained for memory...\n\n> \tconst char *dir;\n> \tconst char *const *env;\n> \tunsigned no_stdin:1;\n> \n\nWhere we read the value of the configuration variable?\n\n"},{"id":"303095","messageId":"xmqqvax974dl.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"C53500E8-7352-4AAC-9F53-40CCFA7F1418@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-03T17:02:14Z","receivedAt":"2016-10-03T17:02:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n>> If the filter process refuses to die forever when Git told it to\n>> shutdown (by closing the pipe to it, for example), that filter\n>> process is simply buggy. I think we want users to become aware of\n>> that, instead of Git leaving it behind, which essentially is to\n>> sweep the problem under the rug.\n>> \n>> I agree with what Peff said elsewhere in the thread; if a filter\n>> process wants to take time to clean things up while letting Git\n>> proceed, it can do its own process management, but I think it is\n>> sensible for Git to wait the filter process it directly spawned.\n>\n> To realize the approach above I prototyped the run-command patch below:\n>\n> I added an \"exit_timeout\" variable to the \"child_process\" struct.\n> On exit, Git will close the pipe to the process and wait \"exit_timeout\" \n> seconds until it kills the child process. If \"exit_timeout\" is negative\n> then Git will wait until the process is done.\n\n> If we use that in the long running filter process, then we could make\n> the timeout even configurable. E.g. with \"filter..process-timeout\".\n>\n> What do you think about this solution? \n\nIs such a configuration (or timeout in general) necessary? I\nsuspect that a need for timeout, especially needing timeout and\nneeding to get killed that happens so often to require a\nconfiguration variable, is a sign of something else seriously wrong.\n\nWhat's the justification for a filter to _require_ getting killed\nall the time when it is spawned? Otherwise you wouldn't configure\n\"this driver does not die when told, so we need a timeout\" variable.\nIs it a sign of the flaw in the protocol to talk to it? e.g. Git\nhas a way to tell it to die, but it somehow is very hard to hear\nfrom filter's end and honor that request?\n\nI think that we would need some timeout in the mechanism, but not to\nbe used for \"killing\".\n\nYou would decide to \"kill\" an filter process in two cases: the\nfilter is buggy and refuses to die when Git tells it to exit, or the\ncode in Git waiting for its death is somehow miscounting its\nchildren, and thought it told to die one process but in fact it\ndidn't (perhaps it told somebody else to die), or it thought it\nhasn't seen the child die when in fact it already did.\n\nCalling kill(2) and exiting would hide these two kind of bugs from\nend users. Not doing so would give the end users a hung Git, which\nis a VERY GOOD thing. Otherwise you would not notice bugs and lose\nthe opportunity to diagnose and fix it.\n\nThe timeout would be good for you to give a message \"filter process\nrunning the script '%s' is not exiting; I am waiting for it\". The\nuser is still left with a hung Git, and can then see if that process\nis hanging around. If it is, then we found a buggy filter. Or we\nfound a buggy Git. Either needs to be fixed. I do not think it\nwould help anybody by doing a kill(2) to sweep possible bugs under\nthe rug.\n\nThanks.\n"},{"id":"303097","messageId":"E9946E9F-6EE5-492B-B122-9078CEB88044@gmail.com","threadId":"44122","inReplyTo":"15ff438f-ec58-e649-b927-b1de4751cc45@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-10-03T17:13:37Z","receivedAt":"2016-10-03T17:13:45Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 01 Oct 2016, at 22:48, Jakub Narębski wrote:\n> \n> W dniu 01.10.2016 o 20:59, Lars Schneider pisze: \n>> On 29 Sep 2016, at 23:27, Junio C Hamano wrote:\n>>> Lars Schneider writes:\n>>> \n>>> If the filter process refuses to die forever when Git told it to\n>>> shutdown (by closing the pipe to it, for example), that filter\n>>> process is simply buggy. I think we want users to become aware of\n>>> that, instead of Git leaving it behind, which essentially is to\n>>> sweep the problem under the rug.\n> \n> Well, it would be good to tell users _why_ Git is hanging, see below.\n\nAgreed. Do you think it is OK to write the message to stderr?\n\n\n>>> I agree with what Peff said elsewhere in the thread; if a filter\n>>> process wants to take time to clean things up while letting Git\n>>> proceed, it can do its own process management, but I think it is\n>>> sensible for Git to wait the filter process it directly spawned.\n>> \n>> To realize the approach above I prototyped the run-command patch below:\n>> \n>> I added an \"exit_timeout\" variable to the \"child_process\" struct.\n>> On exit, Git will close the pipe to the process and wait \"exit_timeout\" \n>> seconds until it kills the child process. If \"exit_timeout\" is negative\n>> then Git will wait until the process is done.\n> \n> That might be good approach. Probably the default would be to wait.\n\nI think I would prefer a 2sec timeout or something as default. This way\nwe can ensure Git would not wait indefinitely for a buggy filter by default.\n\n\n>> If we use that in the long running filter process, then we could make\n>> the timeout even configurable. E.g. with \"filter..process-timeout\".\n> \n> Sidenote: we prefer camelCase rather than kebab-case for config\n> variables, that is, \"filter..processTimeout\".\n\nSure!\n\n\n> Also, how would one set default value of timeout for all process\n> based filters?\n\nI think we don't need that because a timeout is always specific\nto a filter (if the 2sec default is not sufficient).\n\n\n>> \n>> +\t\t\twhile ((waitpid(p->pid, &status, 0)) < 0 && errno == EINTR)\n>> +\t\t\t\t;\t/* nothing */\n> \n> Ah, this loop is here because waiting on waitpid() can be interrupted\n> by the delivery of a signal to the calling process; though the result\n> is -1, not just any < 0.\n\n\"< 0\" is also used in wait_or_whine()\n\n\n>> +\t\t\twhile (getpgid(p->pid) >= 0 && tv.tv_sec - secs < p->timeout) {\n>> +\t\t\t\tgettimeofday(&tv, NULL);\n>> +\t\t\t}\n> \n> WTF? Busy wait loop???\n\nThis was just a quick prototype to show \"my thinking direction\". \nI wasn't expecting a review. Sorry :-)\n\n\n> Also, if we want to wait for child without blocking, then instead\n> of cryptic getpgid(p->pid) maybe use waitpid(p->pid, &status, WNOHANG);\n> it is more explicit.\n\nSure!\n\n\n> There is also another complication: there can be more than one\n> long-running filter driver used. With this implementation we\n> wait for each of one in sequence, e.g. 10s + 10s + 10s.\n\nGood idea, I fixed that in the version below!\n\n\n>> \n>> -static void mark_child_for_cleanup(pid_t pid)\n>> +static void mark_child_for_cleanup(pid_t pid, int timeout, int stdin)\n> \n> Hmmmm... three parameters is not too much, but we might want to\n> pass \"struct child_process *\" directly if this number grows.\n\nI used parameters because this function is also used with the async struct... \n\nI've attached a more serious patch for review below.\nWhat do you think?\n\nThanks,\nLars\n\n\n\ndiff --git a/run-command.c b/run-command.c\nindex 3269362..ca0feef 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -21,6 +21,9 @@ void child_process_clear(struct child_process *child)\n \n struct child_to_clean {\n \tpid_t pid;\n+\tchar *name;\n+\tint stdin;\n+\tint timeout;\n \tstruct child_to_clean *next;\n };\n static struct child_to_clean *children_to_clean;\n@@ -28,12 +31,53 @@ static int installed_child_cleanup_handler;\n \n static void cleanup_children(int sig, int in_signal)\n {\n+\tint status;\n+\tstruct timeval tv;\n+\ttime_t secs;\n+\tstruct child_to_clean *p = children_to_clean;\n+\n+\t// Send EOF to children as indicator that Git will exit soon\n+\twhile (p) {\n+\t\tif (p->timeout != 0) {\n+\t\t\tif (p->stdin > 0)\n+\t\t\t\tclose(p->stdin);\n+\t\t}\n+\t\tp = p->next;\n+\t}\n+\n \twhile (children_to_clean) {\n-\t\tstruct child_to_clean *p = children_to_clean;\n+\t\tp = children_to_clean;\n \t\tchildren_to_clean = p->next;\n+\n+\t\tif (p->timeout != 0) {\n+\t\t\tfprintf(stderr, _(\"Waiting for '%s' to finish...\"), p->name);\n+\t\t\tif (p->timeout < 0) {\n+\t\t\t\t// No timeout given - wait indefinitely\n+\t\t\t\twhile ((waitpid(p->pid, &status, 0)) < 0 && errno == EINTR)\n+\t\t\t\t\t;\t/* nothing */\n+\t\t\t} else {\n+\t\t\t\t// Wait until timeout\n+\t\t\t\tgettimeofday(&tv, NULL);\n+\t\t\t\tsecs = tv.tv_sec;\n+\t\t\t\twhile (!waitpid(p->pid, &status, WNOHANG) &&\n+\t\t\t\t\t tv.tv_sec - secs < p->timeout) {\n+\t\t\t\t\tfprintf(stderr, _(\" \\rWaiting %lds for '%s' to finish...\"),\n+\t\t\t\t\t\tp->timeout - tv.tv_sec + secs - 1, p->name);\n+\t\t\t\t\tgettimeofday(&tv, NULL);\n+\t\t\t\t\tsleep_millisec(10);\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tif (waitpid(p->pid, &status, WNOHANG))\n+\t\t\t\tfprintf(stderr, _(\"done!\\n\"));\n+\t\t\telse\n+\t\t\t\tfprintf(stderr, _(\"timeout. Killing...\\n\"));\n+\t\t}\n+\n \t\tkill(p->pid, sig);\n-\t\tif (!in_signal)\n+\t\tif (!in_signal) {\n+\t\t\tfree(p->name);\n \t\t\tfree(p);\n+\t\t}\n \t}\n }\n \n@@ -49,10 +93,18 @@ static void cleanup_children_on_exit(void)\n \tcleanup_children(SIGTERM, 0);\n }\n \n-static void mark_child_for_cleanup(pid_t pid)\n+static void mark_child_for_cleanup_with_timeout(pid_t pid, const char *name, int stdin, int timeout)\n {\n \tstruct child_to_clean *p = xmalloc(sizeof(*p));\n \tp->pid = pid;\n+\tp->timeout = timeout;\n+\tp->stdin = stdin;\n+\tif (name) {\n+\t\tp->name = xmalloc(strlen(name) + 1);\n+\t\tstrcpy(p->name, name);\n+\t} else {\n+\t\tp->name = \"process\";\n+\t}\n \tp->next = children_to_clean;\n \tchildren_to_clean = p;\n \n@@ -63,6 +115,13 @@ static void mark_child_for_cleanup(pid_t pid)\n \t}\n }\n \n+#ifdef NO_PTHREADS\n+static void mark_child_for_cleanup(pid_t pid, const char *name, int timeout, int stdin)\n+{\n+\tmark_child_for_cleanup_with_timeout(pid, NULL, 0, 0);\n+}\n+#endif\n+\n static void clear_child_for_cleanup(pid_t pid)\n {\n \tstruct child_to_clean **pp;\n@@ -422,7 +481,8 @@ int start_command(struct child_process *cmd)\n \tif (cmd->pid < 0)\n \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n \telse if (cmd->clean_on_exit)\n-\t\tmark_child_for_cleanup(cmd->pid);\n+\t\tmark_child_for_cleanup_with_timeout(\n+\t\t\tcmd->pid, cmd->argv[0], cmd->in, cmd->clean_on_exit_timeout);\n \n \t/*\n \t * Wait for child's execvp. If the execvp succeeds (or if fork()\n@@ -483,7 +543,8 @@ int start_command(struct child_process *cmd)\n \tif (cmd->pid < 0 && (!cmd->silent_exec_failure || errno != ENOENT))\n \t\terror_errno(\"cannot spawn %s\", cmd->argv[0]);\n \tif (cmd->clean_on_exit && cmd->pid >= 0)\n-\t\tmark_child_for_cleanup(cmd->pid);\n+\t\tmark_child_for_cleanup_with_timeout(\n+\t\t\tcmd->pid, cmd->argv[0], cmd->in, cmd->clean_on_exit_timeout);\n \n \targv_array_clear(&nargv);\n \tcmd->argv = sargv;\ndiff --git a/run-command.h b/run-command.h\nindex cf29a31..4c1c1f4 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -43,6 +43,16 @@ struct child_process {\n \tunsigned stdout_to_stderr:1;\n \tunsigned use_shell:1;\n \tunsigned clean_on_exit:1;\n+\t/*\n+\t * clean_on_exit_timeout is only considered if clean_on_exit is set.\n+\t * - Specify 0 to kill the child on Git exit (default)\n+\t * - Specify a negative value to close the child's stdin on Git exit\n+\t * and wait indefinitely for the child's termination.\n+\t * - Specify a positive value to close the child's stdin on Git exit\n+\t * and wait clean_on_exit_timeout seconds for the child's\n+\t * termination.\n+\t */\n+\tint clean_on_exit_timeout;\n };\n \n #define CHILD_PROCESS_INIT { NULL, ARGV_ARRAY_INIT, ARGV_ARRAY_INIT }\n\n"},{"id":"303101","messageId":"56FAEF34-6A31-4E10-9671-F890796EF5A6@gmail.com","threadId":"44122","inReplyTo":"xmqqvax974dl.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-10-03T17:35:19Z","receivedAt":"2016-10-03T17:35:27Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 03 Oct 2016, at 19:02, Junio C Hamano wrote:\n> \n> Lars Schneider writes:\n> \n>>> If the filter process refuses to die forever when Git told it to\n>>> shutdown (by closing the pipe to it, for example), that filter\n>>> process is simply buggy. I think we want users to become aware of\n>>> that, instead of Git leaving it behind, which essentially is to\n>>> sweep the problem under the rug.\n>>> \n>>> I agree with what Peff said elsewhere in the thread; if a filter\n>>> process wants to take time to clean things up while letting Git\n>>> proceed, it can do its own process management, but I think it is\n>>> sensible for Git to wait the filter process it directly spawned.\n>> \n>> To realize the approach above I prototyped the run-command patch below:\n>> \n>> I added an \"exit_timeout\" variable to the \"child_process\" struct.\n>> On exit, Git will close the pipe to the process and wait \"exit_timeout\" \n>> seconds until it kills the child process. If \"exit_timeout\" is negative\n>> then Git will wait until the process is done.\n> \n>> If we use that in the long running filter process, then we could make\n>> the timeout even configurable. E.g. with \"filter..process-timeout\".\n>> \n>> What do you think about this solution? \n> \n> Is such a configuration (or timeout in general) necessary? I\n> suspect that a need for timeout, especially needing timeout and\n> needing to get killed that happens so often to require a\n> configuration variable, is a sign of something else seriously wrong.\n> \n> What's the justification for a filter to _require_ getting killed\n> all the time when it is spawned? Otherwise you wouldn't configure\n> \"this driver does not die when told, so we need a timeout\" variable.\n> Is it a sign of the flaw in the protocol to talk to it? e.g. Git\n> has a way to tell it to die, but it somehow is very hard to hear\n> from filter's end and honor that request?\n> \n> I think that we would need some timeout in the mechanism, but not to\n> be used for \"killing\".\n> \n> You would decide to \"kill\" an filter process in two cases: the\n> filter is buggy and refuses to die when Git tells it to exit, or the\n> code in Git waiting for its death is somehow miscounting its\n> children, and thought it told to die one process but in fact it\n> didn't (perhaps it told somebody else to die), or it thought it\n> hasn't seen the child die when in fact it already did.\n\nAgreed.\n\n\n> Calling kill(2) and exiting would hide these two kind of bugs from\n> end users. Not doing so would give the end users a hung Git, which\n> is a VERY GOOD thing. Otherwise you would not notice bugs and lose\n> the opportunity to diagnose and fix it.\n\nAha. I assumed that a hung Git because of a buggy filter would be a no-no.\nThanks for this clarification.\n\n\n> The timeout would be good for you to give a message \"filter process\n> running the script '%s' is not exiting; I am waiting for it\". The\n> user is still left with a hung Git, and can then see if that process\n> is hanging around. If it is, then we found a buggy filter. Or we\n> found a buggy Git. Either needs to be fixed. I do not think it\n> would help anybody by doing a kill(2) to sweep possible bugs under\n> the rug.\n\nI could achieve that with this run-command patch: \nhttp://public-inbox.org/git/E9946E9F-6EE5-492B-B122-9078CEB88044@gmail.com/\n(I'll remove the \"timeout after x seconds\" parts and keep the \"wait until \ndone\" part with stderr output)\n\n\nThanks,\nLars"},{"id":"303207","messageId":"20161004121111.gqilov3colsgzdft@sigill.intra.peff.net","threadId":"44122","inReplyTo":"xmqqvax974dl.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-10-04T12:11:11Z","receivedAt":"2016-10-04T12:11:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 03, 2016 at 10:02:14AM -0700, Junio C Hamano wrote:\n\n> The timeout would be good for you to give a message \"filter process\n> running the script '%s' is not exiting; I am waiting for it\". The\n> user is still left with a hung Git, and can then see if that process\n> is hanging around. If it is, then we found a buggy filter. Or we\n> found a buggy Git. Either needs to be fixed. I do not think it\n> would help anybody by doing a kill(2) to sweep possible bugs under\n> the rug.\n\nI would argue that we should not even bother with such a timeout. This\nis an exceptional, buggy condition, and hanging is not at all restricted\nto this particular case. If git is hanging, then the right tools are\n\"ps\" or \"strace\" to figure out what is going on. I know that not all\nusers are comfortable with those tools, but enough are in practice that\nthe bugs get ironed out, without git having to carry a bunch of extra\ntiming code that is essentially never exercised.\n\n-Peff\n"},{"id":"303256","messageId":"xmqq4m4svz7a.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"20161004121111.gqilov3colsgzdft@sigill.intra.peff.net","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-04T16:47:05Z","receivedAt":"2016-10-04T16:47:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Mon, Oct 03, 2016 at 10:02:14AM -0700, Junio C Hamano wrote:\n>\n>> The timeout would be good for you to give a message \"filter process\n>> running the script '%s' is not exiting; I am waiting for it\". The\n>> user is still left with a hung Git, and can then see if that process\n>> is hanging around. If it is, then we found a buggy filter. Or we\n>> found a buggy Git. Either needs to be fixed. I do not think it\n>> would help anybody by doing a kill(2) to sweep possible bugs under\n>> the rug.\n>\n> I would argue that we should not even bother with such a timeout. This\n> is an exceptional, buggy condition, and hanging is not at all restricted\n> to this particular case. If git is hanging, then the right tools are\n> \"ps\" or \"strace\" to figure out what is going on. I know that not all\n> users are comfortable with those tools, but enough are in practice that\n> the bugs get ironed out, without git having to carry a bunch of extra\n> timing code that is essentially never exercised.\n\nOK.\n"},{"id":"303285","messageId":"f7f9ca4c-229c-390a-beb0-a58e0d3d66b3@gmail.com","threadId":"44122","inReplyTo":"E9946E9F-6EE5-492B-B122-9078CEB88044@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-10-04T19:04:55Z","receivedAt":"2016-10-04T19:05:24Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 03.10.2016 o 19:13, Lars Schneider pisze: \n>> On 01 Oct 2016, at 22:48, Jakub Narębski wrote:\n>> W dniu 01.10.2016 o 20:59, Lars Schneider pisze: \n>>> On 29 Sep 2016, at 23:27, Junio C Hamano wrote:\n>>>> Lars Schneider writes:\n>>>>\n>>>> If the filter process refuses to die forever when Git told it to\n>>>> shutdown (by closing the pipe to it, for example), that filter\n>>>> process is simply buggy. I think we want users to become aware of\n>>>> that, instead of Git leaving it behind, which essentially is to\n>>>> sweep the problem under the rug.\n>>\n>> Well, it would be good to tell users _why_ Git is hanging, see below.\n> \n> Agreed. Do you think it is OK to write the message to stderr?\n\nOn the other hand, this is why GIT_TRACE (and GIT_TRACE_PERFORMANCE)\nwas invented for. We do not signal troubles with single-shot filters,\nso I guess doing it for multi-file filters is not needed.\n \n>>>> I agree with what Peff said elsewhere in the thread; if a filter\n>>>> process wants to take time to clean things up while letting Git\n>>>> proceed, it can do its own process management, but I think it is\n>>>> sensible for Git to wait the filter process it directly spawned.\n>>>\n>>> To realize the approach above I prototyped the run-command patch below:\n>>>\n>>> I added an \"exit_timeout\" variable to the \"child_process\" struct.\n>>> On exit, Git will close the pipe to the process and wait \"exit_timeout\" \n>>> seconds until it kills the child process. If \"exit_timeout\" is negative\n>>> then Git will wait until the process is done.\n>>\n>> That might be good approach. Probably the default would be to wait.\n> \n> I think I would prefer a 2sec timeout or something as default. This way\n> we can ensure Git would not wait indefinitely for a buggy filter by default.\n\nActually this waiting for multi-file filter is only about waiting for\nthe shutdown process of the filter. The filter could still hang during\nprocessing a file, and git would hang too, if I understand it correctly.\n\n[...]\n>> Also, how would one set default value of timeout for all process\n>> based filters?\n> \n> I think we don't need that because a timeout is always specific\n> to a filter (if the 2sec default is not sufficient).\n\nAll right (assuming that timeouts are good idea). \n\n>>>\n>>> +\t\t\twhile ((waitpid(p->pid, &status, 0)) < 0 && errno == EINTR)\n>>> +\t\t\t\t;\t/* nothing */\n>>\n>> Ah, this loop is here because waiting on waitpid() can be interrupted\n>> by the delivery of a signal to the calling process; though the result\n>> is -1, not just any < 0.\n> \n> \"< 0\" is also used in wait_or_whine()\n\nO.K. (though it doesn't necessary mean that it is correct, there\nis another point for using \"< 0\").\n \n[...]\n>> There is also another complication: there can be more than one\n>> long-running filter driver used. With this implementation we\n>> wait for each of one in sequence, e.g. 10s + 10s + 10s.\n> \n> Good idea, I fixed that in the version below!\n> \n[...]\n> [...] this function is also used with the async struct... \n\nHmmm... now I wonder if it is a good idea (similar treatment for\nsingle-file async-invoked filter, and multi-file pkt-line filters).\n\nFor single-file one-shot filter (correct me if I am wrong):\n\n - git sends contents to filter, signals end with EOF\n (after process is started)\n - in an async process:\n - process is started\n - git reads contents from filter, until EOF\n - if process did not end, it is killed\n\n\nFor multi-process pkt-line based filter (simplified):\n\n - process is started\n - handshake\n - for each file\n - file is send to filter process over pkt-line,\n end signalled with flush packet\n - git reads from filter from pkt-line, until flush\n - ...\n\n\nSee how single-shot filter is sent EOF, though in different part\nof code. We need to signal multi-file filter that no more files\nwill be coming. Simplest solution is to send EOF (we could send\n\"command=shutdown\" for example...) to filter, and wait for EOF\nfrom filter (or for \"status=finished\" and EOF).\n\nWe could kill multi-file filter after sending last file and\nreceiving full response... but I think single-shot filter gets\nkilled only because it allows for very simple filters, and reusing\nexisting commands as filters.\n\n[...]\n> diff --git a/run-command.c b/run-command.c\n> index 3269362..ca0feef 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -21,6 +21,9 @@ void child_process_clear(struct child_process *child)\n> \n> struct child_to_clean {\n> \tpid_t pid;\n> +\tchar *name;\n\nI guess it is here for output purposes?\n\nShould we store full command here, or just name of ?\n\n> +\tint stdin;\n\nI guess the name `stdin` for file _descriptor_ is something\nused in other parts of convert.c code, isn't it?\n\n> +\tint timeout;\n\nHmmm... we assume that timeout is in seconds, not millis or other\nvalue, isn't it. timeout_sec would perhaps be unnecessarily long.\n\n> \tstruct child_to_clean *next;\n> };\n> static struct child_to_clean *children_to_clean;\n> @@ -28,12 +31,53 @@ static int installed_child_cleanup_handler;\n> \n> static void cleanup_children(int sig, int in_signal)\n> {\n> +\tint status;\n> +\tstruct timeval tv;\n> +\ttime_t secs;\n> +\tstruct child_to_clean *p = children_to_clean;\n> +\n> +\t// Send EOF to children as indicator that Git will exit soon\n> +\twhile (p) {\n> +\t\tif (p->timeout != 0) {\n\nHere we use timeout == 0 as a special case, a special indicator\n(IIUC for the single-shot filter case, where it is closed already).\nThis is not documented. Somebody setting timeout to \"0\" would\nbe surprised, isn't it?\n\n> +\t\t\tif (p->stdin > 0)\n> +\t\t\t\tclose(p->stdin);\n> +\t\t}\n> +\t\tp = p->next;\n> +\t}\n> +\n> \twhile (children_to_clean) {\n> -\t\tstruct child_to_clean *p = children_to_clean;\n> +\t\tp = children_to_clean;\n> \t\tchildren_to_clean = p->next;\n> +\n> +\t\tif (p->timeout != 0) {\n> +\t\t\tfprintf(stderr, _(\"Waiting for '%s' to finish...\"), p->name);\n> +\t\t\tif (p->timeout < 0) {\n> +\t\t\t\t// No timeout given - wait indefinitely\n> +\t\t\t\twhile ((waitpid(p->pid, &status, 0)) < 0 && errno == EINTR)\n> +\t\t\t\t\t;\t/* nothing */\n> +\t\t\t} else {\n> +\t\t\t\t// Wait until timeout\n> +\t\t\t\tgettimeofday(&tv, NULL);\n> +\t\t\t\tsecs = tv.tv_sec;\n> +\t\t\t\twhile (!waitpid(p->pid, &status, WNOHANG) &&\n> +\t\t\t\t\t tv.tv_sec - secs < p->timeout) {\n> +\t\t\t\t\tfprintf(stderr, _(\" \\rWaiting %lds for '%s' to finish...\"),\n> +\t\t\t\t\t\tp->timeout - tv.tv_sec + secs - 1, p->name);\n> +\t\t\t\t\tgettimeofday(&tv, NULL);\n> +\t\t\t\t\tsleep_millisec(10);\n> +\t\t\t\t}\n> +\t\t\t}\n\nI wonder if we have some progress-printing code we can borrow\nfrom, or just plain use (like progress report for long checkout).\n\n> +\t\t\tif (waitpid(p->pid, &status, WNOHANG))\n> +\t\t\t\tfprintf(stderr, _(\"done!\\n\"));\n> +\t\t\telse\n> +\t\t\t\tfprintf(stderr, _(\"timeout. Killing...\\n\"));\n> +\t\t}\n> +\n> \t\tkill(p->pid, sig);\n> -\t\tif (!in_signal)\n> +\t\tif (!in_signal) {\n> +\t\t\tfree(p->name);\n> \t\t\tfree(p);\n> +\t\t}\n> \t}\n> }\n> \n> @@ -49,10 +93,18 @@ static void cleanup_children_on_exit(void)\n> \tcleanup_children(SIGTERM, 0);\n> }\n> \n> -static void mark_child_for_cleanup(pid_t pid)\n> +static void mark_child_for_cleanup_with_timeout(pid_t pid, const char *name, int stdin, int timeout)\n> {\n> \tstruct child_to_clean *p = xmalloc(sizeof(*p));\n> \tp->pid = pid;\n> +\tp->timeout = timeout;\n> +\tp->stdin = stdin;\n> +\tif (name) {\n> +\t\tp->name = xmalloc(strlen(name) + 1);\n> +\t\tstrcpy(p->name, name);\n\nDon't we have xstrdup() for that, or am I mistaken?\n\n> +\t} else {\n> +\t\tp->name = \"process\";\n\nHmmmm...\n\n> +\t}\n> \tp->next = children_to_clean;\n> \tchildren_to_clean = p;\n> \n> @@ -63,6 +115,13 @@ static void mark_child_for_cleanup(pid_t pid)\n> \t}\n> }\n> \n> +#ifdef NO_PTHREADS\n> +static void mark_child_for_cleanup(pid_t pid, const char *name, int timeout, int stdin)\n> +{\n> +\tmark_child_for_cleanup_with_timeout(pid, NULL, 0, 0);\n> +}\n> +#endif\n\nUh?\n\n> +\n> static void clear_child_for_cleanup(pid_t pid)\n> {\n> \tstruct child_to_clean **pp;\n> @@ -422,7 +481,8 @@ int start_command(struct child_process *cmd)\n> \tif (cmd->pid < 0)\n> \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n> \telse if (cmd->clean_on_exit)\n> -\t\tmark_child_for_cleanup(cmd->pid);\n> +\t\tmark_child_for_cleanup_with_timeout(\n> +\t\t\tcmd->pid, cmd->argv[0], cmd->in, cmd->clean_on_exit_timeout);\n\nAll right, nice abstraction.\n\n> \n> \t/*\n> \t * Wait for child's execvp. If the execvp succeeds (or if fork()\n> @@ -483,7 +543,8 @@ int start_command(struct child_process *cmd)\n> \tif (cmd->pid < 0 && (!cmd->silent_exec_failure || errno != ENOENT))\n> \t\terror_errno(\"cannot spawn %s\", cmd->argv[0]);\n> \tif (cmd->clean_on_exit && cmd->pid >= 0)\n> -\t\tmark_child_for_cleanup(cmd->pid);\n> +\t\tmark_child_for_cleanup_with_timeout(\n> +\t\t\tcmd->pid, cmd->argv[0], cmd->in, cmd->clean_on_exit_timeout);\n> \n> \targv_array_clear(&nargv);\n> \tcmd->argv = sargv;\n> diff --git a/run-command.h b/run-command.h\n> index cf29a31..4c1c1f4 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -43,6 +43,16 @@ struct child_process {\n> \tunsigned stdout_to_stderr:1;\n> \tunsigned use_shell:1;\n> \tunsigned clean_on_exit:1;\n> +\t/*\n> +\t * clean_on_exit_timeout is only considered if clean_on_exit is set.\n> +\t * - Specify 0 to kill the child on Git exit (default)\n> +\t * - Specify a negative value to close the child's stdin on Git exit\n> +\t * and wait indefinitely for the child's termination.\n> +\t * - Specify a positive value to close the child's stdin on Git exit\n> +\t * and wait clean_on_exit_timeout seconds for the child's\n> +\t * termination.\n\nAll right, so here is this documentation...\n\n> +\t */\n> +\tint clean_on_exit_timeout;\n> };\n> \n> #define CHILD_PROCESS_INIT { NULL, ARGV_ARRAY_INIT, ARGV_ARRAY_INIT }\n> \n> \n\nFor full patch, you would need also to add to Documentation/config.txt\n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"303300","messageId":"2c53b837-4690-dae8-60ff-e7907a9b13ef@gmail.com","threadId":"44122","inReplyTo":"4DE57A65-1020-4F17-81F2-9F319834BB2D@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-10-04T20:50:38Z","receivedAt":"2016-10-04T20:50:59Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[Some of answers may get invalidated by v9]\n\nW dniu 30.09.2016 o 20:56, Lars Schneider pisze:\n>> On 27 Sep 2016, at 00:41, Jakub Narębski wrote:\n>>\n>> Part first of the review of 11/11.\n\n[...]\n>>> +to read a welcome response message (\"git-filter-server\") and exactly\n>>> +one protocol version number from the previously sent list. All further\n>>\n>> I guess that is to provide forward-compatibility, isn't it? Also,\n>> \"Git expects...\" probably means filter process MUST send, in the\n>> RFC2119 (https://tools.ietf.org/html/rfc2119) meaning.\n> \n> True. I feel \"expects\" reads better but I am happy to change it if\n> you feel strong about it.\n\nI don't have strong opinion about this. I guess following the example\nof pkt-line format description may be a good idea. We are not writing\nan RFC here... but having be explicit is better than be good read :-P\n\n>>> +\n>>> +After the version negotiation Git sends a list of supported capabilities\n>>> +and a flush packet.\n>>\n>> Is it that Git SHOULD send list of ALL supported capabilities, or is\n>> it that Git SHOULD NOT send capabilities it does not support, and that\n>> it MAY send only those capabilities it needs (so for example if command\n>> uses only `smudge`, it may not send `clean`, so that filter driver doesn't\n>> need to initialize data it would not need).\n> \n> \"After the version negotiation Git sends a list of all capabilities that\n> it supports and a flush packet.\"\n> \n> Better?\n\nBetter.\n\nI wonder if it is a matter of current implementation, namely that at\nthe point of code where Git is sending capabilities it doesn't know\nwhich of them it will be using; at least in some of cases.\n\nBecause if it would be possible for Git to not send capabilities which\nit supports, but is sure that it would not be using for a given operation,\nthen it would be good to do that. It might be that filter driver must\ndo some prep work for each of capabilities, so skipping some of prep\nwould make git faster. Though all this is for now theoretical musings...\nit might be an argument for such description of protocol so it does\nnot prevent Git from sending only supported capabilities it needs.\n\n>> I wonder why it is \"=true\", and not \"capability=\".\n>> Is there a case where we would want to send \"=false\". Or\n>> is it to allow configurable / value based capabilities? Isn't it going\n>> a bit too far: is there even a hind of an idea for parametrize-able\n>> capability? YAGNI is a thing...\n> \n> Peff suggested that format and I think it is OK:\n> http://public-inbox.org/git/20160803224619.bwtbvmslhuicx2qi@sigill.intra.peff.net/\n\nIt wouldn't kill you to summarize the argument, would it?\n\nFrom what I understand the argument is that \"=true\" allows\nfor simplist parsing into a [intermediate] hash, while the other\nproposal of using\"capability=\" would require something more\nsophisticated. And that it is better to be explicit rather than\nuse synonyms / shortcuts for \"\".\n\nThough one can argue that \"\" is synonym / shortcut for \"=true\";\nit would not complicate parsing much.\n\nNb. we don't use trick of 'parse metadata to hash' in neither example,\nnor filter driver used in test...\n\n[...]\n>>> +\n>>> +After the filter has processed a blob it is expected to wait for\n>>> +the next \"key=value\" list containing a command. Git will close\n>>> +the command pipe on exit. The filter is expected to detect EOF\n>>> +and exit gracefully on its own.\n\nIs this still true?\n\n>>\n>> Good to have it documented. \n>>\n>> Anyway, as it is Git command that spawns the filter driver process,\n>> assuming that the filter process doesn't daemonize itself, wouldn't\n>> the operating system reap it after its parent process, that is the\n>> git command it invoked, dies? So detecting EOF is good, but not\n>> strictly necessary for simple filter that do not need to free\n>> its resources, or can leave freeing resources to the operating\n>> system? But I may be wrong here.\n> \n> The filter process runs independent of Git.\n\nAh. So without some way to tell long-lived filter process that\nit can shut down, because no further data will be incoming, or\nkilling it by Git, it would hang indefinitely?\n\n[...]\n>>> + }\n>>> + elsif ( $pkt_size > 4 ) {\n>>\n>> Isn't a packet of $pkt_size == 4 a valid packet, a keep-alive\n>> one? Or is it forbidden?\n> \n> \"Implementations SHOULD NOT send an empty pkt-line (\"0004\").\"\n> Source: Documentation/technical/protocol-common.txt\n\nAll right. Not that we need keep-alive packet for communication\nbetween two processes on the same host.\n\nThough I wonder why this rule is here...\n\n[...]\n>>> +( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n>>> +( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n>>> +( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n>>\n>> Actually, it is overly strict. It should not fail if there\n>> are other \"version=3\", \"version=4\" etc. lines.\n> \n> True, but I think for an example this is OK. I'll add a note\n> to the file header.\n\nAnyway it would be better to have helper subroutine to get metadata\nlines (packets till flush) and parse them into array or a hash...\n\n>>> +\n>>> +while (1) {\n>>> + my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n>>> + my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;\n>>\n>> Do we require this order? If it is, is that explained in the\n>> documentation?\n> \n> Git sends that order right now but the filter should not rely\n> on that order.\n\n...and the latter would make ignoring order of lines simpler.\n\nThough I wonder if we can ensure in the protocol documentation\nthat those lines would always be send in this order.\n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"303305","messageId":"d4ad73af-d14b-03dd-0634-d8919a1d2ddd@gmail.com","threadId":"44122","inReplyTo":"EF3723DE-B34A-4314-94C9-E3EB38EAB92A@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-10-04T21:00:04Z","receivedAt":"2016-10-04T21:00:26Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[Some of answers and comments may got invalidated by v9]\n\nW dniu 30.09.2016 o 21:38, Lars Schneider pisze:\n>> On 27 Sep 2016, at 17:37, Jakub Narębski wrote:\n>>\n>> Part second of the review of 11/11.\n[...]\n>>> +\n>>> +\tif (start_command(process)) {\n>>> +\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n>>> +\t\tkill_multi_file_filter(hashmap, entry);\n>>> +\t\treturn NULL;\n>>> +\t}\n>>\n>> I guess there is a reason why we init hashmap entry, try to start\n>> external process, then kill entry of unable to start, instead of\n>> trying to start external process, and adding hashmap entry when\n>> we succeed?\n> \n> Yes. This way I can reuse the kill_multi_file_filter() function.\n\nI don't quite understand. If you didn't fill the entry before\nusing start_command(process), you would not need kill_multi_file_filter(),\nwhich in that case IIUC just removes the just created entry from hashmap.\nCouldn't you add entry to hashmap in the 'else' part? Or would it\nbe racy?\n\n[...]\n>>> +static void read_multi_file_filter_values(int fd, struct strbuf *status) {\n>>\n>> This is more\n>>\n>> +static void read_multi_file_filter_status(int fd, struct strbuf *status) {\n>>\n>> It doesn't read arbitrary values, it examines 'metadata' from\n>> filter for \"status=\" lines.\n> \n> True!\n>\n>>> +\t\tif (pair[0] && pair[0]->len && pair[1]) {\n>>> +\t\t\tif (!strcmp(pair[0]->buf, \"status=\")) {\n>>> +\t\t\t\tstrbuf_reset(status);\n>>> +\t\t\t\tstrbuf_addbuf(status, pair[1]);\n>>> +\t\t\t}\n>>\n>> So it is last status= line wins behavior?\n> \n> Correct.\n\nPerhaps this should be described in code comment.\n \n>>>\n>>> +\tfflush(NULL);\n>>\n>> Why this fflush(NULL) is needed here?\n> \n> This flushes all open output streams. The single filter does the same.\n\nI know what it does, but I don't know why. But \"single filter does it\"\nis good enough for me. Still would want to know why, though ;-)\n \n>>>\n>>> +\tif (fd >= 0 && !src) {\n>>> +\t\tif (fstat(fd, &file_stat) == -1)\n>>> +\t\t\treturn 0;\n>>> +\t\tlen = xsize_t(file_stat.st_size);\n>>> +\t}\n>>\n>> Errr... is it necessary? The protocol no longer provides size=\n>> hint, and neither uses such hint if provided.\n> \n> We require the size in write_packetized_from_buf() later.\n\nDon't we use write_packetized_from_fd() in the case of fd >= 0?\n\n[...]\n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"303312","messageId":"5ffb5c17-01b5-a429-85e7-a15be713216b@gmail.com","threadId":"44122","inReplyTo":"7C680AE2-7F5A-47E4-8E79-6C3F4AEBD39B@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-10-04T21:34:00Z","receivedAt":"2016-10-04T21:34:21Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[Some of answers and comments may got invalidated by v9]\n\nW dniu 01.10.2016 o 17:34, Lars Schneider pisze:\n>> On 29 Sep 2016, at 01:14, Jakub Narębski wrote:\n>>\n>> Part third (and last) of the review of v8 11/11.\n\n[...]\n>>> +\n>>> +test_expect_success PERL 'required process filter should filter data' '\n>>> +\ttest_config_global filter.protocol.process \"$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge\" &&\n>>> +\ttest_config_global filter.protocol.required true &&\n>>> +\trm -rf repo &&\n>>> +\tmkdir repo &&\n>>> +\t(\n>>> +\t\tcd repo &&\n>>> +\t\tgit init &&\n>>\n>> Don't you think that creating a fresh test repository for each\n>> separate test is a bit too much? I guess that you want for\n>> each and every test to be completely independent, but this setup\n>> and teardown is a bit excessive.\n>>\n>> Other tests in the same file (should we reuse the test, or use\n>> new test file) do not use this method.\n> \n> I see your point. However, I am always annoyed if Git tests are\n> entangled because it makes working with them way way harder.\n> This test test runs in 4.5s on a slow Travis CI machine. I think\n> that is OK considering that we have tests running 3.5min (t3404).\n\nAll right, this is good argument... though inconsistency (assuming\nthat we don't switch to separate test for multi-file filter) could\nbe argument against.\n\nThough I wonder if test preparation could be extracted into a\ncommon function...\n\n[...]\n>>> +\t\tcp ../test.o test.r &&\n>>> +\t\tcp ../test2.o test2.r &&\n>>\n>> What does this test2.o / test2.r file tests, that test.o / test.r\n>> doesn't? The name doesn't tell us.\n> \n> This just tests multiple files with different content.\n\nAll right, so it is here to test multiple files (and that filter\nactually process multiple files).\n\n>> Why it is test.r, but test2.r? Why it isn't test1.r?\n> \n> test.r already existed (created in setup test).\n\nWith each test in separate repository we could copy test.r prepared\nin 'setup' into test1.r in each of multi-file tests.\n\n[...]\n>>> +\t\t>test4-empty.r &&\n>>\n>> You test ordinary file, file in subdirectory, file with filename\n>> containing spaces, and an empty file.\n>>\n>> Other tests of single file `clean`/`smudge` filters use filename\n>> that requires mangling; maybe we should use similar file?\n>>\n>> special=\"name with '\\''sq'\\'' and \\$x\" &&\n>> echo some test text >\"$special\" &&\n> \n> OK.\n\nThis test was required because of %f to pass filename as parameter\ncoupled with the fact that we use `clean` and `smudge` as shell\nscript fragment (so e.g. pipes and redirection would work in\none-shot filter definition).\n\nThis is not the case with multi-file filter, where filenames are\npassed internally, and we don't need to worry about shell quoting\nat all.\n \n>> In case of `process` filter, a special filename could look like\n>> this:\n>>\n>> process_special=\"name=with equals and\\nembedded newlines\\n\" &&\n>> echo some test text >\"$process_special\" &&\n> \n> I think this test would create trouble on Windows. I'll stick to\n> the special characters used in the single shot filter.\n\nThis would test... example parser. Well, all right, better not\ngive problems for Windows.\n\nBut... you can create such files in Git Bash:\n\n $ touch \"$(echo -n -e \"fo=o\\nbar\\n\")\"\n\nand though they look strange\n\n $ ls -1 fo*\n 'fo=o'$'\\n''bar'\n\nbut work all right\n\n $ echo \"strangest\" >>\"$(echo -n -e \"fo=o\\nbar\\n\")\"\n $ name=\"$(echo -n -e \"fo=o\\nbar\\n\")\"\n $ cat \"$name\"\n strangest\n\n>> Third, why the filter even writes output size? It is no longer\n>> part of `process` filter driver protocol, and it makes test more\n>> fragile.\n> \n> I would prefer to leave that in. I think it is good for the test to\n> check that we are transmitting the amount of content that what we \n> think we transmit.\n\nRight, we test that we processed full file this way, in the multi\npacket test. \n\n>>> +\t\t\t\t<<-\\EOF &&\n>>> +\t\t\t\t\tSTART\n>>> +\t\t\t\t\twrote filter header\n>>> +\t\t\t\t\tSTOP\n>>> +\t\t\t\tEOF\n>>\n>> Why is even filter process invoked? If this is not expected, perhaps\n>> simply ignore what checking out almost empty branch (one without any\n>> files marked for filtering) does.\n>>\n>> Shouldn't we test_expect_failure no-call?\n> \n> Because a clean operation could happen. I added a clean operation to\n> the expected log in order to make this visible (expected log is stripped\n> of clean operations in the same way as the actual log per your suggestion\n> above).\n\nIf we are testing that if there is no \"smudge\" capability, then\nthere were no \"smudge\" operations, why we don't test just that:\nthat grepping for \"smudge\" in long doesn't find anything.\n\nCurrent version feels convoluted (and would stop working if Git\nis improved to not run \"clean\" in this case for optimization).\n \n>>> +\n>>> +\t\tcheck_filter_ignore_clean \\\n>>> +\t\t\tgit checkout master \\\n>>\n>> Does this checks different code path than 'git checkout .'? For\n>> example, does this test increase code coverage (e.g. as measured\n>> by gcov)? If not, then this test could be safely dropped.\n> \n> We checked out the \"empty-branch\" before. That's why we check here\n> that the smudge filter runs for all files (smudge filter did not run\n> for all files with `git checkout .`).\n\nAll right, it runs for more files, but does it cover different\ncode paths? If not, it only makes test run longer...\n\n>>> +\t\t\t\t<<-\\EOF &&\n>>> +\t\t\t\t\tSTART\n>>> +\t\t\t\t\twrote filter header\n>>> +\t\t\t\t\tIN: smudge test.r 57 [OK] -- OUT: 57 . [OK]\n>>> +\t\t\t\t\tIN: smudge test2.r 14 [OK] -- OUT: 14 . [OK]\n>>> +\t\t\t\t\tIN: smudge test4-empty.r 0 [OK] -- OUT: 0 [OK]\n>>> +\t\t\t\t\tIN: smudge testsubdir/test3 - subdir.r 23 [OK] -- OUT: 23 . [OK]\n>>\n>> Can we assume that Git would pass files to filter in alphabetical\n>> order? This assumption might make the test unnecessary fragile.\n> \n> I have never experienced another behavior. If we see fragility we could\n> sort the result...\n \nAll right (perhaps comment for future would be good idea, though).\n \n>>>\n>>> +test_expect_success PERL 'required process filter should clean only and take precedence' '\n>>\n>> Trying to describe it better results in overly long description,\n>> which probably means that this test should be split into few\n>> smaller ones:\n>>\n>> - `process` filter takes precedence over `clean` and/or `smudge`\n>> filters, regardless if it supports relevant (\"clean\" or \"smudge\")\n>> capability or not\n>>\n>> - `process` filter that includes only \"clean\" capability should\n>> clean only (be used only for 'clean' operation)\n> \n> Agreed!\n> \n> \n>> In my opinion all functions should be placed at beginning,\n>> or even in separate file (if they are used in more than\n>> one test).\n> \n> OK\n> \n> \n>>> +generate_test_data () {\n>>\n>> The name is not good, it doesn't describe what kind of data\n>> we want to generate.\n> \n> \"generate_random_characters\" ok?!\n\nAll right.\n\n[...]\n>>> +\t\techo \"*.file filter=protocol\" >.gitattributes &&\n>>> +\t\tcheck_filter \\\n>>> +\t\t\tgit add *.file .gitattributes \\\n>>\n>> Should it be shell expansion, or git expansion, that is\n>>\n>> \t\t\tgit add '*.file' .gitattributes\n> \n> Both have the same output. Would the difference matter?\n\nIn one case *.file is expanded by shell, then expansion passed\nas parameters to `git add` (perhaps not on MS Windows); in the\nother '*.file' is passed as pattern to `git add` and expanded\nby Git itself (this might be case for both patterns on Win).\n\nBut this doesn't matter here, anyway. I think.\n\n[...]\n>>> +\n>>> +test_expect_success PERL 'process filter should not restart in case of an error' '\n>>\n>> Errr... what? This description is not clear. Did you mean\n>> that filter should not be restarted if it *signals* an error\n>> with file (either before sending anything, or after sending\n>> partial contents)?\n> \n> OK renamed to \"process filter should not be restarted if it signals an error\"\n\nThis is better.\n \n>>> +test_expect_success PERL 'process filter should be able to signal an error for all future files' '\n>>\n>> Did you mean here that filter can abort processing of\n>> all future files?\n> \n> \"process filter signals abort once to abort processing of all future files\", better?\n\n\"process filter aborting stops processing of all further files\", maybe?\n\n>>> +\n>>> +\t\tcp ../test.o test.r &&\n>>> +\t\ttest_must_fail git add . 2> git_stderr.log &&\n>>> +\t\tgrep \"not support long running filter protocol\" git_stderr.log\n>>\n>> Shouldn't this use gettext poison (or rather C locale)?\n>> This error message could be translated in the future.\n> \n> I would prefer to adjust that when we translate it.\n\nAll right, good enough.\n\n>>> + $str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n>>\n>> Why not use tr/// version of this quote-like operation?\n>> Or do you follow prior art here?\n> \n> I am not Perl expert. That worked for me :-)\n\ny/// and tr/// are the same operator. Though y/// is supposedly\nmore Perl-ish, and I think it is used more in Git tests (or rather\nit functions).\n\n[...]\n>>> +packet_flush();\n>>> +print $debug \"wrote filter header\\n\";\n>>\n>> Or perhaps \"handshake end\"?\n> \n> \"init handshake complete\", ok?\n\nBetter.\n \n>>> + print $debug \" $pathname\";\n>>\n>> No \" pathname=$pathname\" ?\n> \n> Yes, otherwise it gets too verbose in the tests.\n\nAll right. And the lines gets too long.\n\n[...]\n\nRegards,\n-- \nJakub Narębski\n\n"},{"id":"303483","messageId":"5C30800A-C821-4B39-BEB7-7D55D86BD7EA@gmail.com","threadId":"44122","inReplyTo":"f7f9ca4c-229c-390a-beb0-a58e0d3d66b3@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-10-06T13:13:19Z","receivedAt":"2016-10-06T13:13:28Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 04 Oct 2016, at 21:04, Jakub Narębski wrote:\n> \n> W dniu 03.10.2016 o 19:13, Lars Schneider pisze: \n>>> On 01 Oct 2016, at 22:48, Jakub Narębski wrote:\n>>> W dniu 01.10.2016 o 20:59, Lars Schneider pisze: \n>>>> On 29 Sep 2016, at 23:27, Junio C Hamano wrote:\n>>>>> Lars Schneider writes:\n>>>>> \n>>>>> If the filter process refuses to die forever when Git told it to\n>>>>> shutdown (by closing the pipe to it, for example), that filter\n>>>>> process is simply buggy. I think we want users to become aware of\n>>>>> that, instead of Git leaving it behind, which essentially is to\n>>>>> sweep the problem under the rug.\n>>> \n>>> Well, it would be good to tell users _why_ Git is hanging, see below.\n>> \n>> Agreed. Do you think it is OK to write the message to stderr?\n> \n> On the other hand, this is why GIT_TRACE (and GIT_TRACE_PERFORMANCE)\n> was invented for. We do not signal troubles with single-shot filters,\n> so I guess doing it for multi-file filters is not needed.\n\nI am on the fence with this one.\n\n@Junio/Peff:\nWhere would you prefer to see a \"Waiting for filter 'XYZ'... \" message?\nOn stderr or via GIT_TRACE?\n\n\n> \n>>>>> I agree with what Peff said elsewhere in the thread; if a filter\n>>>>> process wants to take time to clean things up while letting Git\n>>>>> proceed, it can do its own process management, but I think it is\n>>>>> sensible for Git to wait the filter process it directly spawned.\n>>>> \n>>>> To realize the approach above I prototyped the run-command patch below:\n>>>> \n>>>> I added an \"exit_timeout\" variable to the \"child_process\" struct.\n>>>> On exit, Git will close the pipe to the process and wait \"exit_timeout\" \n>>>> seconds until it kills the child process. If \"exit_timeout\" is negative\n>>>> then Git will wait until the process is done.\n>>> \n>>> That might be good approach. Probably the default would be to wait.\n>> \n>> I think I would prefer a 2sec timeout or something as default. This way\n>> we can ensure Git would not wait indefinitely for a buggy filter by default.\n> \n> Actually this waiting for multi-file filter is only about waiting for\n> the shutdown process of the filter. The filter could still hang during\n> processing a file, and git would hang too, if I understand it correctly.\n\nCorrect.\n\n\n>> [...] this function is also used with the async struct... \n> \n> Hmmm... now I wonder if it is a good idea (similar treatment for\n> single-file async-invoked filter, and multi-file pkt-line filters).\n> \n> For single-file one-shot filter (correct me if I am wrong):\n> \n> - git sends contents to filter, signals end with EOF\n> (after process is started)\n> - in an async process:\n> - process is started\n> - git reads contents from filter, until EOF\n> - if process did not end, it is killed\n> \n> \n> For multi-process pkt-line based filter (simplified):\n> \n> - process is started\n> - handshake\n> - for each file\n> - file is send to filter process over pkt-line,\n> end signalled with flush packet\n> - git reads from filter from pkt-line, until flush\n> - ...\n> \n> \n> See how single-shot filter is sent EOF, though in different part\n> of code. We need to signal multi-file filter that no more files\n> will be coming. Simplest solution is to send EOF (we could send\n> \"command=shutdown\" for example...) to filter, and wait for EOF\n> from filter (or for \"status=finished\" and EOF).\n\nThat's what we do. EOF does signal the multi-filter to shutdown.\n\n\n> For full patch, you would need also to add to Documentation/config.txt\n\nWhy config.txt?\n\n\nThanks,\nLars"},{"id":"303484","messageId":"52356944-49B6-482A-8951-2B7F8D8032F7@gmail.com","threadId":"44122","inReplyTo":"2c53b837-4690-dae8-60ff-e7907a9b13ef@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-10-06T13:16:50Z","receivedAt":"2016-10-06T13:17:04Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 04 Oct 2016, at 22:50, Jakub Narębski wrote:\n> \n> [Some of answers may get invalidated by v9]\n> \n> W dniu 30.09.2016 o 20:56, Lars Schneider pisze:\n>>> On 27 Sep 2016, at 00:41, Jakub Narębski wrote:\n>>> \n>>>> +\n>>>> +After the filter has processed a blob it is expected to wait for\n>>>> +the next \"key=value\" list containing a command. Git will close\n>>>> +the command pipe on exit. The filter is expected to detect EOF\n>>>> +and exit gracefully on its own.\n> \n> Is this still true?\n\nYes\n\n\n>>> \n>>> Good to have it documented. \n>>> \n>>> Anyway, as it is Git command that spawns the filter driver process,\n>>> assuming that the filter process doesn't daemonize itself, wouldn't\n>>> the operating system reap it after its parent process, that is the\n>>> git command it invoked, dies? So detecting EOF is good, but not\n>>> strictly necessary for simple filter that do not need to free\n>>> its resources, or can leave freeing resources to the operating\n>>> system? But I may be wrong here.\n>> \n>> The filter process runs independent of Git.\n> \n> Ah. So without some way to tell long-lived filter process that\n> it can shut down, because no further data will be incoming, or\n> killing it by Git, it would hang indefinitely?\n\nYes\n\n- Lars"},{"id":"303493","messageId":"20161006160104.nhpxhfhketr4rb4h@sigill.intra.peff.net","threadId":"44122","inReplyTo":"5C30800A-C821-4B39-BEB7-7D55D86BD7EA@gmail.com","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-10-06T16:01:04Z","receivedAt":"2016-10-06T16:08:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 06, 2016 at 03:13:19PM +0200, Lars Schneider wrote:\n\n> >>> Well, it would be good to tell users _why_ Git is hanging, see below.\n> >> \n> >> Agreed. Do you think it is OK to write the message to stderr?\n> > \n> > On the other hand, this is why GIT_TRACE (and GIT_TRACE_PERFORMANCE)\n> > was invented for. We do not signal troubles with single-shot filters,\n> > so I guess doing it for multi-file filters is not needed.\n> \n> I am on the fence with this one.\n> \n> @Junio/Peff:\n> Where would you prefer to see a \"Waiting for filter 'XYZ'... \" message?\n> On stderr or via GIT_TRACE?\n\nI am not sure if I have followed all of this discussion, but I am of the\nopinion that Git should not be doing any timeouts at all. There are\nsimply too many places where the filter (or any other process that git\nis depending on) could inexplicably hang, and I\ndo not want to pepper random timeouts for all parts of the procedure\nwhere we say \"woah, this is taking longer than expected\" (nor do I want\nto have a timeout for _one_ spot, and ignore all the others).\n\nIf this is debugging output of the form \"now I am calling wait() on all\nof the filters\", without respect to any timeout, that sounds reasonable.\nThough I would argue that run-command should simply trace_printf()\nany processes it is waiting for, which covers _any_ process, not just\nthe filters. That seems like a good match for the rest of the GIT_TRACE\noutput, which describes how and when we spawn the sub-processes.\n\nSomething like:\n\ndiff --git a/run-command.c b/run-command.c\nindex 5a4dbb6..b884605 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -226,6 +226,9 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \tpid_t waiting;\n \tint failed_errno = 0;\n \n+\tif (!in_signal)\n+\t\ttrace_printf(\"waiting for pid %d\", (int)pid);\n+\n \twhile ((waiting = waitpid(pid, &status, 0)) < 0 && errno == EINTR)\n \t\t;\t/* nothing */\n \tif (in_signal)\n\nbut it does not have to be part of this series, I think.\n\n-Peff\n"},{"id":"303503","messageId":"xmqq37k9mm7k.fsf@gitster.mtv.corp.google.com","threadId":"44122","inReplyTo":"20161006160104.nhpxhfhketr4rb4h@sigill.intra.peff.net","subject":"Re: [PATCH v8 00/11] Git filter protocol","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-06T17:17:03Z","receivedAt":"2016-10-06T17:17:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I am not sure if I have followed all of this discussion, but I am of the\n> opinion that Git should not be doing any timeouts at all.\n> ...\n> If this is debugging output of the form \"now I am calling wait() on all\n> of the filters\", without respect to any timeout, that sounds reasonable.\n> Though I would argue that run-command should simply trace_printf()\n> any processes it is waiting for, which covers _any_ process, not just\n> the filters. That seems like a good match for the rest of the GIT_TRACE\n> output, which describes how and when we spawn the sub-processes.\n\nYup, I agree that having trace_printf() report the wait for any\nprocess is the cleanest way to go. As you guessed the reason why\nLars is bringing up \"now we are waiting for this filter\" is because\nI suggested it as a way to encourage users to file bugs when they\nsee a hung Git. Originally Lars wanted to have a timeout on wait\nand after the timeout wanted to kill the process, and because I\nreally did not want such a random \"you are too slow to die, so I'll\nsend a signal to you and exit myself without making sure you died\"\nthere, I suggested that if we were to have a timeout, that would be\nto timeout the wait only to have a chance to tell the user \"we are\nstuck waiting on this thing\" (and then go back to wait), as it would\neither be a buggy filter (i.e. the users need to debug their own\nfilter code) or a buggy use of wait on Git side (i.e. we would want\nto hear about such bugs from them).\n\nWithout such a \"wait with timeout so that we can tell the user\", we\ncan still respond to \"my 'git checkout' hangs forever\" with \"try\nrunning with GIT_TRACE\" as you outlined above, so I do not think we\nneed the timeout.\n\nThanks for straightening us out.\n\n\n"},{"id":"303533","messageId":"C2A14809-8AC0-456A-80C5-7B7F426A4026@gmail.com","threadId":"44122","inReplyTo":"d4ad73af-d14b-03dd-0634-d8919a1d2ddd@gmail.com","subject":"Re: [PATCH v8 11/11] convert: add filter..process option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-10-06T21:27:37Z","receivedAt":"2016-10-06T21:27:45Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 04 Oct 2016, at 23:00, Jakub Narębski wrote:\n> \n> [Some of answers and comments may got invalidated by v9]\n> \n> W dniu 30.09.2016 o 21:38, Lars Schneider pisze:\n>>> On 27 Sep 2016, at 17:37, Jakub Narębski wrote:\n>>> \n>>> Part second of the review of 11/11.\n> [...]\n>>>> +\n>>>> +\tif (start_command(process)) {\n>>>> +\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n>>>> +\t\tkill_multi_file_filter(hashmap, entry);\n>>>> +\t\treturn NULL;\n>>>> +\t}\n>>> \n>>> I guess there is a reason why we init hashmap entry, try to start\n>>> external process, then kill entry of unable to start, instead of\n>>> trying to start external process, and adding hashmap entry when\n>>> we succeed?\n>> \n>> Yes. This way I can reuse the kill_multi_file_filter() function.\n> \n> I don't quite understand. If you didn't fill the entry before\n> using start_command(process), you would not need kill_multi_file_filter(),\n> which in that case IIUC just removes the just created entry from hashmap.\n> Couldn't you add entry to hashmap in the 'else' part? Or would it\n> be racy?\n\nYou are right. I'll fix that.\n\n\n>> \n>>>> +\t\tif (pair[0] && pair[0]->len && pair[1]) {\n>>>> +\t\t\tif (!strcmp(pair[0]->buf, \"status=\")) {\n>>>> +\t\t\t\tstrbuf_reset(status);\n>>>> +\t\t\t\tstrbuf_addbuf(status, pair[1]);\n>>>> +\t\t\t}\n>>> \n>>> So it is last status= line wins behavior?\n>> \n>> Correct.\n> \n> Perhaps this should be described in code comment.\n\nOK\n\n\n>>>> \n>>>> +\tfflush(NULL);\n>>> \n>>> Why this fflush(NULL) is needed here?\n>> \n>> This flushes all open output streams. The single filter does the same.\n> \n> I know what it does, but I don't know why. But \"single filter does it\"\n> is good enough for me. Still would want to know why, though ;-)\n\nTBH I am not 100% sure why, too. I think this ensures that we don't have \nany outdated/unrelated/previous data in the stream buffers.\n\n\n>>>> \n>>>> +\tif (fd >= 0 && !src) {\n>>>> +\t\tif (fstat(fd, &file_stat) == -1)\n>>>> +\t\t\treturn 0;\n>>>> +\t\tlen = xsize_t(file_stat.st_size);\n>>>> +\t}\n>>> \n>>> Errr... is it necessary? The protocol no longer provides size=\n>>> hint, and neither uses such hint if provided.\n>> \n>> We require the size in write_packetized_from_buf() later.\n> \n> Don't we use write_packetized_from_fd() in the case of fd >= 0?\n\nOf course! Ah too many refactorings :-)\nI'll remove that.\n\nThank you,\nLars\n\n"}]}