{"thread":{"id":"58212","subject":"[PATCH 0/2] t0021: convert perl script to C test-tool helper","startedAt":"2022-07-22T19:43:15Z","lastAt":"2022-08-10T19:58:22Z","messageCount":28,"participants":["Matheus Tavares","Ævar Arnfjörð Bjarmason","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"459788","messageId":"cover.1658518769.git.matheus.bernardino@usp.br","threadId":"58212","inReplyTo":null,"subject":"[PATCH 0/2] t0021: convert perl script to C test-tool helper","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-22T19:42:48Z","receivedAt":"2022-07-22T19:43:15Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This addresses the \"left over bits\" comment from [1], converting the\nt0021/rot13-filter.pl script to a C test-tool helper in order to drop\nthe PERL dependency from tests using this script.\n\nThis series builds on top of mt/checkout-count-fix, also adjusting the\nscript invocations from that patchset.\n\n[1]: https://lore.kernel.org/git/xmqqfsj4dhfi.fsf@gitster.g/\n\nMatheus Tavares (2):\n  t/t0021: convert the rot13-filter.pl script to C\n  t/t0021: replace old rot13-filter.pl uses with new test-tool cmd\n\n Makefile                                |   1 +\n pkt-line.c                              |  13 +-\n pkt-line.h                              |   2 +\n t/helper/test-rot13-filter.c            | 396 ++++++++++++++++++++++++\n t/helper/test-tool.c                    |   1 +\n t/helper/test-tool.h                    |   1 +\n t/t0021-conversion.sh                   |  71 ++---\n t/t0021/rot13-filter.pl                 | 247 ---------------\n t/t2080-parallel-checkout-basics.sh     |   7 +-\n t/t2082-parallel-checkout-attributes.sh |   7 +-\n 10 files changed, 450 insertions(+), 296 deletions(-)\n create mode 100644 t/helper/test-rot13-filter.c\n delete mode 100644 t/t0021/rot13-filter.pl\n\n-- \n2.37.1\n\n"},{"id":"459789","messageId":"3b0240c1b92dfb441e9b307a7896145a713ee168.1658518769.git.matheus.bernardino@usp.br","threadId":"58212","inReplyTo":"cover.1658518769.git.matheus.bernardino@usp.br","subject":"[PATCH 2/2] t/t0021: replace old rot13-filter.pl uses with new test-tool cmd","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-22T19:42:50Z","receivedAt":"2022-07-22T19:43:16Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Complete the perl-to-C conversion from the previous commit by actually\nremoving the old perl script and adjusting the test cases to directly\ncall \"test-tool rot13-filter\".\n\nSigned-off-by: Matheus Tavares <matheus.bernardino@usp.br>\n---\n t/t0021-conversion.sh                   | 69 ++++++++++++-------------\n t/t0021/rot13-filter.pl                 |  5 --\n t/t2080-parallel-checkout-basics.sh     |  7 +--\n t/t2082-parallel-checkout-attributes.sh |  7 +--\n 4 files changed, 37 insertions(+), 51 deletions(-)\n delete mode 100644 t/t0021/rot13-filter.pl\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 963b66e08c..aeaa8e02ed 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -17,9 +17,6 @@ tr \\\n   'nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM'\n EOF\n \n-write_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl\n-\n generate_random_characters () {\n \tLEN=$1\n \tNAME=$2\n@@ -365,8 +362,8 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '\n \ttest_line_count = 0 count\n '\n \n-test_expect_success PERL 'required process filter should filter data' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should filter data' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -450,8 +447,8 @@ test_expect_success PERL 'required process filter should filter data' '\n \t)\n '\n \n-test_expect_success PERL 'required process filter should filter data for various subcommands' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should filter data for various subcommands' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \t(\n \t\tcd repo &&\n@@ -561,9 +558,9 @@ test_expect_success PERL 'required process filter should filter data for various\n \t)\n '\n \n-test_expect_success PERL 'required process filter takes precedence' '\n+test_expect_success 'required process filter takes precedence' '\n \ttest_config_global filter.protocol.clean false &&\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -587,8 +584,8 @@ test_expect_success PERL 'required process filter takes precedence' '\n \t)\n '\n \n-test_expect_success PERL 'required process filter should be used only for \"clean\" operation only' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n+test_expect_success 'required process filter should be used only for \"clean\" operation only' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -622,8 +619,8 @@ test_expect_success PERL 'required process filter should be used only for \"clean\n \t)\n '\n \n-test_expect_success PERL 'required process filter should process multiple packets' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should process multiple packets' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \n \trm -rf repo &&\n@@ -687,8 +684,8 @@ test_expect_success PERL 'required process filter should process multiple packet\n \t)\n '\n \n-test_expect_success PERL 'required process filter with clean error should fail' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter with clean error should fail' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -706,8 +703,8 @@ test_expect_success PERL 'required process filter with clean error should fail'\n \t)\n '\n \n-test_expect_success PERL 'process filter should restart after unexpected write failure' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter should restart after unexpected write failure' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -761,8 +758,8 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t)\n '\n \n-test_expect_success PERL 'process filter should not be restarted if it signals an error' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter should not be restarted if it signals an error' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -804,8 +801,8 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n \t)\n '\n \n-test_expect_success PERL 'process filter abort stops processing of all further files' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter abort stops processing of all further files' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -861,10 +858,10 @@ test_expect_success PERL 'invalid process filter must fail (and not hang!)' '\n \t)\n '\n \n-test_expect_success PERL 'delayed checkout in process filter' '\n-\ttest_config_global filter.a.process \"rot13-filter.pl a.log clean smudge delay\" &&\n+test_expect_success 'delayed checkout in process filter' '\n+\ttest_config_global filter.a.process \"test-tool rot13-filter a.log clean smudge delay\" &&\n \ttest_config_global filter.a.required true &&\n-\ttest_config_global filter.b.process \"rot13-filter.pl b.log clean smudge delay\" &&\n+\ttest_config_global filter.b.process \"test-tool rot13-filter b.log clean smudge delay\" &&\n \ttest_config_global filter.b.required true &&\n \n \trm -rf repo &&\n@@ -940,8 +937,8 @@ test_expect_success PERL 'delayed checkout in process filter' '\n \t)\n '\n \n-test_expect_success PERL 'missing file in delayed checkout' '\n-\ttest_config_global filter.bug.process \"rot13-filter.pl bug.log clean smudge delay\" &&\n+test_expect_success 'missing file in delayed checkout' '\n+\ttest_config_global filter.bug.process \"test-tool rot13-filter bug.log clean smudge delay\" &&\n \ttest_config_global filter.bug.required true &&\n \n \trm -rf repo &&\n@@ -960,8 +957,8 @@ test_expect_success PERL 'missing file in delayed checkout' '\n \tgrep \"error: .missing-delay\\.a. was not filtered properly\" git-stderr.log\n '\n \n-test_expect_success PERL 'invalid file in delayed checkout' '\n-\ttest_config_global filter.bug.process \"rot13-filter.pl bug.log clean smudge delay\" &&\n+test_expect_success 'invalid file in delayed checkout' '\n+\ttest_config_global filter.bug.process \"test-tool rot13-filter bug.log clean smudge delay\" &&\n \ttest_config_global filter.bug.required true &&\n \n \trm -rf repo &&\n@@ -990,10 +987,10 @@ do\n \t\tmode_prereq='UTF8_NFD_TO_NFC' ;;\n \tesac\n \n-\ttest_expect_success PERL,SYMLINKS,$mode_prereq \\\n+\ttest_expect_success SYMLINKS,$mode_prereq \\\n \t\"delayed checkout with $mode-collision don't write to the wrong place\" '\n \t\ttest_config_global filter.delay.process \\\n-\t\t\t\"\\\"$TEST_ROOT/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\t\t\"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \t\ttest_config_global filter.delay.required true &&\n \n \t\tgit init $mode-collision &&\n@@ -1026,12 +1023,12 @@ do\n \t'\n done\n \n-test_expect_success PERL,SYMLINKS,CASE_INSENSITIVE_FS \\\n+test_expect_success SYMLINKS,CASE_INSENSITIVE_FS \\\n \"delayed checkout with submodule collision don't write to the wrong place\" '\n \tgit init collision-with-submodule &&\n \t(\n \t\tcd collision-with-submodule &&\n-\t\tgit config filter.delay.process \"\\\"$TEST_ROOT/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\t# We need Git to treat the submodule \"a\" and the\n@@ -1062,11 +1059,11 @@ test_expect_success PERL,SYMLINKS,CASE_INSENSITIVE_FS \\\n \t)\n '\n \n-test_expect_success PERL 'setup for progress tests' '\n+test_expect_success 'setup for progress tests' '\n \tgit init progress &&\n \t(\n \t\tcd progress &&\n-\t\tgit config filter.delay.process \"rot13-filter.pl delay-progress.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter delay-progress.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\techo \"*.a filter=delay\" >.gitattributes &&\n@@ -1132,12 +1129,12 @@ do\n \t'\n done\n \n-test_expect_success PERL 'delayed checkout correctly reports the number of updated entries' '\n+test_expect_success 'delayed checkout correctly reports the number of updated entries' '\n \trm -rf repo &&\n \tgit init repo &&\n \t(\n \t\tcd repo &&\n-\t\tgit config filter.delay.process \"../rot13-filter.pl delayed.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter delayed.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\techo \"*.a filter=delay\" >.gitattributes &&\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\ndeleted file mode 100644\nindex 1447bc0a24..0000000000\n--- a/t/t0021/rot13-filter.pl\n+++ /dev/null\n@@ -1,5 +0,0 @@\n-use 5.008;\n-\n-my @quoted_args = map \"'$_'\", @ARGV;\n-exec \"test-tool rot13-filter @quoted_args\";\n-die \"failed to exec test-tool\";\ndiff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh\nindex c683e60007..7d956625ca 100755\n--- a/t/t2080-parallel-checkout-basics.sh\n+++ b/t/t2080-parallel-checkout-basics.sh\n@@ -230,12 +230,9 @@ test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading d\n # check the final report including sequential, parallel, and delayed entries\n # all at the same time. So we must have finer control of the parallel checkout\n # variables.\n-test_expect_success PERL '\"git checkout .\" report should not include failed entries' '\n-\twrite_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl &&\n-\n+test_expect_success '\"git checkout .\" report should not include failed entries' '\n \ttest_config_global filter.delay.process \\\n-\t\t\"\\\"$(pwd)/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\t\"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \ttest_config_global filter.delay.required true &&\n \ttest_config_global filter.cat.clean cat  &&\n \ttest_config_global filter.cat.smudge cat  &&\ndiff --git a/t/t2082-parallel-checkout-attributes.sh b/t/t2082-parallel-checkout-attributes.sh\nindex 2525457961..2df55b9405 100755\n--- a/t/t2082-parallel-checkout-attributes.sh\n+++ b/t/t2082-parallel-checkout-attributes.sh\n@@ -138,12 +138,9 @@ test_expect_success 'parallel-checkout and external filter' '\n # The delayed queue is independent from the parallel queue, and they should be\n # able to work together in the same checkout process.\n #\n-test_expect_success PERL 'parallel-checkout and delayed checkout' '\n-\twrite_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl &&\n-\n+test_expect_success 'parallel-checkout and delayed checkout' '\n \ttest_config_global filter.delay.process \\\n-\t\t\"\\\"$(pwd)/rot13-filter.pl\\\" --always-delay \\\"$(pwd)/delayed.log\\\" clean smudge delay\" &&\n+\t\t\"test-tool rot13-filter --always-delay \\\"$(pwd)/delayed.log\\\" clean smudge delay\" &&\n \ttest_config_global filter.delay.required true &&\n \n \techo \"abcd\" >original &&\n-- \n2.37.1\n\n"},{"id":"459790","messageId":"99823077be77bc621cfa8ccf3303bd612da343ad.1658518769.git.matheus.bernardino@usp.br","threadId":"58212","inReplyTo":"cover.1658518769.git.matheus.bernardino@usp.br","subject":"[PATCH 1/2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-22T19:42:49Z","receivedAt":"2022-07-22T19:43:17Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This script is currently used by three test files: t0021-conversion.sh,\nt2080-parallel-checkout-basics.sh, and\nt2082-parallel-checkout-attributes.sh. To avoid the need for the PERL\ndependency at these tests, let's convert the script to a C test-tool\ncommand. Note, however, that we still use the script as a wrapper at\nthis commit, in order to minimize the amount of changes it introduces\nand help reviewers. At the next commit we will properly remove the\nscript and adjust the affected tests to use test-tool.\n\nFurthermore, note that there is a small adjustment at test\nt0021-conversion.sh because it depended on a specific error message\ngiven by perl's die routine.\n\nSigned-off-by: Matheus Tavares <matheus.bernardino@usp.br>\n---\n Makefile                     |   1 +\n pkt-line.c                   |  13 +-\n pkt-line.h                   |   2 +\n t/helper/test-rot13-filter.c | 396 +++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c         |   1 +\n t/helper/test-tool.h         |   1 +\n t/t0021-conversion.sh        |   2 +-\n t/t0021/rot13-filter.pl      | 248 +---------------------\n 8 files changed, 416 insertions(+), 248 deletions(-)\n create mode 100644 t/helper/test-rot13-filter.c\n\ndiff --git a/Makefile b/Makefile\nindex 04d0fd1fe6..7cfcf3a911 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -764,6 +764,7 @@ TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n+TEST_BUILTINS_OBJS += test-rot13-filter.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n TEST_BUILTINS_OBJS += test-run-command.o\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 8e43c2def4..4425bdae36 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -309,9 +309,10 @@ int write_packetized_from_fd_no_flush(int fd_in, int fd_out)\n \treturn err;\n }\n \n-int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n+int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n+\t\t\t\t\t     int fd_out, int *count_ptr)\n {\n-\tint err = 0;\n+\tint err = 0, count = 0;\n \tsize_t bytes_written = 0;\n \tsize_t bytes_to_write;\n \n@@ -324,10 +325,18 @@ int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_ou\n \t\t\tbreak;\n \t\terr = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);\n \t\tbytes_written += bytes_to_write;\n+\t\tcount++;\n \t}\n+\tif (count_ptr)\n+\t\t*count_ptr = count;\n \treturn err;\n }\n \n+int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n+{\n+\treturn write_packetized_from_buf_no_flush_count(src_in, len, fd_out, NULL);\n+}\n+\n static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n \t\t\t   void *dst, unsigned size, int options)\n {\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 6d2a63db23..43986c525c 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -33,6 +33,8 @@ int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int write_packetized_from_fd_no_flush(int fd_in, int fd_out);\n int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out);\n+int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n+\t\t\t\t\t     int fd_out, int *count_ptr);\n \n /*\n  * Stdio versions of packet_write functions. When mixing these with fd\ndiff --git a/t/helper/test-rot13-filter.c b/t/helper/test-rot13-filter.c\nnew file mode 100644\nindex 0000000000..bbad031aee\n--- /dev/null\n+++ b/t/helper/test-rot13-filter.c\n@@ -0,0 +1,396 @@\n+/*\n+ * Example implementation for the Git filter protocol version 2\n+ * See Documentation/gitattributes.txt, section \"Filter Protocol\"\n+ *\n+ * Usage: test-tool rot13-filter [--always-delay] <log path> <capabilities>\n+ *\n+ * Log path defines a debug log file that the script writes to. The\n+ * subsequent arguments define a list of supported protocol capabilities\n+ * (\"clean\", \"smudge\", etc).\n+ *\n+ * When --always-delay is given all pathnames with the \"can-delay\" flag\n+ * that don't appear on the list bellow are delayed with a count of 1\n+ * (see more below).\n+ *\n+ * This implementation supports special test cases:\n+ * (1) If data with the pathname \"clean-write-fail.r\" is processed with\n+ *     a \"clean\" operation then the write operation will die.\n+ * (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n+ *     a \"smudge\" operation then the write operation will die.\n+ * (3) If data with the pathname \"error.r\" is processed with any\n+ *     operation then the filter signals that it cannot or does not want\n+ *     to process the file.\n+ * (4) If data with the pathname \"abort.r\" is processed with any\n+ *     operation then the filter signals that it cannot or does not want\n+ *     to process the file and any file after that is processed with the\n+ *     same command.\n+ * (5) If data with a pathname that is a key in the delay hash is\n+ *     requested (e.g. \"test-delay10.a\") then the filter responds with\n+ *     a \"delay\" status and sets the \"requested\" field in the delay hash.\n+ *     The filter will signal the availability of this object after\n+ *     \"count\" (field in delay hash) \"list_available_blobs\" commands.\n+ * (6) If data with the pathname \"missing-delay.a\" is processed that the\n+ *     filter will drop the path from the \"list_available_blobs\" response.\n+ * (7) If data with the pathname \"invalid-delay.a\" is processed that the\n+ *     filter will add the path \"unfiltered\" which was not delayed before\n+ *     to the \"list_available_blobs\" response.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"pkt-line.h\"\n+#include \"string-list.h\"\n+#include \"strmap.h\"\n+\n+static FILE *logfile;\n+static int always_delay;\n+static struct strmap delay = STRMAP_INIT;\n+static struct string_list requested_caps = STRING_LIST_INIT_NODUP;\n+\n+static int has_capability(const char *cap)\n+{\n+\treturn unsorted_string_list_has_string(&requested_caps, cap);\n+}\n+\n+static char *rot13(char *str)\n+{\n+\tchar *c;\n+\tfor (c = str; *c; c++) {\n+\t\tif (*c >= 'a' && *c <= 'z')\n+\t\t\t*c = 'a' + (*c - 'a' + 13) % 26;\n+\t\telse if (*c >= 'A' && *c <= 'Z')\n+\t\t\t*c = 'A' + (*c - 'A' + 13) % 26;\n+\t}\n+\treturn str;\n+}\n+\n+static char *skip_key_dup(const char *buf, size_t size, const char *key)\n+{\n+\tstruct strbuf keybuf = STRBUF_INIT;\n+\tstrbuf_addf(&keybuf, \"%s=\", key);\n+\tif (!skip_prefix_mem(buf, size, keybuf.buf, &buf, &size) || !size)\n+\t\tdie(\"bad %s: '%s'\", key, xstrndup(buf, size));\n+\tstrbuf_release(&keybuf);\n+\treturn xstrndup(buf, size);\n+}\n+\n+/*\n+ * Read a text packet, expecting that it is in the form \"key=value\" for\n+ * the given key. An EOF does not trigger any error and is reported\n+ * back to the caller with NULL. Die if the \"key\" part of \"key=value\" does\n+ * not match the given key, or the value part is empty.\n+ */\n+static char *packet_key_val_read(const char *key)\n+{\n+\tint size;\n+\tchar *buf;\n+\tif (packet_read_line_gently(0, &size, &buf) < 0)\n+\t\treturn NULL;\n+\treturn skip_key_dup(buf, size, key);\n+}\n+\n+static struct string_list *packet_read_capabilities(void)\n+{\n+\tstruct string_list *caps = xmalloc(sizeof(*caps));\n+\tstring_list_init_dup(caps);\n+\twhile (1) {\n+\t\tint size;\n+\t\tchar *buf = packet_read_line(0, &size);\n+\t\tif (!buf)\n+\t\t\tbreak;\n+\t\tstring_list_append_nodup(caps,\n+\t\t\t\t\t skip_key_dup(buf, size, \"capability\"));\n+\t}\n+\treturn caps;\n+}\n+\n+/* Read remote capabilities and check them against capabilities we require */\n+static struct string_list *packet_read_and_check_capabilities(\n+\t\tstruct string_list *required_caps)\n+{\n+\tstruct string_list *remote_caps = packet_read_capabilities();\n+\tstruct string_list_item *item;\n+\tfor_each_string_list_item(item, required_caps) {\n+\t\tif (!unsorted_string_list_has_string(remote_caps, item->string)) {\n+\t\t\tdie(\"required '%s' capability not available from remote\",\n+\t\t\t    item->string);\n+\t\t}\n+\t}\n+\treturn remote_caps;\n+}\n+\n+/*\n+ * Check our capabilities we want to advertise against the remote ones\n+ * and then advertise our capabilities\n+ */\n+static void packet_check_and_write_capabilities(struct string_list *remote_caps,\n+\t\t\t\t\t\tstruct string_list *our_caps)\n+{\n+\tstruct string_list_item *item;\n+\tfor_each_string_list_item(item, our_caps) {\n+\t\tif (!unsorted_string_list_has_string(remote_caps, item->string)) {\n+\t\t\tdie(\"our capability '%s' is not available from remote\",\n+\t\t\t    item->string);\n+\t\t}\n+\t\tpacket_write_fmt(1, \"capability=%s\\n\", item->string);\n+\t}\n+\tpacket_flush(1);\n+}\n+\n+struct delay_entry {\n+\tint requested, count;\n+\tchar *output;\n+};\n+\n+static void command_loop(void)\n+{\n+\twhile (1) {\n+\t\tchar *command = packet_key_val_read(\"command\");\n+\t\tif (!command) {\n+\t\t\tfprintf(logfile, \"STOP\\n\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tfprintf(logfile, \"IN: %s\", command);\n+\n+\t\tif (!strcmp(command, \"list_available_blobs\")) {\n+\t\t\tstruct hashmap_iter iter;\n+\t\t\tstruct strmap_entry *ent;\n+\t\t\tstruct string_list_item *str_item;\n+\t\t\tstruct string_list paths = STRING_LIST_INIT_NODUP;\n+\n+\t\t\t/* flush */\n+\t\t\tif (packet_read_line(0, NULL))\n+\t\t\t\tdie(\"bad list_available_blobs end\");\n+\n+\t\t\tstrmap_for_each_entry(&delay, &iter, ent) {\n+\t\t\t\tstruct delay_entry *delay_entry = ent->value;\n+\t\t\t\tif (!delay_entry->requested)\n+\t\t\t\t\tcontinue;\n+\t\t\t\tdelay_entry->count--;\n+\t\t\t\tif (!strcmp(ent->key, \"invalid-delay.a\")) {\n+\t\t\t\t\t/* Send Git a pathname that was not delayed earlier */\n+\t\t\t\t\tpacket_write_fmt(1, \"pathname=unfiltered\");\n+\t\t\t\t}\n+\t\t\t\tif (!strcmp(ent->key, \"missing-delay.a\")) {\n+\t\t\t\t\t/* Do not signal Git that this file is available */\n+\t\t\t\t} else if (!delay_entry->count) {\n+\t\t\t\t\tstring_list_insert(&paths, ent->key);\n+\t\t\t\t\tpacket_write_fmt(1, \"pathname=%s\", ent->key);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\t/* Print paths in sorted order. */\n+\t\t\tfor_each_string_list_item(str_item, &paths)\n+\t\t\t\tfprintf(logfile, \" %s\", str_item->string);\n+\t\t\tstring_list_clear(&paths, 0);\n+\n+\t\t\tpacket_flush(1);\n+\n+\t\t\tfprintf(logfile, \" [OK]\\n\");\n+\t\t\tpacket_write_fmt(1, \"status=success\");\n+\t\t\tpacket_flush(1);\n+\t\t} else {\n+\t\t\tchar *buf, *output;\n+\t\t\tint size;\n+\t\t\tchar *pathname;\n+\t\t\tstruct delay_entry *entry;\n+\t\t\tstruct strbuf input = STRBUF_INIT;\n+\n+\t\t\tpathname = packet_key_val_read(\"pathname\");\n+\t\t\tif (!pathname)\n+\t\t\t\tdie(\"unexpected EOF while expecting pathname\");\n+\t\t\tfprintf(logfile, \" %s\", pathname);\n+\n+\t\t\t/* Read until flush */\n+\t\t\tbuf = packet_read_line(0, &size);\n+\t\t\twhile (buf) {\n+\t\t\t\tif (!strcmp(buf, \"can-delay=1\")) {\n+\t\t\t\t\tentry = strmap_get(&delay, pathname);\n+\t\t\t\t\tif (entry && !entry->requested) {\n+\t\t\t\t\t\tentry->requested = 1;\n+\t\t\t\t\t} else if (!entry && always_delay) {\n+\t\t\t\t\t\tentry = xcalloc(1, sizeof(*entry));\n+\t\t\t\t\t\tentry->requested = 1;\n+\t\t\t\t\t\tentry->count = 1;\n+\t\t\t\t\t\tstrmap_put(&delay, pathname, entry);\n+\t\t\t\t\t}\n+\t\t\t\t} else if (starts_with(buf, \"ref=\") ||\n+\t\t\t\t\t   starts_with(buf, \"treeish=\") ||\n+\t\t\t\t\t   starts_with(buf, \"blob=\")) {\n+\t\t\t\t\tfprintf(logfile, \" %s\", buf);\n+\t\t\t\t} else {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * In general, filters need to be graceful about\n+\t\t\t\t\t * new metadata, since it's documented that we\n+\t\t\t\t\t * can pass any key-value pairs, but for tests,\n+\t\t\t\t\t * let's be a little stricter.\n+\t\t\t\t\t */\n+\t\t\t\t\tdie(\"Unknown message '%s'\", buf);\n+\t\t\t\t}\n+\t\t\t\tbuf = packet_read_line(0, &size);\n+\t\t\t}\n+\n+\n+\t\t\tread_packetized_to_strbuf(0, &input, 0);\n+\t\t\tfprintf(logfile, \" %\"PRIuMAX\" [OK] -- \", (uintmax_t)input.len);\n+\n+\t\t\tentry = strmap_get(&delay, pathname);\n+\t\t\tif (entry && entry->output) {\n+\t\t\t\toutput = entry->output;\n+\t\t\t} else if (!strcmp(pathname, \"error.r\") || !strcmp(pathname, \"abort.r\")) {\n+\t\t\t\toutput = \"\";\n+\t\t\t} else if (!strcmp(command, \"clean\") && has_capability(\"clean\")) {\n+\t\t\t\toutput = rot13(input.buf);\n+\t\t\t} else if (!strcmp(command, \"smudge\") && has_capability(\"smudge\")) {\n+\t\t\t\toutput = rot13(input.buf);\n+\t\t\t} else {\n+\t\t\t\tdie(\"bad command '%s'\", command);\n+\t\t\t}\n+\n+\t\t\tif (!strcmp(pathname, \"error.r\")) {\n+\t\t\t\tfprintf(logfile, \"[ERROR]\\n\");\n+\t\t\t\tpacket_write_fmt(1, \"status=error\");\n+\t\t\t\tpacket_flush(1);\n+\t\t\t} else if (!strcmp(pathname, \"abort.r\")) {\n+\t\t\t\tfprintf(logfile, \"[ABORT]\\n\");\n+\t\t\t\tpacket_write_fmt(1, \"status=abort\");\n+\t\t\t\tpacket_flush(1);\n+\t\t\t} else if (!strcmp(command, \"smudge\") &&\n+\t\t\t\t   (entry = strmap_get(&delay, pathname)) &&\n+\t\t\t\t   entry->requested == 1) {\n+\t\t\t\tfprintf(logfile, \"[DELAYED]\\n\");\n+\t\t\t\tpacket_write_fmt(1, \"status=delayed\");\n+\t\t\t\tpacket_flush(1);\n+\t\t\t\tentry->requested = 2;\n+\t\t\t\tentry->output = xstrdup(output);\n+\t\t\t} else {\n+\t\t\t\tint i, nr_packets;\n+\t\t\t\tsize_t output_len;\n+\t\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\t\tpacket_write_fmt(1, \"status=success\");\n+\t\t\t\tpacket_flush(1);\n+\n+\t\t\t\tstrbuf_addf(&sb, \"%s-write-fail.r\", command);\n+\t\t\t\tif (!strcmp(pathname, sb.buf)) {\n+\t\t\t\t\tfprintf(logfile, \"[WRITE FAIL]\\n\");\n+\t\t\t\t\tdie(\"%s write error\", command);\n+\t\t\t\t}\n+\n+\t\t\t\toutput_len = strlen(output);\n+\t\t\t\tfprintf(logfile, \"OUT: %\"PRIuMAX\" \", (uintmax_t)output_len);\n+\n+\t\t\t\tif (write_packetized_from_buf_no_flush_count(output,\n+\t\t\t\t\toutput_len, 1, &nr_packets))\n+\t\t\t\t\tdie(\"failed to write buffer to stdout\");\n+\t\t\t\tpacket_flush(1);\n+\n+\t\t\t\tfor (i = 0; i < nr_packets; i++)\n+\t\t\t\t\tfprintf(logfile, \".\");\n+\t\t\t\tfprintf(logfile, \" [OK]\\n\");\n+\n+\t\t\t\tpacket_flush(1);\n+\t\t\t\tstrbuf_release(&sb);\n+\t\t\t}\n+\t\t\tfree(pathname);\n+\t\t\tstrbuf_release(&input);\n+\t\t}\n+\t\tfree(command);\n+\t}\n+}\n+\n+static void free_delay_hash(void)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *ent;\n+\n+\tstrmap_for_each_entry(&delay, &iter, ent) {\n+\t\tstruct delay_entry *delay_entry = ent->value;\n+\t\tfree(delay_entry->output);\n+\t\tfree(delay_entry);\n+\t}\n+\tstrmap_clear(&delay, 0);\n+}\n+\n+static void add_delay_entry(char *pathname, int count)\n+{\n+\tstruct delay_entry *entry = xcalloc(1, sizeof(*entry));\n+\tentry->count = count;\n+\tif (strmap_put(&delay, pathname, entry))\n+\t\tBUG(\"adding the same path twice to delay hash?\");\n+}\n+\n+static void packet_initialize(const char *name, int version)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint size;\n+\tchar *pkt_buf = packet_read_line(0, &size);\n+\n+\tstrbuf_addf(&sb, \"%s-client\", name);\n+\tif (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n+\t\tdie(\"bad initialize: '%s'\", xstrndup(pkt_buf, size));\n+\n+\tstrbuf_reset(&sb);\n+\tstrbuf_addf(&sb, \"version=%d\", version);\n+\tpkt_buf = packet_read_line(0, &size);\n+\tif (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n+\t\tdie(\"bad version: '%s'\", xstrndup(pkt_buf, size));\n+\n+\tpkt_buf = packet_read_line(0, &size);\n+\tif (pkt_buf)\n+\t\tdie(\"bad version end: '%s'\", xstrndup(pkt_buf, size));\n+\n+\tpacket_write_fmt(1, \"%s-server\", name);\n+\tpacket_write_fmt(1, \"version=%d\", version);\n+\tpacket_flush(1);\n+\tstrbuf_release(&sb);\n+}\n+\n+static char *rot13_usage = \"test-tool rot13-filter [--always-delay] <log path> <capabilities>\";\n+\n+int cmd__rot13_filter(int argc, const char **argv)\n+{\n+\tint i = 1;\n+\tstruct string_list *remote_caps, supported_caps = STRING_LIST_INIT_NODUP;\n+\n+\tstring_list_append(&supported_caps, \"clean\");\n+\tstring_list_append(&supported_caps, \"smudge\");\n+\tstring_list_append(&supported_caps, \"delay\");\n+\n+\tif (argc > 1 && !strcmp(argv[i], \"--always-delay\")) {\n+\t\talways_delay = 1;\n+\t\ti++;\n+\t}\n+\tif (argc - i < 2)\n+\t\tusage(rot13_usage);\n+\n+\tlogfile = fopen(argv[i++], \"a\");\n+\tif (!logfile)\n+\t\tdie_errno(\"failed to open log file\");\n+\n+\tfor ( ; i < argc; i++)\n+\t\tstring_list_append(&requested_caps, argv[i]);\n+\n+\tadd_delay_entry(\"test-delay10.a\", 1);\n+\tadd_delay_entry(\"test-delay11.a\", 1);\n+\tadd_delay_entry(\"test-delay20.a\", 2);\n+\tadd_delay_entry(\"test-delay10.b\", 1);\n+\tadd_delay_entry(\"missing-delay.a\", 1);\n+\tadd_delay_entry(\"invalid-delay.a\", 1);\n+\n+\tfprintf(logfile, \"START\\n\");\n+\n+\tpacket_initialize(\"git-filter\", 2);\n+\n+\tremote_caps = packet_read_and_check_capabilities(&supported_caps);\n+\tpacket_check_and_write_capabilities(remote_caps, &requested_caps);\n+\tfprintf(logfile, \"init handshake complete\\n\");\n+\n+\tstring_list_clear(&supported_caps, 0);\n+\tstring_list_clear(remote_caps, 0);\n+\n+\tcommand_loop();\n+\n+\tfclose(logfile);\n+\tstring_list_clear(&requested_caps, 0);\n+\tfree_delay_hash();\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 318fdbab0c..d6a560f832 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -65,6 +65,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"rot13-filter\", cmd__rot13_filter },\n \t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex bb79927163..21a91b1019 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -54,6 +54,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__rot13_filter(int argc, const char **argv);\n int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 1c840348bd..963b66e08c 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -735,7 +735,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\trm -f debug.log &&\n \t\tgit checkout --quiet --no-progress . 2>git-stderr.log &&\n \n-\t\tgrep \"smudge write error at\" git-stderr.log &&\n+\t\tgrep \"smudge write error\" git-stderr.log &&\n \t\ttest_i18ngrep \"error: external filter\" git-stderr.log &&\n \n \t\tcat >expected.log <<-EOF &&\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 7bb93768f3..1447bc0a24 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -1,247 +1,5 @@\n-#\n-# Example implementation for the Git filter protocol version 2\n-# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n-#\n-# Usage: rot13-filter.pl [--always-delay] <log path> <capabilities>\n-#\n-# Log path defines a debug log file that the script writes to. The\n-# subsequent arguments define a list of supported protocol capabilities\n-# (\"clean\", \"smudge\", etc).\n-#\n-# When --always-delay is given all pathnames with the \"can-delay\" flag\n-# that don't appear on the list bellow are delayed with a count of 1\n-# (see more below).\n-#\n-# This implementation supports special test cases:\n-# (1) If data with the pathname \"clean-write-fail.r\" is processed with\n-#     a \"clean\" operation then the write operation will die.\n-# (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n-#     a \"smudge\" operation then the write operation will die.\n-# (3) If data with the pathname \"error.r\" is processed with any\n-#     operation then the filter signals that it cannot or does not want\n-#     to process the file.\n-# (4) If data with the pathname \"abort.r\" is processed with any\n-#     operation then the filter signals that it cannot or does not want\n-#     to process the file and any file after that is processed with the\n-#     same command.\n-# (5) If data with a pathname that is a key in the DELAY hash is\n-#     requested (e.g. \"test-delay10.a\") then the filter responds with\n-#     a \"delay\" status and sets the \"requested\" field in the DELAY hash.\n-#     The filter will signal the availability of this object after\n-#     \"count\" (field in DELAY hash) \"list_available_blobs\" commands.\n-# (6) If data with the pathname \"missing-delay.a\" is processed that the\n-#     filter will drop the path from the \"list_available_blobs\" response.\n-# (7) If data with the pathname \"invalid-delay.a\" is processed that the\n-#     filter will add the path \"unfiltered\" which was not delayed before\n-#     to the \"list_available_blobs\" response.\n-#\n-\n use 5.008;\n-sub gitperllib {\n-\t# Git assumes that all path lists are Unix-y colon-separated ones. But\n-\t# when the Git for Windows executes the test suite, its MSYS2 Bash\n-\t# calls git.exe, and colon-separated path lists are converted into\n-\t# Windows-y semicolon-separated lists of *Windows* paths (which\n-\t# naturally contain a colon after the drive letter, so splitting by\n-\t# colons simply does not cut it).\n-\t#\n-\t# Detect semicolon-separated path list and handle them appropriately.\n \n-\tif ($ENV{GITPERLLIB} =~ /;/) {\n-\t\treturn split(/;/, $ENV{GITPERLLIB});\n-\t}\n-\treturn split(/:/, $ENV{GITPERLLIB});\n-}\n-use lib (gitperllib());\n-use strict;\n-use warnings;\n-use IO::File;\n-use Git::Packet;\n-\n-my $MAX_PACKET_CONTENT_SIZE = 65516;\n-\n-my $always_delay = 0;\n-if ( $ARGV[0] eq '--always-delay' ) {\n-\t$always_delay = 1;\n-\tshift @ARGV;\n-}\n-\n-my $log_file                = shift @ARGV;\n-my @capabilities            = @ARGV;\n-\n-open my $debug, \">>\", $log_file or die \"cannot open log file: $!\";\n-\n-my %DELAY = (\n-\t'test-delay10.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'test-delay11.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'test-delay20.a' => { \"requested\" => 0, \"count\" => 2 },\n-\t'test-delay10.b' => { \"requested\" => 0, \"count\" => 1 },\n-\t'missing-delay.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'invalid-delay.a' => { \"requested\" => 0, \"count\" => 1 },\n-);\n-\n-sub rot13 {\n-\tmy $str = shift;\n-\t$str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n-\treturn $str;\n-}\n-\n-print $debug \"START\\n\";\n-$debug->flush();\n-\n-packet_initialize(\"git-filter\", 2);\n-\n-my %remote_caps = packet_read_and_check_capabilities(\"clean\", \"smudge\", \"delay\");\n-packet_check_and_write_capabilities(\\%remote_caps, @capabilities);\n-\n-print $debug \"init handshake complete\\n\";\n-$debug->flush();\n-\n-while (1) {\n-\tmy ( $res, $command ) = packet_key_val_read(\"command\");\n-\tif ( $res == -1 ) {\n-\t\tprint $debug \"STOP\\n\";\n-\t\texit();\n-\t}\n-\tprint $debug \"IN: $command\";\n-\t$debug->flush();\n-\n-\tif ( $command eq \"list_available_blobs\" ) {\n-\t\t# Flush\n-\t\tpacket_compare_lists([1, \"\"], packet_bin_read()) ||\n-\t\t\tdie \"bad list_available_blobs end\";\n-\n-\t\tforeach my $pathname ( sort keys %DELAY ) {\n-\t\t\tif ( $DELAY{$pathname}{\"requested\"} >= 1 ) {\n-\t\t\t\t$DELAY{$pathname}{\"count\"} = $DELAY{$pathname}{\"count\"} - 1;\n-\t\t\t\tif ( $pathname eq \"invalid-delay.a\" ) {\n-\t\t\t\t\t# Send Git a pathname that was not delayed earlier\n-\t\t\t\t\tpacket_txt_write(\"pathname=unfiltered\");\n-\t\t\t\t}\n-\t\t\t\tif ( $pathname eq \"missing-delay.a\" ) {\n-\t\t\t\t\t# Do not signal Git that this file is available\n-\t\t\t\t} elsif ( $DELAY{$pathname}{\"count\"} == 0 ) {\n-\t\t\t\t\tprint $debug \" $pathname\";\n-\t\t\t\t\tpacket_txt_write(\"pathname=$pathname\");\n-\t\t\t\t}\n-\t\t\t}\n-\t\t}\n-\n-\t\tpacket_flush();\n-\n-\t\tprint $debug \" [OK]\\n\";\n-\t\t$debug->flush();\n-\t\tpacket_txt_write(\"status=success\");\n-\t\tpacket_flush();\n-\t} else {\n-\t\tmy ( $res, $pathname ) = packet_key_val_read(\"pathname\");\n-\t\tif ( $res == -1 ) {\n-\t\t\tdie \"unexpected EOF while expecting pathname\";\n-\t\t}\n-\t\tprint $debug \" $pathname\";\n-\t\t$debug->flush();\n-\n-\t\t# Read until flush\n-\t\tmy ( $done, $buffer ) = packet_txt_read();\n-\t\twhile ( $buffer ne '' ) {\n-\t\t\tif ( $buffer eq \"can-delay=1\" ) {\n-\t\t\t\tif ( exists $DELAY{$pathname} and $DELAY{$pathname}{\"requested\"} == 0 ) {\n-\t\t\t\t\t$DELAY{$pathname}{\"requested\"} = 1;\n-\t\t\t\t} elsif ( !exists $DELAY{$pathname} and $always_delay ) {\n-\t\t\t\t\t$DELAY{$pathname} = { \"requested\" => 1, \"count\" => 1 };\n-\t\t\t\t}\n-\t\t\t} elsif ($buffer =~ /^(ref|treeish|blob)=/) {\n-\t\t\t\tprint $debug \" $buffer\";\n-\t\t\t} else {\n-\t\t\t\t# In general, filters need to be graceful about\n-\t\t\t\t# new metadata, since it's documented that we\n-\t\t\t\t# can pass any key-value pairs, but for tests,\n-\t\t\t\t# let's be a little stricter.\n-\t\t\t\tdie \"Unknown message '$buffer'\";\n-\t\t\t}\n-\n-\t\t\t( $done, $buffer ) = packet_txt_read();\n-\t\t}\n-\t\tif ( $done == -1 ) {\n-\t\t\tdie \"unexpected EOF after pathname '$pathname'\";\n-\t\t}\n-\n-\t\tmy $input = \"\";\n-\t\t{\n-\t\t\tbinmode(STDIN);\n-\t\t\tmy $buffer;\n-\t\t\tmy $done = 0;\n-\t\t\twhile ( !$done ) {\n-\t\t\t\t( $done, $buffer ) = packet_bin_read();\n-\t\t\t\t$input .= $buffer;\n-\t\t\t}\n-\t\t\tif ( $done == -1 ) {\n-\t\t\t\tdie \"unexpected EOF while reading input for '$pathname'\";\n-\t\t\t}\t\t\t\n-\t\t\tprint $debug \" \" . length($input) . \" [OK] -- \";\n-\t\t\t$debug->flush();\n-\t\t}\n-\n-\t\tmy $output;\n-\t\tif ( exists $DELAY{$pathname} and exists $DELAY{$pathname}{\"output\"} ) {\n-\t\t\t$output = $DELAY{$pathname}{\"output\"}\n-\t\t} elsif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n-\t\t\t$output = \"\";\n-\t\t} elsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n-\t\t\t$output = rot13($input);\n-\t\t} elsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n-\t\t\t$output = rot13($input);\n-\t\t} else {\n-\t\t\tdie \"bad command '$command'\";\n-\t\t}\n-\n-\t\tif ( $pathname eq \"error.r\" ) {\n-\t\t\tprint $debug \"[ERROR]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=error\");\n-\t\t\tpacket_flush();\n-\t\t} elsif ( $pathname eq \"abort.r\" ) {\n-\t\t\tprint $debug \"[ABORT]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=abort\");\n-\t\t\tpacket_flush();\n-\t\t} elsif ( $command eq \"smudge\" and\n-\t\t\texists $DELAY{$pathname} and\n-\t\t\t$DELAY{$pathname}{\"requested\"} == 1 ) {\n-\t\t\tprint $debug \"[DELAYED]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=delayed\");\n-\t\t\tpacket_flush();\n-\t\t\t$DELAY{$pathname}{\"requested\"} = 2;\n-\t\t\t$DELAY{$pathname}{\"output\"} = $output;\n-\t\t} else {\n-\t\t\tpacket_txt_write(\"status=success\");\n-\t\t\tpacket_flush();\n-\n-\t\t\tif ( $pathname eq \"${command}-write-fail.r\" ) {\n-\t\t\t\tprint $debug \"[WRITE FAIL]\\n\";\n-\t\t\t\t$debug->flush();\n-\t\t\t\tdie \"${command} write error\";\n-\t\t\t}\n-\n-\t\t\tprint $debug \"OUT: \" . length($output) . \" \";\n-\t\t\t$debug->flush();\n-\n-\t\t\twhile ( length($output) > 0 ) {\n-\t\t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\t\tpacket_bin_write($packet);\n-\t\t\t\t# dots represent the number of packets\n-\t\t\t\tprint $debug \".\";\n-\t\t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n-\t\t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\t\t} else {\n-\t\t\t\t\t$output = \"\";\n-\t\t\t\t}\n-\t\t\t}\n-\t\t\tpacket_flush();\n-\t\t\tprint $debug \" [OK]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_flush();\n-\t\t}\n-\t}\n-}\n+my @quoted_args = map \"'$_'\", @ARGV;\n+exec \"test-tool rot13-filter @quoted_args\";\n+die \"failed to exec test-tool\";\n-- \n2.37.1\n\n"},{"id":"459824","messageId":"220723.86tu78qvjb.gmgdl@evledraar.gmail.com","threadId":"58212","inReplyTo":"99823077be77bc621cfa8ccf3303bd612da343ad.1658518769.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 1/2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-23T04:52:59Z","receivedAt":"2022-07-23T04:53:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 22 2022, Matheus Tavares wrote:\n\n> +my @quoted_args = map \"'$_'\", @ARGV;\n> +exec \"test-tool rot13-filter @quoted_args\";\n> +die \"failed to exec test-tool\";\n\nYou end up throwing this away, but this whole escaping business is just\nbad use of the API, you can pass a list to \"exec\" have it escape\narguments.  See \"perldoc -f exec\".\n"},{"id":"459825","messageId":"220723.86pmhwquie.gmgdl@evledraar.gmail.com","threadId":"58212","inReplyTo":"99823077be77bc621cfa8ccf3303bd612da343ad.1658518769.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 1/2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-23T04:59:49Z","receivedAt":"2022-07-23T05:16:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 22 2022, Matheus Tavares wrote:\n\nLooking a bit closer...\n\n> however, that we still use the script as a wrapper at\n> this commit, in order to minimize the amount of changes it introduces\n> and help reviewers. At the next commit we will properly remove the\n> script and adjust the affected tests to use test-tool.\n\nI'd prefer if we just squashed this, if you want to avoid some of the\ndiff verbosity you could leave the PERL prereq on all the\ntest_expect_success and remove it in a 2/2 (we just wouldn't run the\ntest until then).\n\nBut I think it's all boilerplate, so just doing it in one step would be\nbetter, reasoning about the in-between steps is harder IMO (e.g. \"exec\"\nescaping or whatever)>\n\n> +static char *rot13(char *str)\n> +{\n> +\tchar *c;\n> +\tfor (c = str; *c; c++) {\n> +\t\tif (*c >= 'a' && *c <= 'z')\n> +\t\t\t*c = 'a' + (*c - 'a' + 13) % 26;\n> +\t\telse if (*c >= 'A' && *c <= 'Z')\n> +\t\t\t*c = 'A' + (*c - 'A' + 13) % 26;\n> +\t}\n> +\treturn str;\n> +}\n\nLooks fine, but we should probably put in our CodingGuidelines at some\npoint that we don't care about EBCDIC, as this isn't portable C (but\nprobably portable enough, as we can probably assume ASCII) :)\n\n> +static struct string_list *packet_read_capabilities(void)\n> +{\n> +\tstruct string_list *caps = xmalloc(sizeof(*caps));\n\nmalloc here...\n\n> +\tstring_list_init_dup(caps);\n> +\twhile (1) {\n> +\t\tint size;\n> +\t\tchar *buf = packet_read_line(0, &size);\n> +\t\tif (!buf)\n> +\t\t\tbreak;\n> +\t\tstring_list_append_nodup(caps,\n> +\t\t\t\t\t skip_key_dup(buf, size, \"capability\"));\n> +\t}\n> +\treturn caps;\n> +}\n> +\n> +/* Read remote capabilities and check them against capabilities we require */\n> +static struct string_list *packet_read_and_check_capabilities(\n> +\t\tstruct string_list *required_caps)\n> +{\n> +\tstruct string_list *remote_caps = packet_read_capabilities();\n\n...and here...\n> +\tstruct string_list_item *item;\n> +\tfor_each_string_list_item(item, required_caps) {\n> +\t\tif (!unsorted_string_list_has_string(remote_caps, item->string)) {\n> +\t\t\tdie(\"required '%s' capability not available from remote\",\n> +\t\t\t    item->string);\n> +\t\t}\n> +\t}\n> +\treturn remote_caps;\n\n...we'll return it...\n\n> +\tremote_caps = packet_read_and_check_capabilities(&supported_caps);\n> +\tpacket_check_and_write_capabilities(remote_caps, &requested_caps);\n> +\tfprintf(logfile, \"init handshake complete\\n\");\n> +\n> +\tstring_list_clear(&supported_caps, 0);\n> +\tstring_list_clear(remote_caps, 0);\n\n..and here you're missing a free(), but I wonder why not just declare\nthis string_list in this function, and pass it down instead?\n\nIt's unfortunate that none of these tests seem to pass with\nSANITIZE=leak already, but the new command seems not to leak from a\ntrivial glance except for in that one case.\n\nNot knowing much about the filtering mechanism, I wonder if this code\nhere wouldn't be better as a built-in some day. I.e. isn't this all\nshimmy we need to talk to some arbitrary conversion filter, except for\nthe rot13 part?\n\nSo if we just invoked a \"tr\" with run_command() to do the actual rot13\nfiltering we could do any sort of arbitrary replacement, and present a\nvariant of this this command as a \"if you can't be bothered with\npacket-line\" in gitattributes(5)...\n\n...but maybe that's hopeless for some reason I'm missing, in any case,\nmore #leftoverbits.\n\n"},{"id":"459834","messageId":"CAHd-oW4BCXNrUcSHLzKsrK0BTPCpGTi_fo8Buxte=RQDJahipw@mail.gmail.com","threadId":"58212","inReplyTo":"220723.86pmhwquie.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 1/2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-23T13:36:26Z","receivedAt":"2022-07-23T13:36:45Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sat, Jul 23, 2022 at 2:15 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> On Fri, Jul 22 2022, Matheus Tavares wrote:\n>\n> Looking a bit closer...\n>\n> > however, that we still use the script as a wrapper at\n> > this commit, in order to minimize the amount of changes it introduces\n> > and help reviewers. At the next commit we will properly remove the\n> > script and adjust the affected tests to use test-tool.\n>\n> I'd prefer if we just squashed this, if you want to avoid some of the\n> diff verbosity you could leave the PERL prereq on all the\n> test_expect_success and remove it in a 2/2 (we just wouldn't run the\n> test until then).\n>\n> But I think it's all boilerplate, so just doing it in one step would be\n> better, reasoning about the in-between steps is harder IMO (e.g. \"exec\"\n> escaping or whatever)\n\nSure, will do! My split attempt was to try to reduce the mental load\nfor the reviewers, but if it ended up making it harder instead of\nhelping, let's squash the two patches.\n\n> > +     remote_caps = packet_read_and_check_capabilities(&supported_caps);\n> > +     packet_check_and_write_capabilities(remote_caps, &requested_caps);\n> > +     fprintf(logfile, \"init handshake complete\\n\");\n> > +\n> > +     string_list_clear(&supported_caps, 0);\n> > +     string_list_clear(remote_caps, 0);\n>\n> ..and here you're missing a free(), but I wonder why not just declare\n> this string_list in this function, and pass it down instead?\n\nMakes sense, will do.\n\n> Not knowing much about the filtering mechanism, I wonder if this code\n> here wouldn't be better as a built-in some day. I.e. isn't this all\n> shimmy we need to talk to some arbitrary conversion filter, except for\n> the rot13 part?\n>\n> So if we just invoked a \"tr\" with run_command() to do the actual rot13\n> filtering we could do any sort of arbitrary replacement, and present a\n> variant of this this command as a \"if you can't be bothered with\n> packet-line\" in gitattributes(5)...\n\nHmm, maybe so. But I would expect that someone building a long running\nprocess filter (as opposed to a \"single-shot\" filter, like the \"tr\"\nuse case)  would also want to have finer control over the\ncommunication and \"queueing\" mechanics. And I'm not sure if that would\nbe feasible via an off-the-shelf solution packed with Git itself.\n\nFor example, while some filters may process the received paths\nsequentially, Git-LFS will use the delay capability to queue and\ndownload blobs in the background, examining the queue every time Git\nasks for the list of currently available blobs.\n\nAnyways, I could see these packet-line routines being exported as a\nlibrary for those writing such filters.\n"},{"id":"459851","messageId":"f38f722de7c3323207eda5ea632b5acd3765c285.1658675222.git.matheus.bernardino@usp.br","threadId":"58212","inReplyTo":"cover.1658518769.git.matheus.bernardino@usp.br","subject":"[PATCH v2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-24T15:09:18Z","receivedAt":"2022-07-24T15:09:47Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This script is currently used by three test files: t0021-conversion.sh,\nt2080-parallel-checkout-basics.sh, and\nt2082-parallel-checkout-attributes.sh. To avoid the need for the PERL\ndependency at these tests, let's convert the script to a C test-tool\ncommand.\n\nNote that there is a small adjustment needed at test t0021-conversion.sh\nbecause it depended on a specific error message given by perl's die\nroutine.\n\nSigned-off-by: Matheus Tavares <matheus.bernardino@usp.br>\n---\n\nChanges since v1:\n- Squashed the two patches together.\n- Declared `remote_caps` at cmd__rot13_filter()'s stack and passed it\n  down the call stack instead of dynamic allocation.\n\n Makefile                                |   1 +\n pkt-line.c                              |  13 +-\n pkt-line.h                              |   2 +\n t/helper/test-rot13-filter.c            | 393 ++++++++++++++++++++++++\n t/helper/test-tool.c                    |   1 +\n t/helper/test-tool.h                    |   1 +\n t/t0021-conversion.sh                   |  71 ++---\n t/t0021/rot13-filter.pl                 | 247 ---------------\n t/t2080-parallel-checkout-basics.sh     |   7 +-\n t/t2082-parallel-checkout-attributes.sh |   7 +-\n 10 files changed, 447 insertions(+), 296 deletions(-)\n create mode 100644 t/helper/test-rot13-filter.c\n delete mode 100644 t/t0021/rot13-filter.pl\n\ndiff --git a/Makefile b/Makefile\nindex 04d0fd1fe6..7cfcf3a911 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -764,6 +764,7 @@ TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n+TEST_BUILTINS_OBJS += test-rot13-filter.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n TEST_BUILTINS_OBJS += test-run-command.o\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 8e43c2def4..4425bdae36 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -309,9 +309,10 @@ int write_packetized_from_fd_no_flush(int fd_in, int fd_out)\n \treturn err;\n }\n \n-int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n+int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n+\t\t\t\t\t     int fd_out, int *count_ptr)\n {\n-\tint err = 0;\n+\tint err = 0, count = 0;\n \tsize_t bytes_written = 0;\n \tsize_t bytes_to_write;\n \n@@ -324,10 +325,18 @@ int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_ou\n \t\t\tbreak;\n \t\terr = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);\n \t\tbytes_written += bytes_to_write;\n+\t\tcount++;\n \t}\n+\tif (count_ptr)\n+\t\t*count_ptr = count;\n \treturn err;\n }\n \n+int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n+{\n+\treturn write_packetized_from_buf_no_flush_count(src_in, len, fd_out, NULL);\n+}\n+\n static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n \t\t\t   void *dst, unsigned size, int options)\n {\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 6d2a63db23..43986c525c 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -33,6 +33,8 @@ int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int write_packetized_from_fd_no_flush(int fd_in, int fd_out);\n int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out);\n+int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n+\t\t\t\t\t     int fd_out, int *count_ptr);\n \n /*\n  * Stdio versions of packet_write functions. When mixing these with fd\ndiff --git a/t/helper/test-rot13-filter.c b/t/helper/test-rot13-filter.c\nnew file mode 100644\nindex 0000000000..536111f272\n--- /dev/null\n+++ b/t/helper/test-rot13-filter.c\n@@ -0,0 +1,393 @@\n+/*\n+ * Example implementation for the Git filter protocol version 2\n+ * See Documentation/gitattributes.txt, section \"Filter Protocol\"\n+ *\n+ * Usage: test-tool rot13-filter [--always-delay] <log path> <capabilities>\n+ *\n+ * Log path defines a debug log file that the script writes to. The\n+ * subsequent arguments define a list of supported protocol capabilities\n+ * (\"clean\", \"smudge\", etc).\n+ *\n+ * When --always-delay is given all pathnames with the \"can-delay\" flag\n+ * that don't appear on the list bellow are delayed with a count of 1\n+ * (see more below).\n+ *\n+ * This implementation supports special test cases:\n+ * (1) If data with the pathname \"clean-write-fail.r\" is processed with\n+ *     a \"clean\" operation then the write operation will die.\n+ * (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n+ *     a \"smudge\" operation then the write operation will die.\n+ * (3) If data with the pathname \"error.r\" is processed with any\n+ *     operation then the filter signals that it cannot or does not want\n+ *     to process the file.\n+ * (4) If data with the pathname \"abort.r\" is processed with any\n+ *     operation then the filter signals that it cannot or does not want\n+ *     to process the file and any file after that is processed with the\n+ *     same command.\n+ * (5) If data with a pathname that is a key in the delay hash is\n+ *     requested (e.g. \"test-delay10.a\") then the filter responds with\n+ *     a \"delay\" status and sets the \"requested\" field in the delay hash.\n+ *     The filter will signal the availability of this object after\n+ *     \"count\" (field in delay hash) \"list_available_blobs\" commands.\n+ * (6) If data with the pathname \"missing-delay.a\" is processed that the\n+ *     filter will drop the path from the \"list_available_blobs\" response.\n+ * (7) If data with the pathname \"invalid-delay.a\" is processed that the\n+ *     filter will add the path \"unfiltered\" which was not delayed before\n+ *     to the \"list_available_blobs\" response.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"pkt-line.h\"\n+#include \"string-list.h\"\n+#include \"strmap.h\"\n+\n+static FILE *logfile;\n+static int always_delay;\n+static struct strmap delay = STRMAP_INIT;\n+static struct string_list requested_caps = STRING_LIST_INIT_NODUP;\n+\n+static int has_capability(const char *cap)\n+{\n+\treturn unsorted_string_list_has_string(&requested_caps, cap);\n+}\n+\n+static char *rot13(char *str)\n+{\n+\tchar *c;\n+\tfor (c = str; *c; c++) {\n+\t\tif (*c >= 'a' && *c <= 'z')\n+\t\t\t*c = 'a' + (*c - 'a' + 13) % 26;\n+\t\telse if (*c >= 'A' && *c <= 'Z')\n+\t\t\t*c = 'A' + (*c - 'A' + 13) % 26;\n+\t}\n+\treturn str;\n+}\n+\n+static char *skip_key_dup(const char *buf, size_t size, const char *key)\n+{\n+\tstruct strbuf keybuf = STRBUF_INIT;\n+\tstrbuf_addf(&keybuf, \"%s=\", key);\n+\tif (!skip_prefix_mem(buf, size, keybuf.buf, &buf, &size) || !size)\n+\t\tdie(\"bad %s: '%s'\", key, xstrndup(buf, size));\n+\tstrbuf_release(&keybuf);\n+\treturn xstrndup(buf, size);\n+}\n+\n+/*\n+ * Read a text packet, expecting that it is in the form \"key=value\" for\n+ * the given key. An EOF does not trigger any error and is reported\n+ * back to the caller with NULL. Die if the \"key\" part of \"key=value\" does\n+ * not match the given key, or the value part is empty.\n+ */\n+static char *packet_key_val_read(const char *key)\n+{\n+\tint size;\n+\tchar *buf;\n+\tif (packet_read_line_gently(0, &size, &buf) < 0)\n+\t\treturn NULL;\n+\treturn skip_key_dup(buf, size, key);\n+}\n+\n+static void packet_read_capabilities(struct string_list *caps)\n+{\n+\twhile (1) {\n+\t\tint size;\n+\t\tchar *buf = packet_read_line(0, &size);\n+\t\tif (!buf)\n+\t\t\tbreak;\n+\t\tstring_list_append_nodup(caps,\n+\t\t\t\t\t skip_key_dup(buf, size, \"capability\"));\n+\t}\n+}\n+\n+/* Read remote capabilities and check them against capabilities we require */\n+static void packet_read_and_check_capabilities(struct string_list *remote_caps,\n+\t\t\t\t\t       struct string_list *required_caps)\n+{\n+\tstruct string_list_item *item;\n+\tpacket_read_capabilities(remote_caps);\n+\tfor_each_string_list_item(item, required_caps) {\n+\t\tif (!unsorted_string_list_has_string(remote_caps, item->string)) {\n+\t\t\tdie(\"required '%s' capability not available from remote\",\n+\t\t\t    item->string);\n+\t\t}\n+\t}\n+}\n+\n+/*\n+ * Check our capabilities we want to advertise against the remote ones\n+ * and then advertise our capabilities\n+ */\n+static void packet_check_and_write_capabilities(struct string_list *remote_caps,\n+\t\t\t\t\t\tstruct string_list *our_caps)\n+{\n+\tstruct string_list_item *item;\n+\tfor_each_string_list_item(item, our_caps) {\n+\t\tif (!unsorted_string_list_has_string(remote_caps, item->string)) {\n+\t\t\tdie(\"our capability '%s' is not available from remote\",\n+\t\t\t    item->string);\n+\t\t}\n+\t\tpacket_write_fmt(1, \"capability=%s\\n\", item->string);\n+\t}\n+\tpacket_flush(1);\n+}\n+\n+struct delay_entry {\n+\tint requested, count;\n+\tchar *output;\n+};\n+\n+static void command_loop(void)\n+{\n+\twhile (1) {\n+\t\tchar *command = packet_key_val_read(\"command\");\n+\t\tif (!command) {\n+\t\t\tfprintf(logfile, \"STOP\\n\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tfprintf(logfile, \"IN: %s\", command);\n+\n+\t\tif (!strcmp(command, \"list_available_blobs\")) {\n+\t\t\tstruct hashmap_iter iter;\n+\t\t\tstruct strmap_entry *ent;\n+\t\t\tstruct string_list_item *str_item;\n+\t\t\tstruct string_list paths = STRING_LIST_INIT_NODUP;\n+\n+\t\t\t/* flush */\n+\t\t\tif (packet_read_line(0, NULL))\n+\t\t\t\tdie(\"bad list_available_blobs end\");\n+\n+\t\t\tstrmap_for_each_entry(&delay, &iter, ent) {\n+\t\t\t\tstruct delay_entry *delay_entry = ent->value;\n+\t\t\t\tif (!delay_entry->requested)\n+\t\t\t\t\tcontinue;\n+\t\t\t\tdelay_entry->count--;\n+\t\t\t\tif (!strcmp(ent->key, \"invalid-delay.a\")) {\n+\t\t\t\t\t/* Send Git a pathname that was not delayed earlier */\n+\t\t\t\t\tpacket_write_fmt(1, \"pathname=unfiltered\");\n+\t\t\t\t}\n+\t\t\t\tif (!strcmp(ent->key, \"missing-delay.a\")) {\n+\t\t\t\t\t/* Do not signal Git that this file is available */\n+\t\t\t\t} else if (!delay_entry->count) {\n+\t\t\t\t\tstring_list_insert(&paths, ent->key);\n+\t\t\t\t\tpacket_write_fmt(1, \"pathname=%s\", ent->key);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\t/* Print paths in sorted order. */\n+\t\t\tfor_each_string_list_item(str_item, &paths)\n+\t\t\t\tfprintf(logfile, \" %s\", str_item->string);\n+\t\t\tstring_list_clear(&paths, 0);\n+\n+\t\t\tpacket_flush(1);\n+\n+\t\t\tfprintf(logfile, \" [OK]\\n\");\n+\t\t\tpacket_write_fmt(1, \"status=success\");\n+\t\t\tpacket_flush(1);\n+\t\t} else {\n+\t\t\tchar *buf, *output;\n+\t\t\tint size;\n+\t\t\tchar *pathname;\n+\t\t\tstruct delay_entry *entry;\n+\t\t\tstruct strbuf input = STRBUF_INIT;\n+\n+\t\t\tpathname = packet_key_val_read(\"pathname\");\n+\t\t\tif (!pathname)\n+\t\t\t\tdie(\"unexpected EOF while expecting pathname\");\n+\t\t\tfprintf(logfile, \" %s\", pathname);\n+\n+\t\t\t/* Read until flush */\n+\t\t\tbuf = packet_read_line(0, &size);\n+\t\t\twhile (buf) {\n+\t\t\t\tif (!strcmp(buf, \"can-delay=1\")) {\n+\t\t\t\t\tentry = strmap_get(&delay, pathname);\n+\t\t\t\t\tif (entry && !entry->requested) {\n+\t\t\t\t\t\tentry->requested = 1;\n+\t\t\t\t\t} else if (!entry && always_delay) {\n+\t\t\t\t\t\tentry = xcalloc(1, sizeof(*entry));\n+\t\t\t\t\t\tentry->requested = 1;\n+\t\t\t\t\t\tentry->count = 1;\n+\t\t\t\t\t\tstrmap_put(&delay, pathname, entry);\n+\t\t\t\t\t}\n+\t\t\t\t} else if (starts_with(buf, \"ref=\") ||\n+\t\t\t\t\t   starts_with(buf, \"treeish=\") ||\n+\t\t\t\t\t   starts_with(buf, \"blob=\")) {\n+\t\t\t\t\tfprintf(logfile, \" %s\", buf);\n+\t\t\t\t} else {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * In general, filters need to be graceful about\n+\t\t\t\t\t * new metadata, since it's documented that we\n+\t\t\t\t\t * can pass any key-value pairs, but for tests,\n+\t\t\t\t\t * let's be a little stricter.\n+\t\t\t\t\t */\n+\t\t\t\t\tdie(\"Unknown message '%s'\", buf);\n+\t\t\t\t}\n+\t\t\t\tbuf = packet_read_line(0, &size);\n+\t\t\t}\n+\n+\n+\t\t\tread_packetized_to_strbuf(0, &input, 0);\n+\t\t\tfprintf(logfile, \" %\"PRIuMAX\" [OK] -- \", (uintmax_t)input.len);\n+\n+\t\t\tentry = strmap_get(&delay, pathname);\n+\t\t\tif (entry && entry->output) {\n+\t\t\t\toutput = entry->output;\n+\t\t\t} else if (!strcmp(pathname, \"error.r\") || !strcmp(pathname, \"abort.r\")) {\n+\t\t\t\toutput = \"\";\n+\t\t\t} else if (!strcmp(command, \"clean\") && has_capability(\"clean\")) {\n+\t\t\t\toutput = rot13(input.buf);\n+\t\t\t} else if (!strcmp(command, \"smudge\") && has_capability(\"smudge\")) {\n+\t\t\t\toutput = rot13(input.buf);\n+\t\t\t} else {\n+\t\t\t\tdie(\"bad command '%s'\", command);\n+\t\t\t}\n+\n+\t\t\tif (!strcmp(pathname, \"error.r\")) {\n+\t\t\t\tfprintf(logfile, \"[ERROR]\\n\");\n+\t\t\t\tpacket_write_fmt(1, \"status=error\");\n+\t\t\t\tpacket_flush(1);\n+\t\t\t} else if (!strcmp(pathname, \"abort.r\")) {\n+\t\t\t\tfprintf(logfile, \"[ABORT]\\n\");\n+\t\t\t\tpacket_write_fmt(1, \"status=abort\");\n+\t\t\t\tpacket_flush(1);\n+\t\t\t} else if (!strcmp(command, \"smudge\") &&\n+\t\t\t\t   (entry = strmap_get(&delay, pathname)) &&\n+\t\t\t\t   entry->requested == 1) {\n+\t\t\t\tfprintf(logfile, \"[DELAYED]\\n\");\n+\t\t\t\tpacket_write_fmt(1, \"status=delayed\");\n+\t\t\t\tpacket_flush(1);\n+\t\t\t\tentry->requested = 2;\n+\t\t\t\tentry->output = xstrdup(output);\n+\t\t\t} else {\n+\t\t\t\tint i, nr_packets;\n+\t\t\t\tsize_t output_len;\n+\t\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\t\tpacket_write_fmt(1, \"status=success\");\n+\t\t\t\tpacket_flush(1);\n+\n+\t\t\t\tstrbuf_addf(&sb, \"%s-write-fail.r\", command);\n+\t\t\t\tif (!strcmp(pathname, sb.buf)) {\n+\t\t\t\t\tfprintf(logfile, \"[WRITE FAIL]\\n\");\n+\t\t\t\t\tdie(\"%s write error\", command);\n+\t\t\t\t}\n+\n+\t\t\t\toutput_len = strlen(output);\n+\t\t\t\tfprintf(logfile, \"OUT: %\"PRIuMAX\" \", (uintmax_t)output_len);\n+\n+\t\t\t\tif (write_packetized_from_buf_no_flush_count(output,\n+\t\t\t\t\toutput_len, 1, &nr_packets))\n+\t\t\t\t\tdie(\"failed to write buffer to stdout\");\n+\t\t\t\tpacket_flush(1);\n+\n+\t\t\t\tfor (i = 0; i < nr_packets; i++)\n+\t\t\t\t\tfprintf(logfile, \".\");\n+\t\t\t\tfprintf(logfile, \" [OK]\\n\");\n+\n+\t\t\t\tpacket_flush(1);\n+\t\t\t\tstrbuf_release(&sb);\n+\t\t\t}\n+\t\t\tfree(pathname);\n+\t\t\tstrbuf_release(&input);\n+\t\t}\n+\t\tfree(command);\n+\t}\n+}\n+\n+static void free_delay_hash(void)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *ent;\n+\n+\tstrmap_for_each_entry(&delay, &iter, ent) {\n+\t\tstruct delay_entry *delay_entry = ent->value;\n+\t\tfree(delay_entry->output);\n+\t\tfree(delay_entry);\n+\t}\n+\tstrmap_clear(&delay, 0);\n+}\n+\n+static void add_delay_entry(char *pathname, int count)\n+{\n+\tstruct delay_entry *entry = xcalloc(1, sizeof(*entry));\n+\tentry->count = count;\n+\tif (strmap_put(&delay, pathname, entry))\n+\t\tBUG(\"adding the same path twice to delay hash?\");\n+}\n+\n+static void packet_initialize(const char *name, int version)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint size;\n+\tchar *pkt_buf = packet_read_line(0, &size);\n+\n+\tstrbuf_addf(&sb, \"%s-client\", name);\n+\tif (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n+\t\tdie(\"bad initialize: '%s'\", xstrndup(pkt_buf, size));\n+\n+\tstrbuf_reset(&sb);\n+\tstrbuf_addf(&sb, \"version=%d\", version);\n+\tpkt_buf = packet_read_line(0, &size);\n+\tif (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n+\t\tdie(\"bad version: '%s'\", xstrndup(pkt_buf, size));\n+\n+\tpkt_buf = packet_read_line(0, &size);\n+\tif (pkt_buf)\n+\t\tdie(\"bad version end: '%s'\", xstrndup(pkt_buf, size));\n+\n+\tpacket_write_fmt(1, \"%s-server\", name);\n+\tpacket_write_fmt(1, \"version=%d\", version);\n+\tpacket_flush(1);\n+\tstrbuf_release(&sb);\n+}\n+\n+static char *rot13_usage = \"test-tool rot13-filter [--always-delay] <log path> <capabilities>\";\n+\n+int cmd__rot13_filter(int argc, const char **argv)\n+{\n+\tint i = 1;\n+\tstruct string_list remote_caps = STRING_LIST_INIT_DUP,\n+\t\t\t   supported_caps = STRING_LIST_INIT_NODUP;\n+\n+\tstring_list_append(&supported_caps, \"clean\");\n+\tstring_list_append(&supported_caps, \"smudge\");\n+\tstring_list_append(&supported_caps, \"delay\");\n+\n+\tif (argc > 1 && !strcmp(argv[i], \"--always-delay\")) {\n+\t\talways_delay = 1;\n+\t\ti++;\n+\t}\n+\tif (argc - i < 2)\n+\t\tusage(rot13_usage);\n+\n+\tlogfile = fopen(argv[i++], \"a\");\n+\tif (!logfile)\n+\t\tdie_errno(\"failed to open log file\");\n+\n+\tfor ( ; i < argc; i++)\n+\t\tstring_list_append(&requested_caps, argv[i]);\n+\n+\tadd_delay_entry(\"test-delay10.a\", 1);\n+\tadd_delay_entry(\"test-delay11.a\", 1);\n+\tadd_delay_entry(\"test-delay20.a\", 2);\n+\tadd_delay_entry(\"test-delay10.b\", 1);\n+\tadd_delay_entry(\"missing-delay.a\", 1);\n+\tadd_delay_entry(\"invalid-delay.a\", 1);\n+\n+\tfprintf(logfile, \"START\\n\");\n+\n+\tpacket_initialize(\"git-filter\", 2);\n+\n+\tpacket_read_and_check_capabilities(&remote_caps, &supported_caps);\n+\tpacket_check_and_write_capabilities(&remote_caps, &requested_caps);\n+\tfprintf(logfile, \"init handshake complete\\n\");\n+\n+\tstring_list_clear(&supported_caps, 0);\n+\tstring_list_clear(&remote_caps, 0);\n+\n+\tcommand_loop();\n+\n+\tfclose(logfile);\n+\tstring_list_clear(&requested_caps, 0);\n+\tfree_delay_hash();\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 318fdbab0c..d6a560f832 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -65,6 +65,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"rot13-filter\", cmd__rot13_filter },\n \t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex bb79927163..21a91b1019 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -54,6 +54,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__rot13_filter(int argc, const char **argv);\n int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 1c840348bd..aeaa8e02ed 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -17,9 +17,6 @@ tr \\\n   'nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM'\n EOF\n \n-write_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl\n-\n generate_random_characters () {\n \tLEN=$1\n \tNAME=$2\n@@ -365,8 +362,8 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '\n \ttest_line_count = 0 count\n '\n \n-test_expect_success PERL 'required process filter should filter data' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should filter data' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -450,8 +447,8 @@ test_expect_success PERL 'required process filter should filter data' '\n \t)\n '\n \n-test_expect_success PERL 'required process filter should filter data for various subcommands' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should filter data for various subcommands' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \t(\n \t\tcd repo &&\n@@ -561,9 +558,9 @@ test_expect_success PERL 'required process filter should filter data for various\n \t)\n '\n \n-test_expect_success PERL 'required process filter takes precedence' '\n+test_expect_success 'required process filter takes precedence' '\n \ttest_config_global filter.protocol.clean false &&\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -587,8 +584,8 @@ test_expect_success PERL 'required process filter takes precedence' '\n \t)\n '\n \n-test_expect_success PERL 'required process filter should be used only for \"clean\" operation only' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n+test_expect_success 'required process filter should be used only for \"clean\" operation only' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -622,8 +619,8 @@ test_expect_success PERL 'required process filter should be used only for \"clean\n \t)\n '\n \n-test_expect_success PERL 'required process filter should process multiple packets' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should process multiple packets' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \n \trm -rf repo &&\n@@ -687,8 +684,8 @@ test_expect_success PERL 'required process filter should process multiple packet\n \t)\n '\n \n-test_expect_success PERL 'required process filter with clean error should fail' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter with clean error should fail' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -706,8 +703,8 @@ test_expect_success PERL 'required process filter with clean error should fail'\n \t)\n '\n \n-test_expect_success PERL 'process filter should restart after unexpected write failure' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter should restart after unexpected write failure' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -735,7 +732,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\trm -f debug.log &&\n \t\tgit checkout --quiet --no-progress . 2>git-stderr.log &&\n \n-\t\tgrep \"smudge write error at\" git-stderr.log &&\n+\t\tgrep \"smudge write error\" git-stderr.log &&\n \t\ttest_i18ngrep \"error: external filter\" git-stderr.log &&\n \n \t\tcat >expected.log <<-EOF &&\n@@ -761,8 +758,8 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t)\n '\n \n-test_expect_success PERL 'process filter should not be restarted if it signals an error' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter should not be restarted if it signals an error' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -804,8 +801,8 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n \t)\n '\n \n-test_expect_success PERL 'process filter abort stops processing of all further files' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter abort stops processing of all further files' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -861,10 +858,10 @@ test_expect_success PERL 'invalid process filter must fail (and not hang!)' '\n \t)\n '\n \n-test_expect_success PERL 'delayed checkout in process filter' '\n-\ttest_config_global filter.a.process \"rot13-filter.pl a.log clean smudge delay\" &&\n+test_expect_success 'delayed checkout in process filter' '\n+\ttest_config_global filter.a.process \"test-tool rot13-filter a.log clean smudge delay\" &&\n \ttest_config_global filter.a.required true &&\n-\ttest_config_global filter.b.process \"rot13-filter.pl b.log clean smudge delay\" &&\n+\ttest_config_global filter.b.process \"test-tool rot13-filter b.log clean smudge delay\" &&\n \ttest_config_global filter.b.required true &&\n \n \trm -rf repo &&\n@@ -940,8 +937,8 @@ test_expect_success PERL 'delayed checkout in process filter' '\n \t)\n '\n \n-test_expect_success PERL 'missing file in delayed checkout' '\n-\ttest_config_global filter.bug.process \"rot13-filter.pl bug.log clean smudge delay\" &&\n+test_expect_success 'missing file in delayed checkout' '\n+\ttest_config_global filter.bug.process \"test-tool rot13-filter bug.log clean smudge delay\" &&\n \ttest_config_global filter.bug.required true &&\n \n \trm -rf repo &&\n@@ -960,8 +957,8 @@ test_expect_success PERL 'missing file in delayed checkout' '\n \tgrep \"error: .missing-delay\\.a. was not filtered properly\" git-stderr.log\n '\n \n-test_expect_success PERL 'invalid file in delayed checkout' '\n-\ttest_config_global filter.bug.process \"rot13-filter.pl bug.log clean smudge delay\" &&\n+test_expect_success 'invalid file in delayed checkout' '\n+\ttest_config_global filter.bug.process \"test-tool rot13-filter bug.log clean smudge delay\" &&\n \ttest_config_global filter.bug.required true &&\n \n \trm -rf repo &&\n@@ -990,10 +987,10 @@ do\n \t\tmode_prereq='UTF8_NFD_TO_NFC' ;;\n \tesac\n \n-\ttest_expect_success PERL,SYMLINKS,$mode_prereq \\\n+\ttest_expect_success SYMLINKS,$mode_prereq \\\n \t\"delayed checkout with $mode-collision don't write to the wrong place\" '\n \t\ttest_config_global filter.delay.process \\\n-\t\t\t\"\\\"$TEST_ROOT/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\t\t\"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \t\ttest_config_global filter.delay.required true &&\n \n \t\tgit init $mode-collision &&\n@@ -1026,12 +1023,12 @@ do\n \t'\n done\n \n-test_expect_success PERL,SYMLINKS,CASE_INSENSITIVE_FS \\\n+test_expect_success SYMLINKS,CASE_INSENSITIVE_FS \\\n \"delayed checkout with submodule collision don't write to the wrong place\" '\n \tgit init collision-with-submodule &&\n \t(\n \t\tcd collision-with-submodule &&\n-\t\tgit config filter.delay.process \"\\\"$TEST_ROOT/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\t# We need Git to treat the submodule \"a\" and the\n@@ -1062,11 +1059,11 @@ test_expect_success PERL,SYMLINKS,CASE_INSENSITIVE_FS \\\n \t)\n '\n \n-test_expect_success PERL 'setup for progress tests' '\n+test_expect_success 'setup for progress tests' '\n \tgit init progress &&\n \t(\n \t\tcd progress &&\n-\t\tgit config filter.delay.process \"rot13-filter.pl delay-progress.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter delay-progress.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\techo \"*.a filter=delay\" >.gitattributes &&\n@@ -1132,12 +1129,12 @@ do\n \t'\n done\n \n-test_expect_success PERL 'delayed checkout correctly reports the number of updated entries' '\n+test_expect_success 'delayed checkout correctly reports the number of updated entries' '\n \trm -rf repo &&\n \tgit init repo &&\n \t(\n \t\tcd repo &&\n-\t\tgit config filter.delay.process \"../rot13-filter.pl delayed.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter delayed.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\techo \"*.a filter=delay\" >.gitattributes &&\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\ndeleted file mode 100644\nindex 7bb93768f3..0000000000\n--- a/t/t0021/rot13-filter.pl\n+++ /dev/null\n@@ -1,247 +0,0 @@\n-#\n-# Example implementation for the Git filter protocol version 2\n-# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n-#\n-# Usage: rot13-filter.pl [--always-delay] <log path> <capabilities>\n-#\n-# Log path defines a debug log file that the script writes to. The\n-# subsequent arguments define a list of supported protocol capabilities\n-# (\"clean\", \"smudge\", etc).\n-#\n-# When --always-delay is given all pathnames with the \"can-delay\" flag\n-# that don't appear on the list bellow are delayed with a count of 1\n-# (see more below).\n-#\n-# This implementation supports special test cases:\n-# (1) If data with the pathname \"clean-write-fail.r\" is processed with\n-#     a \"clean\" operation then the write operation will die.\n-# (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n-#     a \"smudge\" operation then the write operation will die.\n-# (3) If data with the pathname \"error.r\" is processed with any\n-#     operation then the filter signals that it cannot or does not want\n-#     to process the file.\n-# (4) If data with the pathname \"abort.r\" is processed with any\n-#     operation then the filter signals that it cannot or does not want\n-#     to process the file and any file after that is processed with the\n-#     same command.\n-# (5) If data with a pathname that is a key in the DELAY hash is\n-#     requested (e.g. \"test-delay10.a\") then the filter responds with\n-#     a \"delay\" status and sets the \"requested\" field in the DELAY hash.\n-#     The filter will signal the availability of this object after\n-#     \"count\" (field in DELAY hash) \"list_available_blobs\" commands.\n-# (6) If data with the pathname \"missing-delay.a\" is processed that the\n-#     filter will drop the path from the \"list_available_blobs\" response.\n-# (7) If data with the pathname \"invalid-delay.a\" is processed that the\n-#     filter will add the path \"unfiltered\" which was not delayed before\n-#     to the \"list_available_blobs\" response.\n-#\n-\n-use 5.008;\n-sub gitperllib {\n-\t# Git assumes that all path lists are Unix-y colon-separated ones. But\n-\t# when the Git for Windows executes the test suite, its MSYS2 Bash\n-\t# calls git.exe, and colon-separated path lists are converted into\n-\t# Windows-y semicolon-separated lists of *Windows* paths (which\n-\t# naturally contain a colon after the drive letter, so splitting by\n-\t# colons simply does not cut it).\n-\t#\n-\t# Detect semicolon-separated path list and handle them appropriately.\n-\n-\tif ($ENV{GITPERLLIB} =~ /;/) {\n-\t\treturn split(/;/, $ENV{GITPERLLIB});\n-\t}\n-\treturn split(/:/, $ENV{GITPERLLIB});\n-}\n-use lib (gitperllib());\n-use strict;\n-use warnings;\n-use IO::File;\n-use Git::Packet;\n-\n-my $MAX_PACKET_CONTENT_SIZE = 65516;\n-\n-my $always_delay = 0;\n-if ( $ARGV[0] eq '--always-delay' ) {\n-\t$always_delay = 1;\n-\tshift @ARGV;\n-}\n-\n-my $log_file                = shift @ARGV;\n-my @capabilities            = @ARGV;\n-\n-open my $debug, \">>\", $log_file or die \"cannot open log file: $!\";\n-\n-my %DELAY = (\n-\t'test-delay10.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'test-delay11.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'test-delay20.a' => { \"requested\" => 0, \"count\" => 2 },\n-\t'test-delay10.b' => { \"requested\" => 0, \"count\" => 1 },\n-\t'missing-delay.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'invalid-delay.a' => { \"requested\" => 0, \"count\" => 1 },\n-);\n-\n-sub rot13 {\n-\tmy $str = shift;\n-\t$str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n-\treturn $str;\n-}\n-\n-print $debug \"START\\n\";\n-$debug->flush();\n-\n-packet_initialize(\"git-filter\", 2);\n-\n-my %remote_caps = packet_read_and_check_capabilities(\"clean\", \"smudge\", \"delay\");\n-packet_check_and_write_capabilities(\\%remote_caps, @capabilities);\n-\n-print $debug \"init handshake complete\\n\";\n-$debug->flush();\n-\n-while (1) {\n-\tmy ( $res, $command ) = packet_key_val_read(\"command\");\n-\tif ( $res == -1 ) {\n-\t\tprint $debug \"STOP\\n\";\n-\t\texit();\n-\t}\n-\tprint $debug \"IN: $command\";\n-\t$debug->flush();\n-\n-\tif ( $command eq \"list_available_blobs\" ) {\n-\t\t# Flush\n-\t\tpacket_compare_lists([1, \"\"], packet_bin_read()) ||\n-\t\t\tdie \"bad list_available_blobs end\";\n-\n-\t\tforeach my $pathname ( sort keys %DELAY ) {\n-\t\t\tif ( $DELAY{$pathname}{\"requested\"} >= 1 ) {\n-\t\t\t\t$DELAY{$pathname}{\"count\"} = $DELAY{$pathname}{\"count\"} - 1;\n-\t\t\t\tif ( $pathname eq \"invalid-delay.a\" ) {\n-\t\t\t\t\t# Send Git a pathname that was not delayed earlier\n-\t\t\t\t\tpacket_txt_write(\"pathname=unfiltered\");\n-\t\t\t\t}\n-\t\t\t\tif ( $pathname eq \"missing-delay.a\" ) {\n-\t\t\t\t\t# Do not signal Git that this file is available\n-\t\t\t\t} elsif ( $DELAY{$pathname}{\"count\"} == 0 ) {\n-\t\t\t\t\tprint $debug \" $pathname\";\n-\t\t\t\t\tpacket_txt_write(\"pathname=$pathname\");\n-\t\t\t\t}\n-\t\t\t}\n-\t\t}\n-\n-\t\tpacket_flush();\n-\n-\t\tprint $debug \" [OK]\\n\";\n-\t\t$debug->flush();\n-\t\tpacket_txt_write(\"status=success\");\n-\t\tpacket_flush();\n-\t} else {\n-\t\tmy ( $res, $pathname ) = packet_key_val_read(\"pathname\");\n-\t\tif ( $res == -1 ) {\n-\t\t\tdie \"unexpected EOF while expecting pathname\";\n-\t\t}\n-\t\tprint $debug \" $pathname\";\n-\t\t$debug->flush();\n-\n-\t\t# Read until flush\n-\t\tmy ( $done, $buffer ) = packet_txt_read();\n-\t\twhile ( $buffer ne '' ) {\n-\t\t\tif ( $buffer eq \"can-delay=1\" ) {\n-\t\t\t\tif ( exists $DELAY{$pathname} and $DELAY{$pathname}{\"requested\"} == 0 ) {\n-\t\t\t\t\t$DELAY{$pathname}{\"requested\"} = 1;\n-\t\t\t\t} elsif ( !exists $DELAY{$pathname} and $always_delay ) {\n-\t\t\t\t\t$DELAY{$pathname} = { \"requested\" => 1, \"count\" => 1 };\n-\t\t\t\t}\n-\t\t\t} elsif ($buffer =~ /^(ref|treeish|blob)=/) {\n-\t\t\t\tprint $debug \" $buffer\";\n-\t\t\t} else {\n-\t\t\t\t# In general, filters need to be graceful about\n-\t\t\t\t# new metadata, since it's documented that we\n-\t\t\t\t# can pass any key-value pairs, but for tests,\n-\t\t\t\t# let's be a little stricter.\n-\t\t\t\tdie \"Unknown message '$buffer'\";\n-\t\t\t}\n-\n-\t\t\t( $done, $buffer ) = packet_txt_read();\n-\t\t}\n-\t\tif ( $done == -1 ) {\n-\t\t\tdie \"unexpected EOF after pathname '$pathname'\";\n-\t\t}\n-\n-\t\tmy $input = \"\";\n-\t\t{\n-\t\t\tbinmode(STDIN);\n-\t\t\tmy $buffer;\n-\t\t\tmy $done = 0;\n-\t\t\twhile ( !$done ) {\n-\t\t\t\t( $done, $buffer ) = packet_bin_read();\n-\t\t\t\t$input .= $buffer;\n-\t\t\t}\n-\t\t\tif ( $done == -1 ) {\n-\t\t\t\tdie \"unexpected EOF while reading input for '$pathname'\";\n-\t\t\t}\t\t\t\n-\t\t\tprint $debug \" \" . length($input) . \" [OK] -- \";\n-\t\t\t$debug->flush();\n-\t\t}\n-\n-\t\tmy $output;\n-\t\tif ( exists $DELAY{$pathname} and exists $DELAY{$pathname}{\"output\"} ) {\n-\t\t\t$output = $DELAY{$pathname}{\"output\"}\n-\t\t} elsif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n-\t\t\t$output = \"\";\n-\t\t} elsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n-\t\t\t$output = rot13($input);\n-\t\t} elsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n-\t\t\t$output = rot13($input);\n-\t\t} else {\n-\t\t\tdie \"bad command '$command'\";\n-\t\t}\n-\n-\t\tif ( $pathname eq \"error.r\" ) {\n-\t\t\tprint $debug \"[ERROR]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=error\");\n-\t\t\tpacket_flush();\n-\t\t} elsif ( $pathname eq \"abort.r\" ) {\n-\t\t\tprint $debug \"[ABORT]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=abort\");\n-\t\t\tpacket_flush();\n-\t\t} elsif ( $command eq \"smudge\" and\n-\t\t\texists $DELAY{$pathname} and\n-\t\t\t$DELAY{$pathname}{\"requested\"} == 1 ) {\n-\t\t\tprint $debug \"[DELAYED]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=delayed\");\n-\t\t\tpacket_flush();\n-\t\t\t$DELAY{$pathname}{\"requested\"} = 2;\n-\t\t\t$DELAY{$pathname}{\"output\"} = $output;\n-\t\t} else {\n-\t\t\tpacket_txt_write(\"status=success\");\n-\t\t\tpacket_flush();\n-\n-\t\t\tif ( $pathname eq \"${command}-write-fail.r\" ) {\n-\t\t\t\tprint $debug \"[WRITE FAIL]\\n\";\n-\t\t\t\t$debug->flush();\n-\t\t\t\tdie \"${command} write error\";\n-\t\t\t}\n-\n-\t\t\tprint $debug \"OUT: \" . length($output) . \" \";\n-\t\t\t$debug->flush();\n-\n-\t\t\twhile ( length($output) > 0 ) {\n-\t\t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\t\tpacket_bin_write($packet);\n-\t\t\t\t# dots represent the number of packets\n-\t\t\t\tprint $debug \".\";\n-\t\t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n-\t\t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\t\t} else {\n-\t\t\t\t\t$output = \"\";\n-\t\t\t\t}\n-\t\t\t}\n-\t\t\tpacket_flush();\n-\t\t\tprint $debug \" [OK]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_flush();\n-\t\t}\n-\t}\n-}\ndiff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh\nindex c683e60007..7d956625ca 100755\n--- a/t/t2080-parallel-checkout-basics.sh\n+++ b/t/t2080-parallel-checkout-basics.sh\n@@ -230,12 +230,9 @@ test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading d\n # check the final report including sequential, parallel, and delayed entries\n # all at the same time. So we must have finer control of the parallel checkout\n # variables.\n-test_expect_success PERL '\"git checkout .\" report should not include failed entries' '\n-\twrite_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl &&\n-\n+test_expect_success '\"git checkout .\" report should not include failed entries' '\n \ttest_config_global filter.delay.process \\\n-\t\t\"\\\"$(pwd)/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\t\"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \ttest_config_global filter.delay.required true &&\n \ttest_config_global filter.cat.clean cat  &&\n \ttest_config_global filter.cat.smudge cat  &&\ndiff --git a/t/t2082-parallel-checkout-attributes.sh b/t/t2082-parallel-checkout-attributes.sh\nindex 2525457961..2df55b9405 100755\n--- a/t/t2082-parallel-checkout-attributes.sh\n+++ b/t/t2082-parallel-checkout-attributes.sh\n@@ -138,12 +138,9 @@ test_expect_success 'parallel-checkout and external filter' '\n # The delayed queue is independent from the parallel queue, and they should be\n # able to work together in the same checkout process.\n #\n-test_expect_success PERL 'parallel-checkout and delayed checkout' '\n-\twrite_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl &&\n-\n+test_expect_success 'parallel-checkout and delayed checkout' '\n \ttest_config_global filter.delay.process \\\n-\t\t\"\\\"$(pwd)/rot13-filter.pl\\\" --always-delay \\\"$(pwd)/delayed.log\\\" clean smudge delay\" &&\n+\t\t\"test-tool rot13-filter --always-delay \\\"$(pwd)/delayed.log\\\" clean smudge delay\" &&\n \ttest_config_global filter.delay.required true &&\n \n \techo \"abcd\" >original &&\n-- \n2.37.1\n\n"},{"id":"460128","messageId":"4n20476q-6ssr-osp8-q5o3-p8ns726q4pn3@tzk.qr","threadId":"58212","inReplyTo":"f38f722de7c3323207eda5ea632b5acd3765c285.1658675222.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-07-28T16:58:05Z","receivedAt":"2022-07-28T16:58:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Sun, 24 Jul 2022, Matheus Tavares wrote:\n\n> This script is currently used by three test files: t0021-conversion.sh,\n> t2080-parallel-checkout-basics.sh, and\n> t2082-parallel-checkout-attributes.sh. To avoid the need for the PERL\n> dependency at these tests, let's convert the script to a C test-tool\n> command.\n\nGreat!\n\n>  - Squashed the two patches together.\n\nI see why this might have been suggested, but it definitely made it more\nchallenging for me to review. You see, it is easy to just fly over a patch\nthat simply removes the `PERL` prereq, but it is much harder to jump back\nand forth over all of these removals when the `.c` version of the filter\nis added before them and the `.pl` version is removed after them. So I\nfind that it was bad advice, but I do not fault you for following it (we\nall want reviews to just be over already and therefore sometimes pander to\nthe reviewers, no matter how much or little sense their feedback makes).\n\nIt just would have been easier for me to review if the chaff was separated\nfrom the wheat, so to say.\n\nTo illustrate my point: it was a bit of a challenge to find the adjustment\nof the \"smudge write error at\" needle in all of that cruft. It would have\nmade my life as a reviewer substantially easier had the patch\nseries been organized this way (which I assume you had before the feedback\nyou received demanded to squash everything in one hot pile):\n\n\t1/3 adjust the needle for the error message\n\t2/3 implement the rot13-filter in C\n\t3/3 use the test-tool in the tests and remove the PERL prereq, and\n\t    remove rot13-filter.pl\n\n> [...]\n> diff --git a/pkt-line.c b/pkt-line.c\n> index 8e43c2def4..4425bdae36 100644\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -309,9 +309,10 @@ int write_packetized_from_fd_no_flush(int fd_in, int fd_out)\n>  \treturn err;\n>  }\n>\n> -int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n> +int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n> +\t\t\t\t\t     int fd_out, int *count_ptr)\n>  {\n> -\tint err = 0;\n> +\tint err = 0, count = 0;\n>  \tsize_t bytes_written = 0;\n>  \tsize_t bytes_to_write;\n>\n> @@ -324,10 +325,18 @@ int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_ou\n>  \t\t\tbreak;\n>  \t\terr = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);\n>  \t\tbytes_written += bytes_to_write;\n> +\t\tcount++;\n>  \t}\n> +\tif (count_ptr)\n> +\t\t*count_ptr = count;\n\nThis is not just a counter, but a packet counter, right? In any case, it\nwould probably make more sense to increment the value directly:\n\n\t\tif (count_ptr)\n\t\t\t(*count_ptr)++;\n\nMore on that below, where you use it.\n\n>  \treturn err;\n>  }\n>\n> +int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n> +{\n> +\treturn write_packetized_from_buf_no_flush_count(src_in, len, fd_out, NULL);\n> +}\n\nHave you considered making this a `static inline` in `pkt-line.h`?\n\n> [...]\n> diff --git a/t/helper/test-rot13-filter.c b/t/helper/test-rot13-filter.c\n> new file mode 100644\n> index 0000000000..536111f272\n> --- /dev/null\n> +++ b/t/helper/test-rot13-filter.c\n> @@ -0,0 +1,393 @@\n> +/*\n> + * Example implementation for the Git filter protocol version 2\n> + * See Documentation/gitattributes.txt, section \"Filter Protocol\"\n> + *\n> + * Usage: test-tool rot13-filter [--always-delay] <log path> <capabilities>\n> + *\n> + * Log path defines a debug log file that the script writes to. The\n> + * subsequent arguments define a list of supported protocol capabilities\n> + * (\"clean\", \"smudge\", etc).\n> + *\n> + * When --always-delay is given all pathnames with the \"can-delay\" flag\n> + * that don't appear on the list bellow are delayed with a count of 1\n> + * (see more below).\n> + *\n> + * This implementation supports special test cases:\n> + * (1) If data with the pathname \"clean-write-fail.r\" is processed with\n> + *     a \"clean\" operation then the write operation will die.\n> + * (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n> + *     a \"smudge\" operation then the write operation will die.\n> + * (3) If data with the pathname \"error.r\" is processed with any\n> + *     operation then the filter signals that it cannot or does not want\n> + *     to process the file.\n> + * (4) If data with the pathname \"abort.r\" is processed with any\n> + *     operation then the filter signals that it cannot or does not want\n> + *     to process the file and any file after that is processed with the\n> + *     same command.\n> + * (5) If data with a pathname that is a key in the delay hash is\n> + *     requested (e.g. \"test-delay10.a\") then the filter responds with\n> + *     a \"delay\" status and sets the \"requested\" field in the delay hash.\n> + *     The filter will signal the availability of this object after\n> + *     \"count\" (field in delay hash) \"list_available_blobs\" commands.\n> + * (6) If data with the pathname \"missing-delay.a\" is processed that the\n> + *     filter will drop the path from the \"list_available_blobs\" response.\n> + * (7) If data with the pathname \"invalid-delay.a\" is processed that the\n> + *     filter will add the path \"unfiltered\" which was not delayed before\n> + *     to the \"list_available_blobs\" response.\n> + */\n> +\n> +#include \"test-tool.h\"\n> +#include \"pkt-line.h\"\n> +#include \"string-list.h\"\n> +#include \"strmap.h\"\n> +\n> +static FILE *logfile;\n> +static int always_delay;\n> +static struct strmap delay = STRMAP_INIT;\n> +static struct string_list requested_caps = STRING_LIST_INIT_NODUP;\n> +\n> +static int has_capability(const char *cap)\n> +{\n> +\treturn unsorted_string_list_has_string(&requested_caps, cap);\n> +}\n> +\n> +static char *rot13(char *str)\n> +{\n> +\tchar *c;\n> +\tfor (c = str; *c; c++) {\n> +\t\tif (*c >= 'a' && *c <= 'z')\n> +\t\t\t*c = 'a' + (*c - 'a' + 13) % 26;\n> +\t\telse if (*c >= 'A' && *c <= 'Z')\n> +\t\t\t*c = 'A' + (*c - 'A' + 13) % 26;\n\nThat's quite verbose, but it _is_ correct (if a bit harder than necessary\nto validate, I admit that I had to look up whether `%`'s precedence is higher\nthan `+` in https://en.cppreference.com/w/c/language/operator_precedence).\n\nA conciser way (also easier to reason about):\n\n\tfor (c = str; *c; c++)\n\t\tif (isalpha(*c))\n\t\t\t*c += tolower(*c) < 'n' ? 13 : -13;\n\nFor fun, you could also look at\nhttps://hea-www.harvard.edu/~fine/Tech/rot13.html whether you want to use\nyet another approach.\n\n> +\t}\n> +\treturn str;\n> +}\n> +\n> +static char *skip_key_dup(const char *buf, size_t size, const char *key)\n> +{\n> +\tstruct strbuf keybuf = STRBUF_INIT;\n> +\tstrbuf_addf(&keybuf, \"%s=\", key);\n> +\tif (!skip_prefix_mem(buf, size, keybuf.buf, &buf, &size) || !size)\n> +\t\tdie(\"bad %s: '%s'\", key, xstrndup(buf, size));\n> +\tstrbuf_release(&keybuf);\n> +\treturn xstrndup(buf, size);\n\nThis does what we want it to do, but it looks as if it was code translated\nfrom a language that does not care one bit about allocations to a language\nthat cares a lot.\n\nFor example, instead of allocating a `strbuf` just to append `=` to the\nkey, in idiomatic C this code would read like this:\n\nstatic char *get_value(char *buf, size_t size, const char *key)\n{\n\tconst char *orig_buf = buf;\n\tint orig_size = (int)size;\n\n\tif (!skip_prefix_mem(buf, size, key, &buf, &size) ||\n\t    !skip_prefix_mem(buf, size, \"=\", &buf, &size) ||\n\t    !size)\n\t\tdie(\"expected key '%s', got '%.*s'\",\n\t\t    key, orig_size, orig_buf);\n\n\treturn xstrndup(buf, size);\n}\n\nI was tempted, even, to suggest returning a `const char *` after\nNUL-terminating the line (via `buf[size] = '\\0';`) instead of\n`xstrndup()`ing it, but `packet_read_line()` reads into the singleton\n`packet_buffer` and we use e.g. the `command` that is returned from this\nfunction after reading the next packet, so the command would most likely\nbe overwritten.\n\n> +}\n> +\n> +/*\n> + * Read a text packet, expecting that it is in the form \"key=value\" for\n> + * the given key. An EOF does not trigger any error and is reported\n> + * back to the caller with NULL. Die if the \"key\" part of \"key=value\" does\n> + * not match the given key, or the value part is empty.\n> + */\n> +static char *packet_key_val_read(const char *key)\n> +{\n> +\tint size;\n> +\tchar *buf;\n> +\tif (packet_read_line_gently(0, &size, &buf) < 0)\n> +\t\treturn NULL;\n> +\treturn skip_key_dup(buf, size, key);\n> +}\n> +\n> +static void packet_read_capabilities(struct string_list *caps)\n> +{\n> +\twhile (1) {\n\nIn Git's source code, I think we prefer `for (;;)`. But not by much:\n\n$ git grep 'while (1)' \\*.c | wc\n    128     508    3745\n\n$ git grep 'for (;;)' \\*.c | wc\n    156     614    4389\n\n> +\t\tint size;\n> +\t\tchar *buf = packet_read_line(0, &size);\n> +\t\tif (!buf)\n> +\t\t\tbreak;\n> +\t\tstring_list_append_nodup(caps,\n> +\t\t\t\t\t skip_key_dup(buf, size, \"capability\"));\n\nIt is tempting to use unsorted string lists for everything because Perl\nmakes that relatively easy.\n\nHowever, in this instance I would strongly recommend using something more\nakin to Perl's \"hash\" data structure, in this instance a `strset`.\n\n> +\t}\n> +}\n> +\n> +/* Read remote capabilities and check them against capabilities we require */\n> +static void packet_read_and_check_capabilities(struct string_list *remote_caps,\n> +\t\t\t\t\t       struct string_list *required_caps)\n> +{\n> +\tstruct string_list_item *item;\n> +\tpacket_read_capabilities(remote_caps);\n> +\tfor_each_string_list_item(item, required_caps) {\n> +\t\tif (!unsorted_string_list_has_string(remote_caps, item->string)) {\n> +\t\t\tdie(\"required '%s' capability not available from remote\",\n> +\t\t\t    item->string);\n> +\t\t}\n> +\t}\n> +}\n\nThis is a pretty literal translation from Perl to C, and a couple of years\nago, I would have done the same.\n\nHowever, these days I would recommend against it. In this instance, we are\nreally only interested in three capabilities: clean, smudge and delay. It\nis much, much simpler to read in the capabilities and then manually verify\nthat the three required ones were included:\n\nstatic void read_capabilities(struct strset *remote_caps)\n{\n\tchar *cap\n\twhile ((cap = packet_key_val_read(\"capability\")))\n\t\tstrset_add(remote_caps, cap);\n\n\tif (!strset_contains(remote_caps, \"clean\"))\n\t\tdie(\"required 'clean' capability not available from remote\");\n\tif (!strset_contains(remote_caps, \"smudge\"))\n\t\tdie(\"required 'smudge' capability not available from remote\");\n\tif (!strset_contains(remote_caps, \"delay\"))\n\t\tdie(\"required 'delay' capability not available from remote\");\n}\n\n> +\n> +/*\n> + * Check our capabilities we want to advertise against the remote ones\n> + * and then advertise our capabilities\n> + */\n> +static void packet_check_and_write_capabilities(struct string_list *remote_caps,\n> +\t\t\t\t\t\tstruct string_list *our_caps)\n\nThe list of \"our caps\" comes from the command-line. In C, this means we\nget a `const char **argv` and an `int argc`. So:\n\nstatic void check_and_write_capabilities(struct strset *remote_caps,\n\t\t\t\t\t const char **caps, int caps_count)\n{\n\tint i;\n\n\tfor (i = 0; i < caps_count; i++) {\n\t\tif (!strset_contains(remote_caps, caps[i]))\n\t\t\tdie(\"our capability '%s' is not available from remote\",\n\t\t\t    caps[i]);\n\n\t\tpacket_write_fmt(1, \"capability=%s\\n\", caps[i]);\n\t}\n\tpacket_flush(1);\n}\n\nAnd then we would call it via\n\n\tcheck_and_write_capabilities(remote_caps, argv + 1, argc - 1);\n\n> +\n> +struct delay_entry {\n> +\tint requested, count;\n> +\tchar *output;\n> +};\n\nSince you declare this here, it makes most sense to define\n`free_delay_hash()` (which should really be named `free_delay_entries()`)\nand `add_delay_entry()` here.\n\n> +\n> +static void command_loop(void)\n> +{\n> +\twhile (1) {\n> +\t\tchar *command = packet_key_val_read(\"command\");\n> +\t\tif (!command) {\n> +\t\t\tfprintf(logfile, \"STOP\\n\");\n> +\t\t\tbreak;\n> +\t\t}\n> +\t\tfprintf(logfile, \"IN: %s\", command);\n\nWe will also need to `fflush(logfile)` here, to imitate the Perl script's\nbehavior more precisely.\n\n> +\n> +\t\tif (!strcmp(command, \"list_available_blobs\")) {\n> +\t\t\tstruct hashmap_iter iter;\n> +\t\t\tstruct strmap_entry *ent;\n> +\t\t\tstruct string_list_item *str_item;\n> +\t\t\tstruct string_list paths = STRING_LIST_INIT_NODUP;\n> +\n> +\t\t\t/* flush */\n> +\t\t\tif (packet_read_line(0, NULL))\n> +\t\t\t\tdie(\"bad list_available_blobs end\");\n> +\n> +\t\t\tstrmap_for_each_entry(&delay, &iter, ent) {\n> +\t\t\t\tstruct delay_entry *delay_entry = ent->value;\n> +\t\t\t\tif (!delay_entry->requested)\n> +\t\t\t\t\tcontinue;\n> +\t\t\t\tdelay_entry->count--;\n> +\t\t\t\tif (!strcmp(ent->key, \"invalid-delay.a\")) {\n> +\t\t\t\t\t/* Send Git a pathname that was not delayed earlier */\n> +\t\t\t\t\tpacket_write_fmt(1, \"pathname=unfiltered\");\n> +\t\t\t\t}\n> +\t\t\t\tif (!strcmp(ent->key, \"missing-delay.a\")) {\n> +\t\t\t\t\t/* Do not signal Git that this file is available */\n> +\t\t\t\t} else if (!delay_entry->count) {\n> +\t\t\t\t\tstring_list_insert(&paths, ent->key);\n> +\t\t\t\t\tpacket_write_fmt(1, \"pathname=%s\", ent->key);\n> +\t\t\t\t}\n> +\t\t\t}\n> +\n> +\t\t\t/* Print paths in sorted order. */\n\nThe Perl script does not order them specifically. Do we really have to do\nthat here?\n\nIn any case, it is more performant to append the paths in an unsorted way\nand then sort them once in the end (that's O(N log(N)) instead of O(N^2)).\n\n> +\t\t\tfor_each_string_list_item(str_item, &paths)\n> +\t\t\t\tfprintf(logfile, \" %s\", str_item->string);\n> +\t\t\tstring_list_clear(&paths, 0);\n> +\n> +\t\t\tpacket_flush(1);\n> +\n> +\t\t\tfprintf(logfile, \" [OK]\\n\");\n> +\t\t\tpacket_write_fmt(1, \"status=success\");\n> +\t\t\tpacket_flush(1);\n\nI know the Perl script uses an else here, but I'd much rather insert a\n`continue` at the end of the `list_available_blobs` clause and de-indent\nthe remainder of the loop body.\n\n> +\t\t} else {\n> +\t\t\tchar *buf, *output;\n> +\t\t\tint size;\n> +\t\t\tchar *pathname;\n> +\t\t\tstruct delay_entry *entry;\n> +\t\t\tstruct strbuf input = STRBUF_INIT;\n> +\n> +\t\t\tpathname = packet_key_val_read(\"pathname\");\n> +\t\t\tif (!pathname)\n> +\t\t\t\tdie(\"unexpected EOF while expecting pathname\");\n> +\t\t\tfprintf(logfile, \" %s\", pathname);\n\nAgain, let's `fflush(logfile)` here.\n\n> +\n> +\t\t\t/* Read until flush */\n> +\t\t\tbuf = packet_read_line(0, &size);\n> +\t\t\twhile (buf) {\n\nLet's write this in more idiomatic C:\n\n\t\t\twhile ((buf = packet_read_line(0, &size))) {\n\n> +\t\t\t\tif (!strcmp(buf, \"can-delay=1\")) {\n> +\t\t\t\t\tentry = strmap_get(&delay, pathname);\n> +\t\t\t\t\tif (entry && !entry->requested) {\n> +\t\t\t\t\t\tentry->requested = 1;\n> +\t\t\t\t\t} else if (!entry && always_delay) {\n> +\t\t\t\t\t\tentry = xcalloc(1, sizeof(*entry));\n> +\t\t\t\t\t\tentry->requested = 1;\n> +\t\t\t\t\t\tentry->count = 1;\n> +\t\t\t\t\t\tstrmap_put(&delay, pathname, entry);\n\nI guess here is our chance to extend the signature of `add_delay_entry()`\nto accept a `requested` parameter, and to call that here.\n\n> +\t\t\t\t\t}\n> +\t\t\t\t} else if (starts_with(buf, \"ref=\") ||\n> +\t\t\t\t\t   starts_with(buf, \"treeish=\") ||\n> +\t\t\t\t\t   starts_with(buf, \"blob=\")) {\n> +\t\t\t\t\tfprintf(logfile, \" %s\", buf);\n> +\t\t\t\t} else {\n> +\t\t\t\t\t/*\n> +\t\t\t\t\t * In general, filters need to be graceful about\n> +\t\t\t\t\t * new metadata, since it's documented that we\n> +\t\t\t\t\t * can pass any key-value pairs, but for tests,\n> +\t\t\t\t\t * let's be a little stricter.\n> +\t\t\t\t\t */\n> +\t\t\t\t\tdie(\"Unknown message '%s'\", buf);\n> +\t\t\t\t}\n> +\t\t\t\tbuf = packet_read_line(0, &size);\n> +\t\t\t}\n> +\n> +\n> +\t\t\tread_packetized_to_strbuf(0, &input, 0);\n> +\t\t\tfprintf(logfile, \" %\"PRIuMAX\" [OK] -- \", (uintmax_t)input.len);\n\nThis reads _so much nicer_ than the Perl version!\n\n> +\n> +\t\t\tentry = strmap_get(&delay, pathname);\n> +\t\t\tif (entry && entry->output) {\n> +\t\t\t\toutput = entry->output;\n> +\t\t\t} else if (!strcmp(pathname, \"error.r\") || !strcmp(pathname, \"abort.r\")) {\n> +\t\t\t\toutput = \"\";\n> +\t\t\t} else if (!strcmp(command, \"clean\") && has_capability(\"clean\")) {\n> +\t\t\t\toutput = rot13(input.buf);\n> +\t\t\t} else if (!strcmp(command, \"smudge\") && has_capability(\"smudge\")) {\n> +\t\t\t\toutput = rot13(input.buf);\n> +\t\t\t} else {\n> +\t\t\t\tdie(\"bad command '%s'\", command);\n> +\t\t\t}\n> +\n> +\t\t\tif (!strcmp(pathname, \"error.r\")) {\n> +\t\t\t\tfprintf(logfile, \"[ERROR]\\n\");\n> +\t\t\t\tpacket_write_fmt(1, \"status=error\");\n> +\t\t\t\tpacket_flush(1);\n> +\t\t\t} else if (!strcmp(pathname, \"abort.r\")) {\n> +\t\t\t\tfprintf(logfile, \"[ABORT]\\n\");\n> +\t\t\t\tpacket_write_fmt(1, \"status=abort\");\n> +\t\t\t\tpacket_flush(1);\n> +\t\t\t} else if (!strcmp(command, \"smudge\") &&\n> +\t\t\t\t   (entry = strmap_get(&delay, pathname)) &&\n> +\t\t\t\t   entry->requested == 1) {\n> +\t\t\t\tfprintf(logfile, \"[DELAYED]\\n\");\n> +\t\t\t\tpacket_write_fmt(1, \"status=delayed\");\n> +\t\t\t\tpacket_flush(1);\n> +\t\t\t\tentry->requested = 2;\n> +\t\t\t\tentry->output = xstrdup(output);\n\nWe need to call `free(entry->output)` before that lest we leak memory, but\nonly if `output` is not identical anyway:\n\n\t\t\t\tif (entry->output != output) {\n\t\t\t\t\tfree(entry->output);\n\t\t\t\t\tentry->output = xstrdup(output);\n\t\t\t\t}\n\n\n> +\t\t\t} else {\n> +\t\t\t\tint i, nr_packets;\n> +\t\t\t\tsize_t output_len;\n> +\t\t\t\tstruct strbuf sb = STRBUF_INIT;\n> +\t\t\t\tpacket_write_fmt(1, \"status=success\");\n> +\t\t\t\tpacket_flush(1);\n> +\n> +\t\t\t\tstrbuf_addf(&sb, \"%s-write-fail.r\", command);\n> +\t\t\t\tif (!strcmp(pathname, sb.buf)) {\n\nWe can easily avoid allocating the string just for comparing it:\n\n\t\t\t\tconst char *p;\n\n\t\t\t\tif (skip_prefix(pathname, command, &p) &&\n\t\t\t\t    !strcmp(p, \"-write-fail.r\")) {\n\n> +\t\t\t\t\tfprintf(logfile, \"[WRITE FAIL]\\n\");\n\n\t\t\t\t\tfflush(logfile) ;-)\n\n> +\t\t\t\t\tdie(\"%s write error\", command);\n> +\t\t\t\t}\n> +\n> +\t\t\t\toutput_len = strlen(output);\n> +\t\t\t\tfprintf(logfile, \"OUT: %\"PRIuMAX\" \", (uintmax_t)output_len);\n> +\n> +\t\t\t\tif (write_packetized_from_buf_no_flush_count(output,\n> +\t\t\t\t\toutput_len, 1, &nr_packets))\n> +\t\t\t\t\tdie(\"failed to write buffer to stdout\");\n> +\t\t\t\tpacket_flush(1);\n> +\n> +\t\t\t\tfor (i = 0; i < nr_packets; i++)\n> +\t\t\t\t\tfprintf(logfile, \".\");\n\nThat's not quite the same as the Perl script does: it prints a '.'\n(without flushing, though) _every_ time it wrote a packet.\n\nIf you want to emulate that, you will have to copy/edit that loop (and in\nthat case, the insanely long-named function\n`write_packetized_from_buf_no_flush_count()` is unnecessary, too).\n\n> +\t\t\t\tfprintf(logfile, \" [OK]\\n\");\n> +\n> +\t\t\t\tpacket_flush(1);\n> +\t\t\t\tstrbuf_release(&sb);\n> +\t\t\t}\n> +\t\t\tfree(pathname);\n> +\t\t\tstrbuf_release(&input);\n> +\t\t}\n> +\t\tfree(command);\n> +\t}\n> +}\n> +\n> +static void free_delay_hash(void)\n> +{\n> +\tstruct hashmap_iter iter;\n> +\tstruct strmap_entry *ent;\n> +\n> +\tstrmap_for_each_entry(&delay, &iter, ent) {\n> +\t\tstruct delay_entry *delay_entry = ent->value;\n> +\t\tfree(delay_entry->output);\n> +\t\tfree(delay_entry);\n> +\t}\n> +\tstrmap_clear(&delay, 0);\n> +}\n> +\n> +static void add_delay_entry(char *pathname, int count)\n> +{\n> +\tstruct delay_entry *entry = xcalloc(1, sizeof(*entry));\n> +\tentry->count = count;\n> +\tif (strmap_put(&delay, pathname, entry))\n> +\t\tBUG(\"adding the same path twice to delay hash?\");\n> +}\n> +\n> +static void packet_initialize(const char *name, int version)\n> +{\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tint size;\n> +\tchar *pkt_buf = packet_read_line(0, &size);\n> +\n> +\tstrbuf_addf(&sb, \"%s-client\", name);\n> +\tif (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n\nWe do not need the flexibility of the Perl package, where `name` is a\nparameter. We can hard-code `git-filter-client` here. I.e. something like\nthis:\n\n\tif (!pkt_buf || size != 17 ||\n\t    strncmp(pkt_buf, \"git-filter-client\", 17))\n\n> +\t\tdie(\"bad initialize: '%s'\", xstrndup(pkt_buf, size));\n> +\n> +\tstrbuf_reset(&sb);\n> +\tstrbuf_addf(&sb, \"version=%d\", version);\n\nSame here. We do not need to allocate a string just to compare it to the\npacket's payload.\n\n> +\tpkt_buf = packet_read_line(0, &size);\n> +\tif (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n> +\t\tdie(\"bad version: '%s'\", xstrndup(pkt_buf, size));\n> +\n> +\tpkt_buf = packet_read_line(0, &size);\n> +\tif (pkt_buf)\n> +\t\tdie(\"bad version end: '%s'\", xstrndup(pkt_buf, size));\n> +\n> +\tpacket_write_fmt(1, \"%s-server\", name);\n> +\tpacket_write_fmt(1, \"version=%d\", version);\n> +\tpacket_flush(1);\n> +\tstrbuf_release(&sb);\n> +}\n> +\n> +static char *rot13_usage = \"test-tool rot13-filter [--always-delay] <log path> <capabilities>\";\n> +\n> +int cmd__rot13_filter(int argc, const char **argv)\n> +{\n> +\tint i = 1;\n> +\tstruct string_list remote_caps = STRING_LIST_INIT_DUP,\n> +\t\t\t   supported_caps = STRING_LIST_INIT_NODUP;\n> +\n> +\tstring_list_append(&supported_caps, \"clean\");\n> +\tstring_list_append(&supported_caps, \"smudge\");\n> +\tstring_list_append(&supported_caps, \"delay\");\n> +\n> +\tif (argc > 1 && !strcmp(argv[i], \"--always-delay\")) {\n> +\t\talways_delay = 1;\n> +\t\ti++;\n> +\t}\n> +\tif (argc - i < 2)\n> +\t\tusage(rot13_usage);\n> +\n> +\tlogfile = fopen(argv[i++], \"a\");\n> +\tif (!logfile)\n> +\t\tdie_errno(\"failed to open log file\");\n> +\n> +\tfor ( ; i < argc; i++)\n> +\t\tstring_list_append(&requested_caps, argv[i]);\n> +\n> +\tadd_delay_entry(\"test-delay10.a\", 1);\n> +\tadd_delay_entry(\"test-delay11.a\", 1);\n> +\tadd_delay_entry(\"test-delay20.a\", 2);\n> +\tadd_delay_entry(\"test-delay10.b\", 1);\n> +\tadd_delay_entry(\"missing-delay.a\", 1);\n> +\tadd_delay_entry(\"invalid-delay.a\", 1);\n> +\n> +\tfprintf(logfile, \"START\\n\");\n> +\n> +\tpacket_initialize(\"git-filter\", 2);\n> +\n> +\tpacket_read_and_check_capabilities(&remote_caps, &supported_caps);\n> +\tpacket_check_and_write_capabilities(&remote_caps, &requested_caps);\n> +\tfprintf(logfile, \"init handshake complete\\n\");\n> +\n> +\tstring_list_clear(&supported_caps, 0);\n> +\tstring_list_clear(&remote_caps, 0);\n> +\n> +\tcommand_loop();\n> +\n> +\tfclose(logfile);\n> +\tstring_list_clear(&requested_caps, 0);\n> +\tfree_delay_hash();\n> +\treturn 0;\n> +}\n\nOther than that, this looks great!\n\nThank you,\nDscho\n\n"},{"id":"460138","messageId":"xmqqlesdazsi.fsf@gitster.g","threadId":"58212","inReplyTo":"4n20476q-6ssr-osp8-q5o3-p8ns726q4pn3@tzk.qr","subject":"Re: [PATCH v2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-28T17:54:21Z","receivedAt":"2022-07-28T17:54:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n>>  - Squashed the two patches together.\n>\n> I see why this might have been suggested, but it definitely made it more\n> challenging for me to review. You see, it is easy to just fly over a patch\n> that simply removes the `PERL` prereq, but it is much harder to jump back\n> and forth over all of these removals when the `.c` version of the filter\n> is added before them and the `.pl` version is removed after them.\n\nYeah, I tend to agree.\n\n> ...\n> Other than that, this looks great!\n\nYup, thanks for an excellent review.\n"},{"id":"460146","messageId":"220728.86bkt9j8xk.gmgdl@evledraar.gmail.com","threadId":"58212","inReplyTo":"4n20476q-6ssr-osp8-q5o3-p8ns726q4pn3@tzk.qr","subject":"Re: [PATCH v2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-28T19:50:41Z","receivedAt":"2022-07-28T20:09:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 28 2022, Johannes Schindelin wrote:\n\n> [...]\n> I see why this might have been suggested, but it definitely made it more\n> challenging for me to review. You see, it is easy to just fly over a patch\n> that simply removes the `PERL` prereq, but it is much harder to jump back\n> and forth over all of these removals when the `.c` version of the filter\n> is added before them and the `.pl` version is removed after them. So I\n> find that it was bad advice, but I do not fault you for following it (we\n> all want reviews to just be over already and therefore sometimes pander to\n> the reviewers, no matter how much or little sense their feedback makes).\n> [...]\n> To illustrate my point: it was a bit of a challenge to find the adjustment\n> of the \"smudge write error at\" needle in all of that cruft. It would have\n> made my life as a reviewer substantially easier had the patch\n> series been organized this way (which I assume you had before the feedback\n> you received demanded to squash everything in one hot pile):\n\nIf you don't think a suggestion of mine makes sense, I'd appreciate it\nif you just replied me directly, instead of sending this sort of comment\nto someone else. I find your wording here to be somewhere between snarky\nand mean-spirited. I didn't demand anything.\n\nIf this was the first time this sort of thing has occurred I wouldn't\nsay anything about it, but this is far from being the first time.\n\nIn any case, if you read more than a few words into\nhttps://lore.kernel.org/git/220723.86pmhwquie.gmgdl@evledraar.gmail.com/\nyou'll see that I suggested splitting the removal of the PERL prereq\ninto its own change, which I think would address what you're bringing up\nhere.\n\nWhat I was mainly commenting on was that this series could avoid\nintroducing code in-between the v1 1/2 and 2/2 which is only needed\nbecause of that split-up. I.e. the \"exec\", and needing to quote those\narguments.\n\nWhich I stand by, I think it's much easier to just do a \"git show\n--word-diff\" on this than reason about how that \"chain-loading\" is\nworking, and whether the inter-series state is buggy. But again, the\nconcern you about the associated verbosity is easy to mitigate.\n\nOn the point of pandering to reviewers I find it really nitpicky to ask\nfor changes to change some working O(N^2)) code in a test-tool to O(N\nlog(N)), or to avoid a few allocations here & there.\n\nIf it was a new git built-in, then sure, but I think our collective time\nis much better spend by just letting that sort of thing slide when it\ncomes to test-tools, which are almost always going to be operating on\nthe relatively tiny set of test data we expose them too.\n\nUnless Matheus is keenly interested on optimizing this code, that is.\n\n> [...]\n> This does what we want it to do, but it looks as if it was code translated\n> from a language that does not care one bit about allocations to a language\n> that cares a lot.\n\nFWIW Perl cares a lot about allocations, the sort of code you're\ncommenting on here doesn't involve allocations in Perl in the general\ncase, since it \"allocates ahead\", similar to how we use alloc_nr() and\nstrbuf_reset() patterns.\n\nWhat it doesn't care about is free()-ing memory, which is an orthagonal\nthing. But that's just an optimization, the general assumptios in that\nif your program ever needs X MB of memory it's likely to need at least\nthat much again, or it'll exit() and have the OS clean it up.\n\n\n\n"},{"id":"460323","messageId":"CAHd-oW6LZay=MX2FdFjgTh1pjE=g-XTm63mGWuMhHd=-N=tXRA@mail.gmail.com","threadId":"58212","inReplyTo":"4n20476q-6ssr-osp8-q5o3-p8ns726q4pn3@tzk.qr","subject":"Re: [PATCH v2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-31T02:52:16Z","receivedAt":"2022-07-31T03:04:17Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Dscho\n\nOn Thu, Jul 28, 2022 at 1:58 PM Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n>\n> > On Sun, 24 Jul 2022, Matheus Tavares wrote:\n> >\n> > diff --git a/t/helper/test-rot13-filter.c b/t/helper/test-rot13-filter.c\n> > +static char *rot13(char *str)\n> > +{\n> > +     char *c;\n> > +     for (c = str; *c; c++) {\n> > +             if (*c >= 'a' && *c <= 'z')\n> > +                     *c = 'a' + (*c - 'a' + 13) % 26;\n> > +             else if (*c >= 'A' && *c <= 'Z')\n> > +                     *c = 'A' + (*c - 'A' + 13) % 26;\n>\n> That's quite verbose, but it _is_ correct (if a bit harder than necessary\n> to validate, I admit that I had to look up whether `%`'s precedence is higher\n> than `+` in https://en.cppreference.com/w/c/language/operator_precedence).\n>\n> A conciser way (also easier to reason about):\n>\n>         for (c = str; *c; c++)\n>                 if (isalpha(*c))\n>                         *c += tolower(*c) < 'n' ? 13 : -13;\n\nNice :) Thanks.\n\n> > [...]\n> > +static void packet_read_capabilities(struct string_list *caps)\n> > +{\n> > +     while (1) {\n> > +             int size;\n> > +             char *buf = packet_read_line(0, &size);\n> > +             if (!buf)\n> > +                     break;\n> > +             string_list_append_nodup(caps,\n> > +                                      skip_key_dup(buf, size, \"capability\"));\n>\n> It is tempting to use unsorted string lists for everything because Perl\n> makes that relatively easy.\n>\n> However, in this instance I would strongly recommend using something more\n> akin to Perl's \"hash\" data structure, in this instance a `strset`.\n\nOk, will do.\n\n> > +\n> > +/*\n> > + * Check our capabilities we want to advertise against the remote ones\n> > + * and then advertise our capabilities\n> > + */\n> > +static void packet_check_and_write_capabilities(struct string_list *remote_caps,\n> > +                                             struct string_list *our_caps)\n>\n> The list of \"our caps\" comes from the command-line. In C, this means we\n> get a `const char **argv` and an `int argc`. So:\n>\n> static void check_and_write_capabilities(struct strset *remote_caps,\n>                                          const char **caps, int caps_count)\n> {\n>         int i;\n>\n>         for (i = 0; i < caps_count; i++) {\n>                 if (!strset_contains(remote_caps, caps[i]))\n>                         die(\"our capability '%s' is not available from remote\",\n>                             caps[i]);\n>\n>                 packet_write_fmt(1, \"capability=%s\\n\", caps[i]);\n>         }\n>         packet_flush(1);\n> }\n\nMakes sense. We also use the list elsewhere (has_capability()), but we\ncan easily replace that with two global flags to indicate if we have\nthe \"clean\" and \"smudge\" caps.\n\n> And then we would call it via\n>\n>         check_and_write_capabilities(remote_caps, argv + 1, argc - 1);\n>\n> [...]\n> > +static void command_loop(void)\n> > +{\n> > +     while (1) {\n> > +             char *command = packet_key_val_read(\"command\");\n> > +             if (!command) {\n> > +                     fprintf(logfile, \"STOP\\n\");\n> > +                     break;\n> > +             }\n> > +             fprintf(logfile, \"IN: %s\", command);\n>\n> We will also need to `fflush(logfile)` here, to imitate the Perl script's\n> behavior more precisely.\n\nI was somewhat intrigued as to why the flushes were needed in the Perl\nscript. But reading [1] and [2], now, it seems to have been an\noversight.\n\nThat is, Eric suggested splictily flushing stdout because it is a\npipe, but the author ended up erroneously disabling autoflush for\nstdout too, so that's why we needed the flushes there. They later\nacknowledged that and said that they would re-enabled it (see [2]),\nbut it seems to have been forgotten. So I think we can safely drop the\nflush calls.\n\n[1]: http://public-inbox.org/git/20160723072721.GA20875%40starla/\n[2]: https://lore.kernel.org/git/7F1F1A0E-8FC3-4FBD-81AA-37786DE0EF50@gmail.com/\n\n> > +\n> > +             if (!strcmp(command, \"list_available_blobs\")) {\n> > +                     struct hashmap_iter iter;\n> > +                     struct strmap_entry *ent;\n> > +                     struct string_list_item *str_item;\n> > +                     struct string_list paths = STRING_LIST_INIT_NODUP;\n> > +\n> > +                     /* flush */\n> > +                     if (packet_read_line(0, NULL))\n> > +                             die(\"bad list_available_blobs end\");\n> > +\n> > +                     strmap_for_each_entry(&delay, &iter, ent) {\n> > +                             struct delay_entry *delay_entry = ent->value;\n> > +                             if (!delay_entry->requested)\n> > +                                     continue;\n> > +                             delay_entry->count--;\n> > +                             if (!strcmp(ent->key, \"invalid-delay.a\")) {\n> > +                                     /* Send Git a pathname that was not delayed earlier */\n> > +                                     packet_write_fmt(1, \"pathname=unfiltered\");\n> > +                             }\n> > +                             if (!strcmp(ent->key, \"missing-delay.a\")) {\n> > +                                     /* Do not signal Git that this file is available */\n> > +                             } else if (!delay_entry->count) {\n> > +                                     string_list_insert(&paths, ent->key);\n> > +                                     packet_write_fmt(1, \"pathname=%s\", ent->key);\n> > +                             }\n> > +                     }\n> > +\n> > +                     /* Print paths in sorted order. */\n>\n> The Perl script does not order them specifically. Do we really have to do\n> that here?\n\nIt actually prints them in sorted order:\n\n        foreach my $pathname ( sort keys %DELAY )\n\nThat is required because some test cases will compare the output using\nthis order.\n\n> In any case, it is more performant to append the paths in an unsorted way\n> and then sort them once in the end (that's O(N log(N)) instead of O(N^2)).\n\nOK, will do.\n\n> > +                     for_each_string_list_item(str_item, &paths)\n> > +                             fprintf(logfile, \" %s\", str_item->string);\n> > +                     string_list_clear(&paths, 0);\n> > +\n> > +                     packet_flush(1);\n> > +\n> > +                     fprintf(logfile, \" [OK]\\n\");\n> > +                     packet_write_fmt(1, \"status=success\");\n> > +                     packet_flush(1);\n>\n> I know the Perl script uses an else here, but I'd much rather insert a\n> `continue` at the end of the `list_available_blobs` clause and de-indent\n> the remainder of the loop body.\n\nSure! I think we can take a step further and extract the if logic to a\nseparate function.\n\n> > +             } else {\n> > +                     char *buf, *output;\n> > +                     int size;\n> > +                     char *pathname;\n> > +                     struct delay_entry *entry;\n> > +                     struct strbuf input = STRBUF_INIT;\n> > +\n> > +                     pathname = packet_key_val_read(\"pathname\");\n> > +                     if (!pathname)\n> > +                             die(\"unexpected EOF while expecting pathname\");\n> > +                     fprintf(logfile, \" %s\", pathname);\n>\n> Again, let's `fflush(logfile)` here.\n>\n> > +\n> > +                     /* Read until flush */\n> > +                     buf = packet_read_line(0, &size);\n> > +                     while (buf) {\n>\n> Let's write this in more idiomatic C:\n>\n>                         while ((buf = packet_read_line(0, &size))) {\n>\n> > +                             if (!strcmp(buf, \"can-delay=1\")) {\n> > +                                     entry = strmap_get(&delay, pathname);\n> > +                                     if (entry && !entry->requested) {\n> > +                                             entry->requested = 1;\n> > +                                     } else if (!entry && always_delay) {\n> > +                                             entry = xcalloc(1, sizeof(*entry));\n> > +                                             entry->requested = 1;\n> > +                                             entry->count = 1;\n> > +                                             strmap_put(&delay, pathname, entry);\n>\n> I guess here is our chance to extend the signature of `add_delay_entry()`\n> to accept a `requested` parameter, and to call that here.\n>\n> > +                                     }\n> > +                             } else if (starts_with(buf, \"ref=\") ||\n> > +                                        starts_with(buf, \"treeish=\") ||\n> > +                                        starts_with(buf, \"blob=\")) {\n> > +                                     fprintf(logfile, \" %s\", buf);\n> > +                             } else {\n> > +                                     /*\n> > +                                      * In general, filters need to be graceful about\n> > +                                      * new metadata, since it's documented that we\n> > +                                      * can pass any key-value pairs, but for tests,\n> > +                                      * let's be a little stricter.\n> > +                                      */\n> > +                                     die(\"Unknown message '%s'\", buf);\n> > +                             }\n> > +                             buf = packet_read_line(0, &size);\n> > +                     }\n> > +\n> > +\n> > +                     read_packetized_to_strbuf(0, &input, 0);\n> > +                     fprintf(logfile, \" %\"PRIuMAX\" [OK] -- \", (uintmax_t)input.len);\n>\n> This reads _so much nicer_ than the Perl version!\n>\n> > +\n> > +                     entry = strmap_get(&delay, pathname);\n> > +                     if (entry && entry->output) {\n> > +                             output = entry->output;\n> > +                     } else if (!strcmp(pathname, \"error.r\") || !strcmp(pathname, \"abort.r\")) {\n> > +                             output = \"\";\n> > +                     } else if (!strcmp(command, \"clean\") && has_capability(\"clean\")) {\n> > +                             output = rot13(input.buf);\n> > +                     } else if (!strcmp(command, \"smudge\") && has_capability(\"smudge\")) {\n> > +                             output = rot13(input.buf);\n> > +                     } else {\n> > +                             die(\"bad command '%s'\", command);\n> > +                     }\n> > +\n> > +                     if (!strcmp(pathname, \"error.r\")) {\n> > +                             fprintf(logfile, \"[ERROR]\\n\");\n> > +                             packet_write_fmt(1, \"status=error\");\n> > +                             packet_flush(1);\n> > +                     } else if (!strcmp(pathname, \"abort.r\")) {\n> > +                             fprintf(logfile, \"[ABORT]\\n\");\n> > +                             packet_write_fmt(1, \"status=abort\");\n> > +                             packet_flush(1);\n> > +                     } else if (!strcmp(command, \"smudge\") &&\n> > +                                (entry = strmap_get(&delay, pathname)) &&\n> > +                                entry->requested == 1) {\n> > +                             fprintf(logfile, \"[DELAYED]\\n\");\n> > +                             packet_write_fmt(1, \"status=delayed\");\n> > +                             packet_flush(1);\n> > +                             entry->requested = 2;\n> > +                             entry->output = xstrdup(output);\n>\n> We need to call `free(entry->output)` before that lest we leak memory, but\n> only if `output` is not identical anyway:\n>\n>                                 if (entry->output != output) {\n>                                         free(entry->output);\n>                                         entry->output = xstrdup(output);\n>                                 }\n\nI think, entry->output will always be NULL here, since we only get\ninside this if block after entry->requested has been set to 1 at the\ntop of the function; and, at that point, we haven't run ro13 yet.\nNevertheless, it doesn't hurt to add the free call anyway :)\n\n>\n> > +                     } else {\n> > +                             int i, nr_packets;\n> > +                             size_t output_len;\n> > +                             struct strbuf sb = STRBUF_INIT;\n> > +                             packet_write_fmt(1, \"status=success\");\n> > +                             packet_flush(1);\n> > +\n> > +                             strbuf_addf(&sb, \"%s-write-fail.r\", command);\n> > +                             if (!strcmp(pathname, sb.buf)) {\n>\n> We can easily avoid allocating the string just for comparing it:\n>\n>                                 const char *p;\n>\n>                                 if (skip_prefix(pathname, command, &p) &&\n>                                     !strcmp(p, \"-write-fail.r\")) {\n>\n> > +                                     fprintf(logfile, \"[WRITE FAIL]\\n\");\n>\n>                                         fflush(logfile) ;-)\n>\n> > +                                     die(\"%s write error\", command);\n> > +                             }\n> > +\n> > +                             output_len = strlen(output);\n> > +                             fprintf(logfile, \"OUT: %\"PRIuMAX\" \", (uintmax_t)output_len);\n> > +\n> > +                             if (write_packetized_from_buf_no_flush_count(output,\n> > +                                     output_len, 1, &nr_packets))\n> > +                                     die(\"failed to write buffer to stdout\");\n> > +                             packet_flush(1);\n> > +\n> > +                             for (i = 0; i < nr_packets; i++)\n> > +                                     fprintf(logfile, \".\");\n>\n> That's not quite the same as the Perl script does: it prints a '.'\n> (without flushing, though) _every_ time it wrote a packet.\n>\n> If you want to emulate that, you will have to copy/edit that loop (and in\n> that case, the insanely long-named function\n> `write_packetized_from_buf_no_flush_count()` is unnecessary, too).\n\nHmm, I'm not sure we need to emulate that. I do dislike the huge\nfunction name as well, but I also don't quite like to repeat code\ncopying that loop here...\n\n> > +                             fprintf(logfile, \" [OK]\\n\");\n> > +\n> > +                             packet_flush(1);\n> > +                             strbuf_release(&sb);\n> > +                     }\n> > +                     free(pathname);\n> > +                     strbuf_release(&input);\n> > +             }\n> > +             free(command);\n> > +     }\n> > +}\n> > [...]\n> > +static void packet_initialize(const char *name, int version)\n> > +{\n> > +     struct strbuf sb = STRBUF_INIT;\n> > +     int size;\n> > +     char *pkt_buf = packet_read_line(0, &size);\n> > +\n> > +     strbuf_addf(&sb, \"%s-client\", name);\n> > +     if (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n>\n> We do not need the flexibility of the Perl package, where `name` is a\n> parameter. We can hard-code `git-filter-client` here. I.e. something like\n> this:\n>\n>         if (!pkt_buf || size != 17 ||\n>             strncmp(pkt_buf, \"git-filter-client\", 17))\n\nGood idea! Thanks. Perhaps, can't we do:\n\n        if (!pkt_buf || strncmp(pkt_buf, \"git-filter-client\", size))\n\nto avoid the hard-coded and possibly error-prone 17?\n\n> > +             die(\"bad initialize: '%s'\", xstrndup(pkt_buf, size));\n> > +\n> > +     strbuf_reset(&sb);\n> > +     strbuf_addf(&sb, \"version=%d\", version);\n\nThanks for a very detailed review and great suggestions!\n"},{"id":"460330","messageId":"cover.1659291025.git.matheus.bernardino@usp.br","threadId":"58212","inReplyTo":"f38f722de7c3323207eda5ea632b5acd3765c285.1658675222.git.matheus.bernardino@usp.br","subject":"[PATCH v3 0/3] t0021: convert perl script to C test-tool helper","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-31T18:19:47Z","receivedAt":"2022-07-31T18:20:08Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Convert t/t0021/rot13-filter.pl to a test-tool helper to avoid the PERL\nprereq in various tests.\n\nChanges since v2:\n\n- Split into 3 patches.\n- write_packetized_from_buf_no_flush(): s/counter_ptr/packet_counter/ and\n  incremented ptr directly.\n- Convert write_packetized_from_buf_no_flush_count() to static inline.\n- Simplified rot13 routine.\n- Avoided memory allocations at skip_key_dup (now get_value()) and\n  packet_initialize().\n- Replace unsorted list \"remote_caps\" by strset.\n- Simplified packet_read_and_check_capabilities() (now read_capabilities()),\n  to test the tree capabilities directly.\n- check_and_write_capabilities(): operate on (argv, argc) directly, instead of\n  creating a list.\n- Moved \"struct delay_entry\" routines closed to the struct declaration.\n- command_loop(): sorted paths after their insertion to the list.\n- command_loop(): extracted list_available_blobs logic to separated function.\n- Other small refactoring for more idiomatic code.\n\nMatheus Tavares (3):\n  t0021: avoid grepping for a Perl-specific string at filter output\n  t0021: implementation the rot13-filter.pl script in C\n  tests: use the new C rot13-filter helper to avoid PERL prereq\n\n Makefile                                |   1 +\n pkt-line.c                              |   5 +-\n pkt-line.h                              |   8 +-\n t/helper/test-rot13-filter.c            | 379 ++++++++++++++++++++++++\n t/helper/test-tool.c                    |   1 +\n t/helper/test-tool.h                    |   1 +\n t/t0021-conversion.sh                   |  71 +++--\n t/t0021/rot13-filter.pl                 | 247 ---------------\n t/t2080-parallel-checkout-basics.sh     |   7 +-\n t/t2082-parallel-checkout-attributes.sh |   7 +-\n 10 files changed, 431 insertions(+), 296 deletions(-)\n create mode 100644 t/helper/test-rot13-filter.c\n delete mode 100644 t/t0021/rot13-filter.pl\n\nRange-diff against v2:\n-:  ---------- > 1:  5ec95c7e69 t0021: avoid grepping for a Perl-specific string at filter output\n-:  ---------- > 2:  86e6baba46 t0021: implementation the rot13-filter.pl script in C\n1:  f38f722de7 ! 3:  c66fc0a186 t/t0021: convert the rot13-filter.pl script to C\n    @@ Metadata\n     Author: Matheus Tavares <matheus.bernardino@usp.br>\n     \n      ## Commit message ##\n    -    t/t0021: convert the rot13-filter.pl script to C\n    +    tests: use the new C rot13-filter helper to avoid PERL prereq\n     \n    -    This script is currently used by three test files: t0021-conversion.sh,\n    -    t2080-parallel-checkout-basics.sh, and\n    -    t2082-parallel-checkout-attributes.sh. To avoid the need for the PERL\n    -    dependency at these tests, let's convert the script to a C test-tool\n    -    command.\n    -\n    -    Note that there is a small adjustment needed at test t0021-conversion.sh\n    -    because it depended on a specific error message given by perl's die\n    -    routine.\n    +    The previous commit implemented a C version of the t0021/rot13-filter.pl\n    +    script. Let's use this new C helper to eliminate the PERL prereq from\n    +    various tests, and also remove the superseded Perl script.\n     \n         Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>\n     \n    - ## Makefile ##\n    -@@ Makefile: TEST_BUILTINS_OBJS += test-read-midx.o\n    - TEST_BUILTINS_OBJS += test-ref-store.o\n    - TEST_BUILTINS_OBJS += test-reftable.o\n    - TEST_BUILTINS_OBJS += test-regex.o\n    -+TEST_BUILTINS_OBJS += test-rot13-filter.o\n    - TEST_BUILTINS_OBJS += test-repository.o\n    - TEST_BUILTINS_OBJS += test-revision-walking.o\n    - TEST_BUILTINS_OBJS += test-run-command.o\n    -\n    - ## pkt-line.c ##\n    -@@ pkt-line.c: int write_packetized_from_fd_no_flush(int fd_in, int fd_out)\n    - \treturn err;\n    - }\n    - \n    --int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n    -+int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n    -+\t\t\t\t\t     int fd_out, int *count_ptr)\n    - {\n    --\tint err = 0;\n    -+\tint err = 0, count = 0;\n    - \tsize_t bytes_written = 0;\n    - \tsize_t bytes_to_write;\n    - \n    -@@ pkt-line.c: int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_ou\n    - \t\t\tbreak;\n    - \t\terr = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);\n    - \t\tbytes_written += bytes_to_write;\n    -+\t\tcount++;\n    - \t}\n    -+\tif (count_ptr)\n    -+\t\t*count_ptr = count;\n    - \treturn err;\n    - }\n    - \n    -+int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n    -+{\n    -+\treturn write_packetized_from_buf_no_flush_count(src_in, len, fd_out, NULL);\n    -+}\n    -+\n    - static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n    - \t\t\t   void *dst, unsigned size, int options)\n    - {\n    -\n    - ## pkt-line.h ##\n    -@@ pkt-line.h: int packet_flush_gently(int fd);\n    - int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n    - int write_packetized_from_fd_no_flush(int fd_in, int fd_out);\n    - int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out);\n    -+int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n    -+\t\t\t\t\t     int fd_out, int *count_ptr);\n    - \n    - /*\n    -  * Stdio versions of packet_write functions. When mixing these with fd\n    -\n    - ## t/helper/test-rot13-filter.c (new) ##\n    -@@\n    -+/*\n    -+ * Example implementation for the Git filter protocol version 2\n    -+ * See Documentation/gitattributes.txt, section \"Filter Protocol\"\n    -+ *\n    -+ * Usage: test-tool rot13-filter [--always-delay] <log path> <capabilities>\n    -+ *\n    -+ * Log path defines a debug log file that the script writes to. The\n    -+ * subsequent arguments define a list of supported protocol capabilities\n    -+ * (\"clean\", \"smudge\", etc).\n    -+ *\n    -+ * When --always-delay is given all pathnames with the \"can-delay\" flag\n    -+ * that don't appear on the list bellow are delayed with a count of 1\n    -+ * (see more below).\n    -+ *\n    -+ * This implementation supports special test cases:\n    -+ * (1) If data with the pathname \"clean-write-fail.r\" is processed with\n    -+ *     a \"clean\" operation then the write operation will die.\n    -+ * (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n    -+ *     a \"smudge\" operation then the write operation will die.\n    -+ * (3) If data with the pathname \"error.r\" is processed with any\n    -+ *     operation then the filter signals that it cannot or does not want\n    -+ *     to process the file.\n    -+ * (4) If data with the pathname \"abort.r\" is processed with any\n    -+ *     operation then the filter signals that it cannot or does not want\n    -+ *     to process the file and any file after that is processed with the\n    -+ *     same command.\n    -+ * (5) If data with a pathname that is a key in the delay hash is\n    -+ *     requested (e.g. \"test-delay10.a\") then the filter responds with\n    -+ *     a \"delay\" status and sets the \"requested\" field in the delay hash.\n    -+ *     The filter will signal the availability of this object after\n    -+ *     \"count\" (field in delay hash) \"list_available_blobs\" commands.\n    -+ * (6) If data with the pathname \"missing-delay.a\" is processed that the\n    -+ *     filter will drop the path from the \"list_available_blobs\" response.\n    -+ * (7) If data with the pathname \"invalid-delay.a\" is processed that the\n    -+ *     filter will add the path \"unfiltered\" which was not delayed before\n    -+ *     to the \"list_available_blobs\" response.\n    -+ */\n    -+\n    -+#include \"test-tool.h\"\n    -+#include \"pkt-line.h\"\n    -+#include \"string-list.h\"\n    -+#include \"strmap.h\"\n    -+\n    -+static FILE *logfile;\n    -+static int always_delay;\n    -+static struct strmap delay = STRMAP_INIT;\n    -+static struct string_list requested_caps = STRING_LIST_INIT_NODUP;\n    -+\n    -+static int has_capability(const char *cap)\n    -+{\n    -+\treturn unsorted_string_list_has_string(&requested_caps, cap);\n    -+}\n    -+\n    -+static char *rot13(char *str)\n    -+{\n    -+\tchar *c;\n    -+\tfor (c = str; *c; c++) {\n    -+\t\tif (*c >= 'a' && *c <= 'z')\n    -+\t\t\t*c = 'a' + (*c - 'a' + 13) % 26;\n    -+\t\telse if (*c >= 'A' && *c <= 'Z')\n    -+\t\t\t*c = 'A' + (*c - 'A' + 13) % 26;\n    -+\t}\n    -+\treturn str;\n    -+}\n    -+\n    -+static char *skip_key_dup(const char *buf, size_t size, const char *key)\n    -+{\n    -+\tstruct strbuf keybuf = STRBUF_INIT;\n    -+\tstrbuf_addf(&keybuf, \"%s=\", key);\n    -+\tif (!skip_prefix_mem(buf, size, keybuf.buf, &buf, &size) || !size)\n    -+\t\tdie(\"bad %s: '%s'\", key, xstrndup(buf, size));\n    -+\tstrbuf_release(&keybuf);\n    -+\treturn xstrndup(buf, size);\n    -+}\n    -+\n    -+/*\n    -+ * Read a text packet, expecting that it is in the form \"key=value\" for\n    -+ * the given key. An EOF does not trigger any error and is reported\n    -+ * back to the caller with NULL. Die if the \"key\" part of \"key=value\" does\n    -+ * not match the given key, or the value part is empty.\n    -+ */\n    -+static char *packet_key_val_read(const char *key)\n    -+{\n    -+\tint size;\n    -+\tchar *buf;\n    -+\tif (packet_read_line_gently(0, &size, &buf) < 0)\n    -+\t\treturn NULL;\n    -+\treturn skip_key_dup(buf, size, key);\n    -+}\n    -+\n    -+static void packet_read_capabilities(struct string_list *caps)\n    -+{\n    -+\twhile (1) {\n    -+\t\tint size;\n    -+\t\tchar *buf = packet_read_line(0, &size);\n    -+\t\tif (!buf)\n    -+\t\t\tbreak;\n    -+\t\tstring_list_append_nodup(caps,\n    -+\t\t\t\t\t skip_key_dup(buf, size, \"capability\"));\n    -+\t}\n    -+}\n    -+\n    -+/* Read remote capabilities and check them against capabilities we require */\n    -+static void packet_read_and_check_capabilities(struct string_list *remote_caps,\n    -+\t\t\t\t\t       struct string_list *required_caps)\n    -+{\n    -+\tstruct string_list_item *item;\n    -+\tpacket_read_capabilities(remote_caps);\n    -+\tfor_each_string_list_item(item, required_caps) {\n    -+\t\tif (!unsorted_string_list_has_string(remote_caps, item->string)) {\n    -+\t\t\tdie(\"required '%s' capability not available from remote\",\n    -+\t\t\t    item->string);\n    -+\t\t}\n    -+\t}\n    -+}\n    -+\n    -+/*\n    -+ * Check our capabilities we want to advertise against the remote ones\n    -+ * and then advertise our capabilities\n    -+ */\n    -+static void packet_check_and_write_capabilities(struct string_list *remote_caps,\n    -+\t\t\t\t\t\tstruct string_list *our_caps)\n    -+{\n    -+\tstruct string_list_item *item;\n    -+\tfor_each_string_list_item(item, our_caps) {\n    -+\t\tif (!unsorted_string_list_has_string(remote_caps, item->string)) {\n    -+\t\t\tdie(\"our capability '%s' is not available from remote\",\n    -+\t\t\t    item->string);\n    -+\t\t}\n    -+\t\tpacket_write_fmt(1, \"capability=%s\\n\", item->string);\n    -+\t}\n    -+\tpacket_flush(1);\n    -+}\n    -+\n    -+struct delay_entry {\n    -+\tint requested, count;\n    -+\tchar *output;\n    -+};\n    -+\n    -+static void command_loop(void)\n    -+{\n    -+\twhile (1) {\n    -+\t\tchar *command = packet_key_val_read(\"command\");\n    -+\t\tif (!command) {\n    -+\t\t\tfprintf(logfile, \"STOP\\n\");\n    -+\t\t\tbreak;\n    -+\t\t}\n    -+\t\tfprintf(logfile, \"IN: %s\", command);\n    -+\n    -+\t\tif (!strcmp(command, \"list_available_blobs\")) {\n    -+\t\t\tstruct hashmap_iter iter;\n    -+\t\t\tstruct strmap_entry *ent;\n    -+\t\t\tstruct string_list_item *str_item;\n    -+\t\t\tstruct string_list paths = STRING_LIST_INIT_NODUP;\n    -+\n    -+\t\t\t/* flush */\n    -+\t\t\tif (packet_read_line(0, NULL))\n    -+\t\t\t\tdie(\"bad list_available_blobs end\");\n    -+\n    -+\t\t\tstrmap_for_each_entry(&delay, &iter, ent) {\n    -+\t\t\t\tstruct delay_entry *delay_entry = ent->value;\n    -+\t\t\t\tif (!delay_entry->requested)\n    -+\t\t\t\t\tcontinue;\n    -+\t\t\t\tdelay_entry->count--;\n    -+\t\t\t\tif (!strcmp(ent->key, \"invalid-delay.a\")) {\n    -+\t\t\t\t\t/* Send Git a pathname that was not delayed earlier */\n    -+\t\t\t\t\tpacket_write_fmt(1, \"pathname=unfiltered\");\n    -+\t\t\t\t}\n    -+\t\t\t\tif (!strcmp(ent->key, \"missing-delay.a\")) {\n    -+\t\t\t\t\t/* Do not signal Git that this file is available */\n    -+\t\t\t\t} else if (!delay_entry->count) {\n    -+\t\t\t\t\tstring_list_insert(&paths, ent->key);\n    -+\t\t\t\t\tpacket_write_fmt(1, \"pathname=%s\", ent->key);\n    -+\t\t\t\t}\n    -+\t\t\t}\n    -+\n    -+\t\t\t/* Print paths in sorted order. */\n    -+\t\t\tfor_each_string_list_item(str_item, &paths)\n    -+\t\t\t\tfprintf(logfile, \" %s\", str_item->string);\n    -+\t\t\tstring_list_clear(&paths, 0);\n    -+\n    -+\t\t\tpacket_flush(1);\n    -+\n    -+\t\t\tfprintf(logfile, \" [OK]\\n\");\n    -+\t\t\tpacket_write_fmt(1, \"status=success\");\n    -+\t\t\tpacket_flush(1);\n    -+\t\t} else {\n    -+\t\t\tchar *buf, *output;\n    -+\t\t\tint size;\n    -+\t\t\tchar *pathname;\n    -+\t\t\tstruct delay_entry *entry;\n    -+\t\t\tstruct strbuf input = STRBUF_INIT;\n    -+\n    -+\t\t\tpathname = packet_key_val_read(\"pathname\");\n    -+\t\t\tif (!pathname)\n    -+\t\t\t\tdie(\"unexpected EOF while expecting pathname\");\n    -+\t\t\tfprintf(logfile, \" %s\", pathname);\n    -+\n    -+\t\t\t/* Read until flush */\n    -+\t\t\tbuf = packet_read_line(0, &size);\n    -+\t\t\twhile (buf) {\n    -+\t\t\t\tif (!strcmp(buf, \"can-delay=1\")) {\n    -+\t\t\t\t\tentry = strmap_get(&delay, pathname);\n    -+\t\t\t\t\tif (entry && !entry->requested) {\n    -+\t\t\t\t\t\tentry->requested = 1;\n    -+\t\t\t\t\t} else if (!entry && always_delay) {\n    -+\t\t\t\t\t\tentry = xcalloc(1, sizeof(*entry));\n    -+\t\t\t\t\t\tentry->requested = 1;\n    -+\t\t\t\t\t\tentry->count = 1;\n    -+\t\t\t\t\t\tstrmap_put(&delay, pathname, entry);\n    -+\t\t\t\t\t}\n    -+\t\t\t\t} else if (starts_with(buf, \"ref=\") ||\n    -+\t\t\t\t\t   starts_with(buf, \"treeish=\") ||\n    -+\t\t\t\t\t   starts_with(buf, \"blob=\")) {\n    -+\t\t\t\t\tfprintf(logfile, \" %s\", buf);\n    -+\t\t\t\t} else {\n    -+\t\t\t\t\t/*\n    -+\t\t\t\t\t * In general, filters need to be graceful about\n    -+\t\t\t\t\t * new metadata, since it's documented that we\n    -+\t\t\t\t\t * can pass any key-value pairs, but for tests,\n    -+\t\t\t\t\t * let's be a little stricter.\n    -+\t\t\t\t\t */\n    -+\t\t\t\t\tdie(\"Unknown message '%s'\", buf);\n    -+\t\t\t\t}\n    -+\t\t\t\tbuf = packet_read_line(0, &size);\n    -+\t\t\t}\n    -+\n    -+\n    -+\t\t\tread_packetized_to_strbuf(0, &input, 0);\n    -+\t\t\tfprintf(logfile, \" %\"PRIuMAX\" [OK] -- \", (uintmax_t)input.len);\n    -+\n    -+\t\t\tentry = strmap_get(&delay, pathname);\n    -+\t\t\tif (entry && entry->output) {\n    -+\t\t\t\toutput = entry->output;\n    -+\t\t\t} else if (!strcmp(pathname, \"error.r\") || !strcmp(pathname, \"abort.r\")) {\n    -+\t\t\t\toutput = \"\";\n    -+\t\t\t} else if (!strcmp(command, \"clean\") && has_capability(\"clean\")) {\n    -+\t\t\t\toutput = rot13(input.buf);\n    -+\t\t\t} else if (!strcmp(command, \"smudge\") && has_capability(\"smudge\")) {\n    -+\t\t\t\toutput = rot13(input.buf);\n    -+\t\t\t} else {\n    -+\t\t\t\tdie(\"bad command '%s'\", command);\n    -+\t\t\t}\n    -+\n    -+\t\t\tif (!strcmp(pathname, \"error.r\")) {\n    -+\t\t\t\tfprintf(logfile, \"[ERROR]\\n\");\n    -+\t\t\t\tpacket_write_fmt(1, \"status=error\");\n    -+\t\t\t\tpacket_flush(1);\n    -+\t\t\t} else if (!strcmp(pathname, \"abort.r\")) {\n    -+\t\t\t\tfprintf(logfile, \"[ABORT]\\n\");\n    -+\t\t\t\tpacket_write_fmt(1, \"status=abort\");\n    -+\t\t\t\tpacket_flush(1);\n    -+\t\t\t} else if (!strcmp(command, \"smudge\") &&\n    -+\t\t\t\t   (entry = strmap_get(&delay, pathname)) &&\n    -+\t\t\t\t   entry->requested == 1) {\n    -+\t\t\t\tfprintf(logfile, \"[DELAYED]\\n\");\n    -+\t\t\t\tpacket_write_fmt(1, \"status=delayed\");\n    -+\t\t\t\tpacket_flush(1);\n    -+\t\t\t\tentry->requested = 2;\n    -+\t\t\t\tentry->output = xstrdup(output);\n    -+\t\t\t} else {\n    -+\t\t\t\tint i, nr_packets;\n    -+\t\t\t\tsize_t output_len;\n    -+\t\t\t\tstruct strbuf sb = STRBUF_INIT;\n    -+\t\t\t\tpacket_write_fmt(1, \"status=success\");\n    -+\t\t\t\tpacket_flush(1);\n    -+\n    -+\t\t\t\tstrbuf_addf(&sb, \"%s-write-fail.r\", command);\n    -+\t\t\t\tif (!strcmp(pathname, sb.buf)) {\n    -+\t\t\t\t\tfprintf(logfile, \"[WRITE FAIL]\\n\");\n    -+\t\t\t\t\tdie(\"%s write error\", command);\n    -+\t\t\t\t}\n    -+\n    -+\t\t\t\toutput_len = strlen(output);\n    -+\t\t\t\tfprintf(logfile, \"OUT: %\"PRIuMAX\" \", (uintmax_t)output_len);\n    -+\n    -+\t\t\t\tif (write_packetized_from_buf_no_flush_count(output,\n    -+\t\t\t\t\toutput_len, 1, &nr_packets))\n    -+\t\t\t\t\tdie(\"failed to write buffer to stdout\");\n    -+\t\t\t\tpacket_flush(1);\n    -+\n    -+\t\t\t\tfor (i = 0; i < nr_packets; i++)\n    -+\t\t\t\t\tfprintf(logfile, \".\");\n    -+\t\t\t\tfprintf(logfile, \" [OK]\\n\");\n    -+\n    -+\t\t\t\tpacket_flush(1);\n    -+\t\t\t\tstrbuf_release(&sb);\n    -+\t\t\t}\n    -+\t\t\tfree(pathname);\n    -+\t\t\tstrbuf_release(&input);\n    -+\t\t}\n    -+\t\tfree(command);\n    -+\t}\n    -+}\n    -+\n    -+static void free_delay_hash(void)\n    -+{\n    -+\tstruct hashmap_iter iter;\n    -+\tstruct strmap_entry *ent;\n    -+\n    -+\tstrmap_for_each_entry(&delay, &iter, ent) {\n    -+\t\tstruct delay_entry *delay_entry = ent->value;\n    -+\t\tfree(delay_entry->output);\n    -+\t\tfree(delay_entry);\n    -+\t}\n    -+\tstrmap_clear(&delay, 0);\n    -+}\n    -+\n    -+static void add_delay_entry(char *pathname, int count)\n    -+{\n    -+\tstruct delay_entry *entry = xcalloc(1, sizeof(*entry));\n    -+\tentry->count = count;\n    -+\tif (strmap_put(&delay, pathname, entry))\n    -+\t\tBUG(\"adding the same path twice to delay hash?\");\n    -+}\n    -+\n    -+static void packet_initialize(const char *name, int version)\n    -+{\n    -+\tstruct strbuf sb = STRBUF_INIT;\n    -+\tint size;\n    -+\tchar *pkt_buf = packet_read_line(0, &size);\n    -+\n    -+\tstrbuf_addf(&sb, \"%s-client\", name);\n    -+\tif (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n    -+\t\tdie(\"bad initialize: '%s'\", xstrndup(pkt_buf, size));\n    -+\n    -+\tstrbuf_reset(&sb);\n    -+\tstrbuf_addf(&sb, \"version=%d\", version);\n    -+\tpkt_buf = packet_read_line(0, &size);\n    -+\tif (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n    -+\t\tdie(\"bad version: '%s'\", xstrndup(pkt_buf, size));\n    -+\n    -+\tpkt_buf = packet_read_line(0, &size);\n    -+\tif (pkt_buf)\n    -+\t\tdie(\"bad version end: '%s'\", xstrndup(pkt_buf, size));\n    -+\n    -+\tpacket_write_fmt(1, \"%s-server\", name);\n    -+\tpacket_write_fmt(1, \"version=%d\", version);\n    -+\tpacket_flush(1);\n    -+\tstrbuf_release(&sb);\n    -+}\n    -+\n    -+static char *rot13_usage = \"test-tool rot13-filter [--always-delay] <log path> <capabilities>\";\n    -+\n    -+int cmd__rot13_filter(int argc, const char **argv)\n    -+{\n    -+\tint i = 1;\n    -+\tstruct string_list remote_caps = STRING_LIST_INIT_DUP,\n    -+\t\t\t   supported_caps = STRING_LIST_INIT_NODUP;\n    -+\n    -+\tstring_list_append(&supported_caps, \"clean\");\n    -+\tstring_list_append(&supported_caps, \"smudge\");\n    -+\tstring_list_append(&supported_caps, \"delay\");\n    -+\n    -+\tif (argc > 1 && !strcmp(argv[i], \"--always-delay\")) {\n    -+\t\talways_delay = 1;\n    -+\t\ti++;\n    -+\t}\n    -+\tif (argc - i < 2)\n    -+\t\tusage(rot13_usage);\n    -+\n    -+\tlogfile = fopen(argv[i++], \"a\");\n    -+\tif (!logfile)\n    -+\t\tdie_errno(\"failed to open log file\");\n    -+\n    -+\tfor ( ; i < argc; i++)\n    -+\t\tstring_list_append(&requested_caps, argv[i]);\n    -+\n    -+\tadd_delay_entry(\"test-delay10.a\", 1);\n    -+\tadd_delay_entry(\"test-delay11.a\", 1);\n    -+\tadd_delay_entry(\"test-delay20.a\", 2);\n    -+\tadd_delay_entry(\"test-delay10.b\", 1);\n    -+\tadd_delay_entry(\"missing-delay.a\", 1);\n    -+\tadd_delay_entry(\"invalid-delay.a\", 1);\n    -+\n    -+\tfprintf(logfile, \"START\\n\");\n    -+\n    -+\tpacket_initialize(\"git-filter\", 2);\n    -+\n    -+\tpacket_read_and_check_capabilities(&remote_caps, &supported_caps);\n    -+\tpacket_check_and_write_capabilities(&remote_caps, &requested_caps);\n    -+\tfprintf(logfile, \"init handshake complete\\n\");\n    -+\n    -+\tstring_list_clear(&supported_caps, 0);\n    -+\tstring_list_clear(&remote_caps, 0);\n    -+\n    -+\tcommand_loop();\n    -+\n    -+\tfclose(logfile);\n    -+\tstring_list_clear(&requested_caps, 0);\n    -+\tfree_delay_hash();\n    -+\treturn 0;\n    -+}\n    -\n    - ## t/helper/test-tool.c ##\n    -@@ t/helper/test-tool.c: static struct test_cmd cmds[] = {\n    - \t{ \"read-midx\", cmd__read_midx },\n    - \t{ \"ref-store\", cmd__ref_store },\n    - \t{ \"reftable\", cmd__reftable },\n    -+\t{ \"rot13-filter\", cmd__rot13_filter },\n    - \t{ \"dump-reftable\", cmd__dump_reftable },\n    - \t{ \"regex\", cmd__regex },\n    - \t{ \"repository\", cmd__repository },\n    -\n    - ## t/helper/test-tool.h ##\n    -@@ t/helper/test-tool.h: int cmd__read_cache(int argc, const char **argv);\n    - int cmd__read_graph(int argc, const char **argv);\n    - int cmd__read_midx(int argc, const char **argv);\n    - int cmd__ref_store(int argc, const char **argv);\n    -+int cmd__rot13_filter(int argc, const char **argv);\n    - int cmd__reftable(int argc, const char **argv);\n    - int cmd__regex(int argc, const char **argv);\n    - int cmd__repository(int argc, const char **argv);\n    -\n      ## t/t0021-conversion.sh ##\n     @@ t/t0021-conversion.sh: tr \\\n        'nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM'\n    @@ t/t0021-conversion.sh: test_expect_success PERL 'required process filter with cl\n      \trm -rf repo &&\n      \tmkdir repo &&\n      \t(\n    -@@ t/t0021-conversion.sh: test_expect_success PERL 'process filter should restart after unexpected write f\n    - \t\trm -f debug.log &&\n    - \t\tgit checkout --quiet --no-progress . 2>git-stderr.log &&\n    - \n    --\t\tgrep \"smudge write error at\" git-stderr.log &&\n    -+\t\tgrep \"smudge write error\" git-stderr.log &&\n    - \t\ttest_i18ngrep \"error: external filter\" git-stderr.log &&\n    - \n    - \t\tcat >expected.log <<-EOF &&\n     @@ t/t0021-conversion.sh: test_expect_success PERL 'process filter should restart after unexpected write f\n      \t)\n      '\n-- \n2.37.1\n\n"},{"id":"460331","messageId":"5ec95c7e696a49104322d243bee1d5f137bc8222.1659291025.git.matheus.bernardino@usp.br","threadId":"58212","inReplyTo":"cover.1659291025.git.matheus.bernardino@usp.br","subject":"[PATCH v3 1/3] t0021: avoid grepping for a Perl-specific string at filter output","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-31T18:19:48Z","receivedAt":"2022-07-31T18:20:09Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This test sets the t0021/rot13-filter.pl script as a long-running\nprocess filter for a git checkout command. It then expects the filter to\nfail producing a specific error message at stderr. In the following\ncommits we are going to replace the script with a C test-tool helper,\nbut the test currently expects the error message in a Perl-specific\nformat. That is, when you call `die <msg>` in Perl, it emits\n\"<msg> at - line 1.\" In preparation for the conversion, let's avoid the\nPerl-specific part and only grep for <msg> itself.\n\nSigned-off-by: Matheus Tavares <matheus.bernardino@usp.br>\n---\n t/t0021-conversion.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 1c840348bd..963b66e08c 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -735,7 +735,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t\trm -f debug.log &&\n \t\tgit checkout --quiet --no-progress . 2>git-stderr.log &&\n \n-\t\tgrep \"smudge write error at\" git-stderr.log &&\n+\t\tgrep \"smudge write error\" git-stderr.log &&\n \t\ttest_i18ngrep \"error: external filter\" git-stderr.log &&\n \n \t\tcat >expected.log <<-EOF &&\n-- \n2.37.1\n\n"},{"id":"460332","messageId":"86e6baba460f4d0fce353d1fb6a0e18b57ecadaa.1659291025.git.matheus.bernardino@usp.br","threadId":"58212","inReplyTo":"cover.1659291025.git.matheus.bernardino@usp.br","subject":"[PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-31T18:19:49Z","receivedAt":"2022-07-31T18:20:20Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This script is currently used by three test files: t0021-conversion.sh,\nt2080-parallel-checkout-basics.sh, and\nt2082-parallel-checkout-attributes.sh. To avoid the need for the PERL\ndependency at these tests, let's convert the script to a C test-tool\ncommand. The following commit will take care of actually modifying the\nsaid tests to use the new C helper and removing the Perl script.\n\nHelped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>\nSigned-off-by: Matheus Tavares <matheus.bernardino@usp.br>\n---\n Makefile                     |   1 +\n pkt-line.c                   |   5 +-\n pkt-line.h                   |   8 +-\n t/helper/test-rot13-filter.c | 379 +++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c         |   1 +\n t/helper/test-tool.h         |   1 +\n 6 files changed, 393 insertions(+), 2 deletions(-)\n create mode 100644 t/helper/test-rot13-filter.c\n\ndiff --git a/Makefile b/Makefile\nindex 04d0fd1fe6..7cfcf3a911 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -764,6 +764,7 @@ TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n+TEST_BUILTINS_OBJS += test-rot13-filter.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n TEST_BUILTINS_OBJS += test-run-command.o\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 8e43c2def4..ce4e73b683 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -309,7 +309,8 @@ int write_packetized_from_fd_no_flush(int fd_in, int fd_out)\n \treturn err;\n }\n \n-int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n+int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n+\t\t\t\t\t     int fd_out, int *packet_counter)\n {\n \tint err = 0;\n \tsize_t bytes_written = 0;\n@@ -324,6 +325,8 @@ int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_ou\n \t\t\tbreak;\n \t\terr = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);\n \t\tbytes_written += bytes_to_write;\n+\t\tif (packet_counter)\n+\t\t\t(*packet_counter)++;\n \t}\n \treturn err;\n }\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 6d2a63db23..804fe687fb 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -32,7 +32,13 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((f\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int write_packetized_from_fd_no_flush(int fd_in, int fd_out);\n-int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out);\n+int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n+\t\t\t\t\t     int fd_out, int *packet_counter);\n+static inline int write_packetized_from_buf_no_flush(const char *src_in,\n+\t\t\t\t\t\t     size_t len, int fd_out)\n+{\n+\treturn write_packetized_from_buf_no_flush_count(src_in, len, fd_out, NULL);\n+}\n \n /*\n  * Stdio versions of packet_write functions. When mixing these with fd\ndiff --git a/t/helper/test-rot13-filter.c b/t/helper/test-rot13-filter.c\nnew file mode 100644\nindex 0000000000..d584511f8e\n--- /dev/null\n+++ b/t/helper/test-rot13-filter.c\n@@ -0,0 +1,379 @@\n+/*\n+ * Example implementation for the Git filter protocol version 2\n+ * See Documentation/gitattributes.txt, section \"Filter Protocol\"\n+ *\n+ * Usage: test-tool rot13-filter [--always-delay] <log path> <capabilities>\n+ *\n+ * Log path defines a debug log file that the script writes to. The\n+ * subsequent arguments define a list of supported protocol capabilities\n+ * (\"clean\", \"smudge\", etc).\n+ *\n+ * When --always-delay is given all pathnames with the \"can-delay\" flag\n+ * that don't appear on the list bellow are delayed with a count of 1\n+ * (see more below).\n+ *\n+ * This implementation supports special test cases:\n+ * (1) If data with the pathname \"clean-write-fail.r\" is processed with\n+ *     a \"clean\" operation then the write operation will die.\n+ * (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n+ *     a \"smudge\" operation then the write operation will die.\n+ * (3) If data with the pathname \"error.r\" is processed with any\n+ *     operation then the filter signals that it cannot or does not want\n+ *     to process the file.\n+ * (4) If data with the pathname \"abort.r\" is processed with any\n+ *     operation then the filter signals that it cannot or does not want\n+ *     to process the file and any file after that is processed with the\n+ *     same command.\n+ * (5) If data with a pathname that is a key in the delay hash is\n+ *     requested (e.g. \"test-delay10.a\") then the filter responds with\n+ *     a \"delay\" status and sets the \"requested\" field in the delay hash.\n+ *     The filter will signal the availability of this object after\n+ *     \"count\" (field in delay hash) \"list_available_blobs\" commands.\n+ * (6) If data with the pathname \"missing-delay.a\" is processed that the\n+ *     filter will drop the path from the \"list_available_blobs\" response.\n+ * (7) If data with the pathname \"invalid-delay.a\" is processed that the\n+ *     filter will add the path \"unfiltered\" which was not delayed before\n+ *     to the \"list_available_blobs\" response.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"pkt-line.h\"\n+#include \"string-list.h\"\n+#include \"strmap.h\"\n+\n+static FILE *logfile;\n+static int always_delay, has_clean_cap, has_smudge_cap;\n+static struct strmap delay = STRMAP_INIT;\n+\n+static char *rot13(char *str)\n+{\n+\tchar *c;\n+\tfor (c = str; *c; c++)\n+\t\tif (isalpha(*c))\n+\t\t\t*c += tolower(*c) < 'n' ? 13 : -13;\n+\treturn str;\n+}\n+\n+static char *get_value(char *buf, size_t size, const char *key)\n+{\n+\tconst char *orig_buf = buf;\n+\tint orig_size = (int)size;\n+\n+\tif (!skip_prefix_mem((const char *)buf, size, key, (const char **)&buf, &size) ||\n+\t    !skip_prefix_mem((const char *)buf, size, \"=\", (const char **)&buf, &size) ||\n+\t    !size)\n+\t\tdie(\"expected key '%s', got '%.*s'\",\n+\t\t    key, orig_size, orig_buf);\n+\n+\tbuf[size] = '\\0';\n+\treturn buf;\n+}\n+\n+/*\n+ * Read a text packet, expecting that it is in the form \"key=value\" for\n+ * the given key. An EOF does not trigger any error and is reported\n+ * back to the caller with NULL. Die if the \"key\" part of \"key=value\" does\n+ * not match the given key, or the value part is empty.\n+ */\n+static char *packet_key_val_read(const char *key)\n+{\n+\tint size;\n+\tchar *buf;\n+\tif (packet_read_line_gently(0, &size, &buf) < 0)\n+\t\treturn NULL;\n+\treturn xstrdup(get_value(buf, size, key));\n+}\n+\n+static inline void assert_remote_capability(struct strset *caps, const char *cap)\n+{\n+\tif (!strset_contains(caps, cap))\n+\t\tdie(\"required '%s' capability not available from remote\", cap);\n+}\n+\n+static void read_capabilities(struct strset *remote_caps)\n+{\n+\tfor (;;) {\n+\t\tint size;\n+\t\tchar *buf = packet_read_line(0, &size);\n+\t\tif (!buf)\n+\t\t\tbreak;\n+\t\tstrset_add(remote_caps, get_value(buf, size, \"capability\"));\n+\t}\n+\n+\tassert_remote_capability(remote_caps, \"clean\");\n+\tassert_remote_capability(remote_caps, \"smudge\");\n+\tassert_remote_capability(remote_caps, \"delay\");\n+}\n+\n+static void check_and_write_capabilities(struct strset *remote_caps,\n+\t\t\t\t\t const char **caps, int caps_count)\n+{\n+\tint i;\n+\tfor (i = 0; i < caps_count; i++) {\n+\t\tif (!strset_contains(remote_caps, caps[i]))\n+\t\t\tdie(\"our capability '%s' is not available from remote\",\n+\t\t\t    caps[i]);\n+\t\tpacket_write_fmt(1, \"capability=%s\\n\", caps[i]);\n+\t}\n+\tpacket_flush(1);\n+}\n+\n+struct delay_entry {\n+\tint requested, count;\n+\tchar *output;\n+};\n+\n+static void free_delay_entries(void)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *ent;\n+\n+\tstrmap_for_each_entry(&delay, &iter, ent) {\n+\t\tstruct delay_entry *delay_entry = ent->value;\n+\t\tfree(delay_entry->output);\n+\t\tfree(delay_entry);\n+\t}\n+\tstrmap_clear(&delay, 0);\n+}\n+\n+static void add_delay_entry(char *pathname, int count, int requested)\n+{\n+\tstruct delay_entry *entry = xcalloc(1, sizeof(*entry));\n+\tentry->count = count;\n+\tentry->requested = requested;\n+\tif (strmap_put(&delay, pathname, entry))\n+\t\tBUG(\"adding the same path twice to delay hash?\");\n+}\n+\n+static void reply_list_available_blobs_cmd(void)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *ent;\n+\tstruct string_list_item *str_item;\n+\tstruct string_list paths = STRING_LIST_INIT_NODUP;\n+\n+\t/* flush */\n+\tif (packet_read_line(0, NULL))\n+\t\tdie(\"bad list_available_blobs end\");\n+\n+\tstrmap_for_each_entry(&delay, &iter, ent) {\n+\t\tstruct delay_entry *delay_entry = ent->value;\n+\t\tif (!delay_entry->requested)\n+\t\t\tcontinue;\n+\t\tdelay_entry->count--;\n+\t\tif (!strcmp(ent->key, \"invalid-delay.a\")) {\n+\t\t\t/* Send Git a pathname that was not delayed earlier */\n+\t\t\tpacket_write_fmt(1, \"pathname=unfiltered\");\n+\t\t}\n+\t\tif (!strcmp(ent->key, \"missing-delay.a\")) {\n+\t\t\t/* Do not signal Git that this file is available */\n+\t\t} else if (!delay_entry->count) {\n+\t\t\tstring_list_append(&paths, ent->key);\n+\t\t\tpacket_write_fmt(1, \"pathname=%s\", ent->key);\n+\t\t}\n+\t}\n+\n+\t/* Print paths in sorted order. */\n+\tstring_list_sort(&paths);\n+\tfor_each_string_list_item(str_item, &paths)\n+\t\tfprintf(logfile, \" %s\", str_item->string);\n+\tstring_list_clear(&paths, 0);\n+\n+\tpacket_flush(1);\n+\n+\tfprintf(logfile, \" [OK]\\n\");\n+\tpacket_write_fmt(1, \"status=success\");\n+\tpacket_flush(1);\n+}\n+\n+static void command_loop(void)\n+{\n+\tfor (;;) {\n+\t\tchar *buf, *output;\n+\t\tint size;\n+\t\tchar *pathname;\n+\t\tstruct delay_entry *entry;\n+\t\tstruct strbuf input = STRBUF_INIT;\n+\t\tchar *command = packet_key_val_read(\"command\");\n+\n+\t\tif (!command) {\n+\t\t\tfprintf(logfile, \"STOP\\n\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tfprintf(logfile, \"IN: %s\", command);\n+\n+\t\tif (!strcmp(command, \"list_available_blobs\")) {\n+\t\t\treply_list_available_blobs_cmd();\n+\t\t\tfree(command);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tpathname = packet_key_val_read(\"pathname\");\n+\t\tif (!pathname)\n+\t\t\tdie(\"unexpected EOF while expecting pathname\");\n+\t\tfprintf(logfile, \" %s\", pathname);\n+\n+\t\t/* Read until flush */\n+\t\twhile ((buf = packet_read_line(0, &size))) {\n+\t\t\tif (!strcmp(buf, \"can-delay=1\")) {\n+\t\t\t\tentry = strmap_get(&delay, pathname);\n+\t\t\t\tif (entry && !entry->requested) {\n+\t\t\t\t\tentry->requested = 1;\n+\t\t\t\t} else if (!entry && always_delay) {\n+\t\t\t\t\tadd_delay_entry(pathname, 1, 1);\n+\t\t\t\t}\n+\t\t\t} else if (starts_with(buf, \"ref=\") ||\n+\t\t\t\t   starts_with(buf, \"treeish=\") ||\n+\t\t\t\t   starts_with(buf, \"blob=\")) {\n+\t\t\t\tfprintf(logfile, \" %s\", buf);\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * In general, filters need to be graceful about\n+\t\t\t\t * new metadata, since it's documented that we\n+\t\t\t\t * can pass any key-value pairs, but for tests,\n+\t\t\t\t * let's be a little stricter.\n+\t\t\t\t */\n+\t\t\t\tdie(\"Unknown message '%s'\", buf);\n+\t\t\t}\n+\t\t}\n+\n+\n+\t\tread_packetized_to_strbuf(0, &input, 0);\n+\t\tfprintf(logfile, \" %\"PRIuMAX\" [OK] -- \", (uintmax_t)input.len);\n+\n+\t\tentry = strmap_get(&delay, pathname);\n+\t\tif (entry && entry->output) {\n+\t\t\toutput = entry->output;\n+\t\t} else if (!strcmp(pathname, \"error.r\") || !strcmp(pathname, \"abort.r\")) {\n+\t\t\toutput = \"\";\n+\t\t} else if (!strcmp(command, \"clean\") && has_clean_cap) {\n+\t\t\toutput = rot13(input.buf);\n+\t\t} else if (!strcmp(command, \"smudge\") && has_smudge_cap) {\n+\t\t\toutput = rot13(input.buf);\n+\t\t} else {\n+\t\t\tdie(\"bad command '%s'\", command);\n+\t\t}\n+\n+\t\tif (!strcmp(pathname, \"error.r\")) {\n+\t\t\tfprintf(logfile, \"[ERROR]\\n\");\n+\t\t\tpacket_write_fmt(1, \"status=error\");\n+\t\t\tpacket_flush(1);\n+\t\t} else if (!strcmp(pathname, \"abort.r\")) {\n+\t\t\tfprintf(logfile, \"[ABORT]\\n\");\n+\t\t\tpacket_write_fmt(1, \"status=abort\");\n+\t\t\tpacket_flush(1);\n+\t\t} else if (!strcmp(command, \"smudge\") &&\n+\t\t\t   (entry = strmap_get(&delay, pathname)) &&\n+\t\t\t   entry->requested == 1) {\n+\t\t\tfprintf(logfile, \"[DELAYED]\\n\");\n+\t\t\tpacket_write_fmt(1, \"status=delayed\");\n+\t\t\tpacket_flush(1);\n+\t\t\tentry->requested = 2;\n+\t\t\tif (entry->output != output) {\n+\t\t\t\tfree(entry->output);\n+\t\t\t\tentry->output = xstrdup(output);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tint i, nr_packets = 0;\n+\t\t\tsize_t output_len;\n+\t\t\tconst char *p;\n+\t\t\tpacket_write_fmt(1, \"status=success\");\n+\t\t\tpacket_flush(1);\n+\n+\t\t\tif (skip_prefix(pathname, command, &p) &&\n+\t\t\t    !strcmp(p, \"-write-fail.r\")) {\n+\t\t\t\tfprintf(logfile, \"[WRITE FAIL]\\n\");\n+\t\t\t\tdie(\"%s write error\", command);\n+\t\t\t}\n+\n+\t\t\toutput_len = strlen(output);\n+\t\t\tfprintf(logfile, \"OUT: %\"PRIuMAX\" \", (uintmax_t)output_len);\n+\n+\t\t\tif (write_packetized_from_buf_no_flush_count(output,\n+\t\t\t\toutput_len, 1, &nr_packets))\n+\t\t\t\tdie(\"failed to write buffer to stdout\");\n+\t\t\tpacket_flush(1);\n+\n+\t\t\tfor (i = 0; i < nr_packets; i++)\n+\t\t\t\tfprintf(logfile, \".\");\n+\t\t\tfprintf(logfile, \" [OK]\\n\");\n+\n+\t\t\tpacket_flush(1);\n+\t\t}\n+\t\tfree(pathname);\n+\t\tstrbuf_release(&input);\n+\t\tfree(command);\n+\t}\n+}\n+\n+static void packet_initialize(void)\n+{\n+\tint size;\n+\tchar *pkt_buf = packet_read_line(0, &size);\n+\n+\tif (!pkt_buf || strncmp(pkt_buf, \"git-filter-client\", size))\n+\t\tdie(\"bad initialize: '%s'\", xstrndup(pkt_buf, size));\n+\n+\tpkt_buf = packet_read_line(0, &size);\n+\tif (!pkt_buf || strncmp(pkt_buf, \"version=2\", size))\n+\t\tdie(\"bad version: '%.*s'\", (int)size, pkt_buf);\n+\n+\tpkt_buf = packet_read_line(0, &size);\n+\tif (pkt_buf)\n+\t\tdie(\"bad version end: '%.*s'\", (int)size, pkt_buf);\n+\n+\tpacket_write_fmt(1, \"git-filter-server\");\n+\tpacket_write_fmt(1, \"version=2\");\n+\tpacket_flush(1);\n+}\n+\n+static char *rot13_usage = \"test-tool rot13-filter [--always-delay] <log path> <capabilities>\";\n+\n+int cmd__rot13_filter(int argc, const char **argv)\n+{\n+\tconst char **caps;\n+\tint cap_count, i = 1;\n+\tstruct strset remote_caps = STRSET_INIT;\n+\n+\tif (argc > 1 && !strcmp(argv[1], \"--always-delay\")) {\n+\t\talways_delay = 1;\n+\t\ti++;\n+\t}\n+\tif (argc - i < 2)\n+\t\tusage(rot13_usage);\n+\n+\tlogfile = fopen(argv[i++], \"a\");\n+\tif (!logfile)\n+\t\tdie_errno(\"failed to open log file\");\n+\n+\tcaps = argv + i;\n+\tcap_count = argc - i;\n+\n+\tfor (i = 0; i < cap_count; i++) {\n+\t\tif (!strcmp(caps[i], \"clean\"))\n+\t\t\thas_clean_cap = 1;\n+\t\telse if (!strcmp(caps[i], \"smudge\"))\n+\t\t\thas_smudge_cap = 1;\n+\t}\n+\n+\tadd_delay_entry(\"test-delay10.a\", 1, 0);\n+\tadd_delay_entry(\"test-delay11.a\", 1, 0);\n+\tadd_delay_entry(\"test-delay20.a\", 2, 0);\n+\tadd_delay_entry(\"test-delay10.b\", 1, 0);\n+\tadd_delay_entry(\"missing-delay.a\", 1, 0);\n+\tadd_delay_entry(\"invalid-delay.a\", 1, 0);\n+\n+\tfprintf(logfile, \"START\\n\");\n+\tpacket_initialize();\n+\n+\tread_capabilities(&remote_caps);\n+\tcheck_and_write_capabilities(&remote_caps, caps, cap_count);\n+\tfprintf(logfile, \"init handshake complete\\n\");\n+\tstrset_clear(&remote_caps);\n+\n+\tcommand_loop();\n+\n+\tfclose(logfile);\n+\tfree_delay_entries();\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 318fdbab0c..d6a560f832 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -65,6 +65,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"rot13-filter\", cmd__rot13_filter },\n \t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex bb79927163..21a91b1019 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -54,6 +54,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__rot13_filter(int argc, const char **argv);\n int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n-- \n2.37.1\n\n"},{"id":"460333","messageId":"c66fc0a18699663b4440dbfd2887c2689dc486fe.1659291026.git.matheus.bernardino@usp.br","threadId":"58212","inReplyTo":"cover.1659291025.git.matheus.bernardino@usp.br","subject":"[PATCH v3 3/3] tests: use the new C rot13-filter helper to avoid PERL prereq","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-07-31T18:19:50Z","receivedAt":"2022-07-31T18:20:22Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The previous commit implemented a C version of the t0021/rot13-filter.pl\nscript. Let's use this new C helper to eliminate the PERL prereq from\nvarious tests, and also remove the superseded Perl script.\n\nSigned-off-by: Matheus Tavares <matheus.bernardino@usp.br>\n---\n t/t0021-conversion.sh                   |  69 ++++---\n t/t0021/rot13-filter.pl                 | 247 ------------------------\n t/t2080-parallel-checkout-basics.sh     |   7 +-\n t/t2082-parallel-checkout-attributes.sh |   7 +-\n 4 files changed, 37 insertions(+), 293 deletions(-)\n delete mode 100644 t/t0021/rot13-filter.pl\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 963b66e08c..aeaa8e02ed 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -17,9 +17,6 @@ tr \\\n   'nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM'\n EOF\n \n-write_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl\n-\n generate_random_characters () {\n \tLEN=$1\n \tNAME=$2\n@@ -365,8 +362,8 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '\n \ttest_line_count = 0 count\n '\n \n-test_expect_success PERL 'required process filter should filter data' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should filter data' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -450,8 +447,8 @@ test_expect_success PERL 'required process filter should filter data' '\n \t)\n '\n \n-test_expect_success PERL 'required process filter should filter data for various subcommands' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should filter data for various subcommands' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \t(\n \t\tcd repo &&\n@@ -561,9 +558,9 @@ test_expect_success PERL 'required process filter should filter data for various\n \t)\n '\n \n-test_expect_success PERL 'required process filter takes precedence' '\n+test_expect_success 'required process filter takes precedence' '\n \ttest_config_global filter.protocol.clean false &&\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -587,8 +584,8 @@ test_expect_success PERL 'required process filter takes precedence' '\n \t)\n '\n \n-test_expect_success PERL 'required process filter should be used only for \"clean\" operation only' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean\" &&\n+test_expect_success 'required process filter should be used only for \"clean\" operation only' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -622,8 +619,8 @@ test_expect_success PERL 'required process filter should be used only for \"clean\n \t)\n '\n \n-test_expect_success PERL 'required process filter should process multiple packets' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter should process multiple packets' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \n \trm -rf repo &&\n@@ -687,8 +684,8 @@ test_expect_success PERL 'required process filter should process multiple packet\n \t)\n '\n \n-test_expect_success PERL 'required process filter with clean error should fail' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'required process filter with clean error should fail' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \ttest_config_global filter.protocol.required true &&\n \trm -rf repo &&\n \tmkdir repo &&\n@@ -706,8 +703,8 @@ test_expect_success PERL 'required process filter with clean error should fail'\n \t)\n '\n \n-test_expect_success PERL 'process filter should restart after unexpected write failure' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter should restart after unexpected write failure' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -761,8 +758,8 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n \t)\n '\n \n-test_expect_success PERL 'process filter should not be restarted if it signals an error' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter should not be restarted if it signals an error' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -804,8 +801,8 @@ test_expect_success PERL 'process filter should not be restarted if it signals a\n \t)\n '\n \n-test_expect_success PERL 'process filter abort stops processing of all further files' '\n-\ttest_config_global filter.protocol.process \"rot13-filter.pl debug.log clean smudge\" &&\n+test_expect_success 'process filter abort stops processing of all further files' '\n+\ttest_config_global filter.protocol.process \"test-tool rot13-filter debug.log clean smudge\" &&\n \trm -rf repo &&\n \tmkdir repo &&\n \t(\n@@ -861,10 +858,10 @@ test_expect_success PERL 'invalid process filter must fail (and not hang!)' '\n \t)\n '\n \n-test_expect_success PERL 'delayed checkout in process filter' '\n-\ttest_config_global filter.a.process \"rot13-filter.pl a.log clean smudge delay\" &&\n+test_expect_success 'delayed checkout in process filter' '\n+\ttest_config_global filter.a.process \"test-tool rot13-filter a.log clean smudge delay\" &&\n \ttest_config_global filter.a.required true &&\n-\ttest_config_global filter.b.process \"rot13-filter.pl b.log clean smudge delay\" &&\n+\ttest_config_global filter.b.process \"test-tool rot13-filter b.log clean smudge delay\" &&\n \ttest_config_global filter.b.required true &&\n \n \trm -rf repo &&\n@@ -940,8 +937,8 @@ test_expect_success PERL 'delayed checkout in process filter' '\n \t)\n '\n \n-test_expect_success PERL 'missing file in delayed checkout' '\n-\ttest_config_global filter.bug.process \"rot13-filter.pl bug.log clean smudge delay\" &&\n+test_expect_success 'missing file in delayed checkout' '\n+\ttest_config_global filter.bug.process \"test-tool rot13-filter bug.log clean smudge delay\" &&\n \ttest_config_global filter.bug.required true &&\n \n \trm -rf repo &&\n@@ -960,8 +957,8 @@ test_expect_success PERL 'missing file in delayed checkout' '\n \tgrep \"error: .missing-delay\\.a. was not filtered properly\" git-stderr.log\n '\n \n-test_expect_success PERL 'invalid file in delayed checkout' '\n-\ttest_config_global filter.bug.process \"rot13-filter.pl bug.log clean smudge delay\" &&\n+test_expect_success 'invalid file in delayed checkout' '\n+\ttest_config_global filter.bug.process \"test-tool rot13-filter bug.log clean smudge delay\" &&\n \ttest_config_global filter.bug.required true &&\n \n \trm -rf repo &&\n@@ -990,10 +987,10 @@ do\n \t\tmode_prereq='UTF8_NFD_TO_NFC' ;;\n \tesac\n \n-\ttest_expect_success PERL,SYMLINKS,$mode_prereq \\\n+\ttest_expect_success SYMLINKS,$mode_prereq \\\n \t\"delayed checkout with $mode-collision don't write to the wrong place\" '\n \t\ttest_config_global filter.delay.process \\\n-\t\t\t\"\\\"$TEST_ROOT/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\t\t\"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \t\ttest_config_global filter.delay.required true &&\n \n \t\tgit init $mode-collision &&\n@@ -1026,12 +1023,12 @@ do\n \t'\n done\n \n-test_expect_success PERL,SYMLINKS,CASE_INSENSITIVE_FS \\\n+test_expect_success SYMLINKS,CASE_INSENSITIVE_FS \\\n \"delayed checkout with submodule collision don't write to the wrong place\" '\n \tgit init collision-with-submodule &&\n \t(\n \t\tcd collision-with-submodule &&\n-\t\tgit config filter.delay.process \"\\\"$TEST_ROOT/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\t# We need Git to treat the submodule \"a\" and the\n@@ -1062,11 +1059,11 @@ test_expect_success PERL,SYMLINKS,CASE_INSENSITIVE_FS \\\n \t)\n '\n \n-test_expect_success PERL 'setup for progress tests' '\n+test_expect_success 'setup for progress tests' '\n \tgit init progress &&\n \t(\n \t\tcd progress &&\n-\t\tgit config filter.delay.process \"rot13-filter.pl delay-progress.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter delay-progress.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\techo \"*.a filter=delay\" >.gitattributes &&\n@@ -1132,12 +1129,12 @@ do\n \t'\n done\n \n-test_expect_success PERL 'delayed checkout correctly reports the number of updated entries' '\n+test_expect_success 'delayed checkout correctly reports the number of updated entries' '\n \trm -rf repo &&\n \tgit init repo &&\n \t(\n \t\tcd repo &&\n-\t\tgit config filter.delay.process \"../rot13-filter.pl delayed.log clean smudge delay\" &&\n+\t\tgit config filter.delay.process \"test-tool rot13-filter delayed.log clean smudge delay\" &&\n \t\tgit config filter.delay.required true &&\n \n \t\techo \"*.a filter=delay\" >.gitattributes &&\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\ndeleted file mode 100644\nindex 7bb93768f3..0000000000\n--- a/t/t0021/rot13-filter.pl\n+++ /dev/null\n@@ -1,247 +0,0 @@\n-#\n-# Example implementation for the Git filter protocol version 2\n-# See Documentation/gitattributes.txt, section \"Filter Protocol\"\n-#\n-# Usage: rot13-filter.pl [--always-delay] <log path> <capabilities>\n-#\n-# Log path defines a debug log file that the script writes to. The\n-# subsequent arguments define a list of supported protocol capabilities\n-# (\"clean\", \"smudge\", etc).\n-#\n-# When --always-delay is given all pathnames with the \"can-delay\" flag\n-# that don't appear on the list bellow are delayed with a count of 1\n-# (see more below).\n-#\n-# This implementation supports special test cases:\n-# (1) If data with the pathname \"clean-write-fail.r\" is processed with\n-#     a \"clean\" operation then the write operation will die.\n-# (2) If data with the pathname \"smudge-write-fail.r\" is processed with\n-#     a \"smudge\" operation then the write operation will die.\n-# (3) If data with the pathname \"error.r\" is processed with any\n-#     operation then the filter signals that it cannot or does not want\n-#     to process the file.\n-# (4) If data with the pathname \"abort.r\" is processed with any\n-#     operation then the filter signals that it cannot or does not want\n-#     to process the file and any file after that is processed with the\n-#     same command.\n-# (5) If data with a pathname that is a key in the DELAY hash is\n-#     requested (e.g. \"test-delay10.a\") then the filter responds with\n-#     a \"delay\" status and sets the \"requested\" field in the DELAY hash.\n-#     The filter will signal the availability of this object after\n-#     \"count\" (field in DELAY hash) \"list_available_blobs\" commands.\n-# (6) If data with the pathname \"missing-delay.a\" is processed that the\n-#     filter will drop the path from the \"list_available_blobs\" response.\n-# (7) If data with the pathname \"invalid-delay.a\" is processed that the\n-#     filter will add the path \"unfiltered\" which was not delayed before\n-#     to the \"list_available_blobs\" response.\n-#\n-\n-use 5.008;\n-sub gitperllib {\n-\t# Git assumes that all path lists are Unix-y colon-separated ones. But\n-\t# when the Git for Windows executes the test suite, its MSYS2 Bash\n-\t# calls git.exe, and colon-separated path lists are converted into\n-\t# Windows-y semicolon-separated lists of *Windows* paths (which\n-\t# naturally contain a colon after the drive letter, so splitting by\n-\t# colons simply does not cut it).\n-\t#\n-\t# Detect semicolon-separated path list and handle them appropriately.\n-\n-\tif ($ENV{GITPERLLIB} =~ /;/) {\n-\t\treturn split(/;/, $ENV{GITPERLLIB});\n-\t}\n-\treturn split(/:/, $ENV{GITPERLLIB});\n-}\n-use lib (gitperllib());\n-use strict;\n-use warnings;\n-use IO::File;\n-use Git::Packet;\n-\n-my $MAX_PACKET_CONTENT_SIZE = 65516;\n-\n-my $always_delay = 0;\n-if ( $ARGV[0] eq '--always-delay' ) {\n-\t$always_delay = 1;\n-\tshift @ARGV;\n-}\n-\n-my $log_file                = shift @ARGV;\n-my @capabilities            = @ARGV;\n-\n-open my $debug, \">>\", $log_file or die \"cannot open log file: $!\";\n-\n-my %DELAY = (\n-\t'test-delay10.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'test-delay11.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'test-delay20.a' => { \"requested\" => 0, \"count\" => 2 },\n-\t'test-delay10.b' => { \"requested\" => 0, \"count\" => 1 },\n-\t'missing-delay.a' => { \"requested\" => 0, \"count\" => 1 },\n-\t'invalid-delay.a' => { \"requested\" => 0, \"count\" => 1 },\n-);\n-\n-sub rot13 {\n-\tmy $str = shift;\n-\t$str =~ y/A-Za-z/N-ZA-Mn-za-m/;\n-\treturn $str;\n-}\n-\n-print $debug \"START\\n\";\n-$debug->flush();\n-\n-packet_initialize(\"git-filter\", 2);\n-\n-my %remote_caps = packet_read_and_check_capabilities(\"clean\", \"smudge\", \"delay\");\n-packet_check_and_write_capabilities(\\%remote_caps, @capabilities);\n-\n-print $debug \"init handshake complete\\n\";\n-$debug->flush();\n-\n-while (1) {\n-\tmy ( $res, $command ) = packet_key_val_read(\"command\");\n-\tif ( $res == -1 ) {\n-\t\tprint $debug \"STOP\\n\";\n-\t\texit();\n-\t}\n-\tprint $debug \"IN: $command\";\n-\t$debug->flush();\n-\n-\tif ( $command eq \"list_available_blobs\" ) {\n-\t\t# Flush\n-\t\tpacket_compare_lists([1, \"\"], packet_bin_read()) ||\n-\t\t\tdie \"bad list_available_blobs end\";\n-\n-\t\tforeach my $pathname ( sort keys %DELAY ) {\n-\t\t\tif ( $DELAY{$pathname}{\"requested\"} >= 1 ) {\n-\t\t\t\t$DELAY{$pathname}{\"count\"} = $DELAY{$pathname}{\"count\"} - 1;\n-\t\t\t\tif ( $pathname eq \"invalid-delay.a\" ) {\n-\t\t\t\t\t# Send Git a pathname that was not delayed earlier\n-\t\t\t\t\tpacket_txt_write(\"pathname=unfiltered\");\n-\t\t\t\t}\n-\t\t\t\tif ( $pathname eq \"missing-delay.a\" ) {\n-\t\t\t\t\t# Do not signal Git that this file is available\n-\t\t\t\t} elsif ( $DELAY{$pathname}{\"count\"} == 0 ) {\n-\t\t\t\t\tprint $debug \" $pathname\";\n-\t\t\t\t\tpacket_txt_write(\"pathname=$pathname\");\n-\t\t\t\t}\n-\t\t\t}\n-\t\t}\n-\n-\t\tpacket_flush();\n-\n-\t\tprint $debug \" [OK]\\n\";\n-\t\t$debug->flush();\n-\t\tpacket_txt_write(\"status=success\");\n-\t\tpacket_flush();\n-\t} else {\n-\t\tmy ( $res, $pathname ) = packet_key_val_read(\"pathname\");\n-\t\tif ( $res == -1 ) {\n-\t\t\tdie \"unexpected EOF while expecting pathname\";\n-\t\t}\n-\t\tprint $debug \" $pathname\";\n-\t\t$debug->flush();\n-\n-\t\t# Read until flush\n-\t\tmy ( $done, $buffer ) = packet_txt_read();\n-\t\twhile ( $buffer ne '' ) {\n-\t\t\tif ( $buffer eq \"can-delay=1\" ) {\n-\t\t\t\tif ( exists $DELAY{$pathname} and $DELAY{$pathname}{\"requested\"} == 0 ) {\n-\t\t\t\t\t$DELAY{$pathname}{\"requested\"} = 1;\n-\t\t\t\t} elsif ( !exists $DELAY{$pathname} and $always_delay ) {\n-\t\t\t\t\t$DELAY{$pathname} = { \"requested\" => 1, \"count\" => 1 };\n-\t\t\t\t}\n-\t\t\t} elsif ($buffer =~ /^(ref|treeish|blob)=/) {\n-\t\t\t\tprint $debug \" $buffer\";\n-\t\t\t} else {\n-\t\t\t\t# In general, filters need to be graceful about\n-\t\t\t\t# new metadata, since it's documented that we\n-\t\t\t\t# can pass any key-value pairs, but for tests,\n-\t\t\t\t# let's be a little stricter.\n-\t\t\t\tdie \"Unknown message '$buffer'\";\n-\t\t\t}\n-\n-\t\t\t( $done, $buffer ) = packet_txt_read();\n-\t\t}\n-\t\tif ( $done == -1 ) {\n-\t\t\tdie \"unexpected EOF after pathname '$pathname'\";\n-\t\t}\n-\n-\t\tmy $input = \"\";\n-\t\t{\n-\t\t\tbinmode(STDIN);\n-\t\t\tmy $buffer;\n-\t\t\tmy $done = 0;\n-\t\t\twhile ( !$done ) {\n-\t\t\t\t( $done, $buffer ) = packet_bin_read();\n-\t\t\t\t$input .= $buffer;\n-\t\t\t}\n-\t\t\tif ( $done == -1 ) {\n-\t\t\t\tdie \"unexpected EOF while reading input for '$pathname'\";\n-\t\t\t}\t\t\t\n-\t\t\tprint $debug \" \" . length($input) . \" [OK] -- \";\n-\t\t\t$debug->flush();\n-\t\t}\n-\n-\t\tmy $output;\n-\t\tif ( exists $DELAY{$pathname} and exists $DELAY{$pathname}{\"output\"} ) {\n-\t\t\t$output = $DELAY{$pathname}{\"output\"}\n-\t\t} elsif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n-\t\t\t$output = \"\";\n-\t\t} elsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n-\t\t\t$output = rot13($input);\n-\t\t} elsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n-\t\t\t$output = rot13($input);\n-\t\t} else {\n-\t\t\tdie \"bad command '$command'\";\n-\t\t}\n-\n-\t\tif ( $pathname eq \"error.r\" ) {\n-\t\t\tprint $debug \"[ERROR]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=error\");\n-\t\t\tpacket_flush();\n-\t\t} elsif ( $pathname eq \"abort.r\" ) {\n-\t\t\tprint $debug \"[ABORT]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=abort\");\n-\t\t\tpacket_flush();\n-\t\t} elsif ( $command eq \"smudge\" and\n-\t\t\texists $DELAY{$pathname} and\n-\t\t\t$DELAY{$pathname}{\"requested\"} == 1 ) {\n-\t\t\tprint $debug \"[DELAYED]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_txt_write(\"status=delayed\");\n-\t\t\tpacket_flush();\n-\t\t\t$DELAY{$pathname}{\"requested\"} = 2;\n-\t\t\t$DELAY{$pathname}{\"output\"} = $output;\n-\t\t} else {\n-\t\t\tpacket_txt_write(\"status=success\");\n-\t\t\tpacket_flush();\n-\n-\t\t\tif ( $pathname eq \"${command}-write-fail.r\" ) {\n-\t\t\t\tprint $debug \"[WRITE FAIL]\\n\";\n-\t\t\t\t$debug->flush();\n-\t\t\t\tdie \"${command} write error\";\n-\t\t\t}\n-\n-\t\t\tprint $debug \"OUT: \" . length($output) . \" \";\n-\t\t\t$debug->flush();\n-\n-\t\t\twhile ( length($output) > 0 ) {\n-\t\t\t\tmy $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\t\tpacket_bin_write($packet);\n-\t\t\t\t# dots represent the number of packets\n-\t\t\t\tprint $debug \".\";\n-\t\t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n-\t\t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\t\t} else {\n-\t\t\t\t\t$output = \"\";\n-\t\t\t\t}\n-\t\t\t}\n-\t\t\tpacket_flush();\n-\t\t\tprint $debug \" [OK]\\n\";\n-\t\t\t$debug->flush();\n-\t\t\tpacket_flush();\n-\t\t}\n-\t}\n-}\ndiff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh\nindex c683e60007..7d956625ca 100755\n--- a/t/t2080-parallel-checkout-basics.sh\n+++ b/t/t2080-parallel-checkout-basics.sh\n@@ -230,12 +230,9 @@ test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading d\n # check the final report including sequential, parallel, and delayed entries\n # all at the same time. So we must have finer control of the parallel checkout\n # variables.\n-test_expect_success PERL '\"git checkout .\" report should not include failed entries' '\n-\twrite_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl &&\n-\n+test_expect_success '\"git checkout .\" report should not include failed entries' '\n \ttest_config_global filter.delay.process \\\n-\t\t\"\\\"$(pwd)/rot13-filter.pl\\\" --always-delay delayed.log clean smudge delay\" &&\n+\t\t\"test-tool rot13-filter --always-delay delayed.log clean smudge delay\" &&\n \ttest_config_global filter.delay.required true &&\n \ttest_config_global filter.cat.clean cat  &&\n \ttest_config_global filter.cat.smudge cat  &&\ndiff --git a/t/t2082-parallel-checkout-attributes.sh b/t/t2082-parallel-checkout-attributes.sh\nindex 2525457961..2df55b9405 100755\n--- a/t/t2082-parallel-checkout-attributes.sh\n+++ b/t/t2082-parallel-checkout-attributes.sh\n@@ -138,12 +138,9 @@ test_expect_success 'parallel-checkout and external filter' '\n # The delayed queue is independent from the parallel queue, and they should be\n # able to work together in the same checkout process.\n #\n-test_expect_success PERL 'parallel-checkout and delayed checkout' '\n-\twrite_script rot13-filter.pl \"$PERL_PATH\" \\\n-\t\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl &&\n-\n+test_expect_success 'parallel-checkout and delayed checkout' '\n \ttest_config_global filter.delay.process \\\n-\t\t\"\\\"$(pwd)/rot13-filter.pl\\\" --always-delay \\\"$(pwd)/delayed.log\\\" clean smudge delay\" &&\n+\t\t\"test-tool rot13-filter --always-delay \\\"$(pwd)/delayed.log\\\" clean smudge delay\" &&\n \ttest_config_global filter.delay.required true &&\n \n \techo \"abcd\" >original &&\n-- \n2.37.1\n\n"},{"id":"460341","messageId":"220801.86les8i495.gmgdl@evledraar.gmail.com","threadId":"58212","inReplyTo":"86e6baba460f4d0fce353d1fb6a0e18b57ecadaa.1659291025.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-01T11:33:36Z","receivedAt":"2022-08-01T11:37:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jul 31 2022, Matheus Tavares wrote:\n\n\n> +static char *rot13_usage = \"test-tool rot13-filter [--always-delay] <log path> <capabilities>\";\n> +\n> +int cmd__rot13_filter(int argc, const char **argv)\n> +{\n> +\tconst char **caps;\n> +\tint cap_count, i = 1;\n> +\tstruct strset remote_caps = STRSET_INIT;\n> +\n> +\tif (argc > 1 && !strcmp(argv[1], \"--always-delay\")) {\n> +\t\talways_delay = 1;\n> +\t\ti++;\n> +\t}\n> +\tif (argc - i < 2)\n> +\t\tusage(rot13_usage);\n> +\n> +\tlogfile = fopen(argv[i++], \"a\");\n> +\tif (!logfile)\n> +\t\tdie_errno(\"failed to open log file\");\n> +\n> +\tcaps = argv + i;\n> +\tcap_count = argc - i;\n\nSince you need to change every single caller consider just starting out\nwith parse_options() here instead of rolling your own parsing. You could\nuse it for --always-delay in any case, but you could also just add a\n--log-path and --capability (an OPT_STRING_LIST), so:\n\n\ttest-tool rot13-filter [--always-delay] --log-path=<path> [--capability <capbility]...\n\n> +\n> +\tfor (i = 0; i < cap_count; i++) {\n> +\t\tif (!strcmp(caps[i], \"clean\"))\n> +\t\t\thas_clean_cap = 1;\n> +\t\telse if (!strcmp(caps[i], \"smudge\"))\n> +\t\t\thas_smudge_cap = 1;\n\nIn any case, maybe BUG() in an \"else\" here with \"unknown capability\"?\n\n> +\tfclose(logfile);\n\nPerhaps check the return value & die_errno() if we fail to fclose()\n(happens e.g. if the disk fills up).\n"},{"id":"460342","messageId":"220801.86h72wi3kr.gmgdl@evledraar.gmail.com","threadId":"58212","inReplyTo":"86e6baba460f4d0fce353d1fb6a0e18b57ecadaa.1659291025.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-01T11:39:08Z","receivedAt":"2022-08-01T11:58:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jul 31 2022, Matheus Tavares wrote:\n\n> +static void reply_list_available_blobs_cmd(void)\n> +{\n> +\tstruct hashmap_iter iter;\n> +\tstruct strmap_entry *ent;\n> +\tstruct string_list_item *str_item;\n> +\tstruct string_list paths = STRING_LIST_INIT_NODUP;\n> +\n> +\t/* flush */\n> +\tif (packet_read_line(0, NULL))\n> +\t\tdie(\"bad list_available_blobs end\");\n\nShouldn't anything that's not an OS error (e.g. write error) be a BUG()\ninstead in this code? I.e. it would be a bug in our own testcode if we\nfeed the wrong data here, or if pkt-line doesn't work as we expect...\n\n> +\n> +\tstrmap_for_each_entry(&delay, &iter, ent) {\n> +\t\tstruct delay_entry *delay_entry = ent->value;\n> +\t\tif (!delay_entry->requested)\n> +\t\t\tcontinue;\n> +\t\tdelay_entry->count--;\n> +\t\tif (!strcmp(ent->key, \"invalid-delay.a\")) {\n> +\t\t\t/* Send Git a pathname that was not delayed earlier */\n> +\t\t\tpacket_write_fmt(1, \"pathname=unfiltered\");\n> +\t\t}\n> +\t\tif (!strcmp(ent->key, \"missing-delay.a\")) {\n> +\t\t\t/* Do not signal Git that this file is available */\n> +\t\t} else if (!delay_entry->count) {\n> +\t\t\tstring_list_append(&paths, ent->key);\n> +\t\t\tpacket_write_fmt(1, \"pathname=%s\", ent->key);\n> +\t\t}\n> +\t}\n> +\n> +\t/* Print paths in sorted order. */\n> +\tstring_list_sort(&paths);\n> +\tfor_each_string_list_item(str_item, &paths)\n> +\t\tfprintf(logfile, \" %s\", str_item->string);\n> +\tstring_list_clear(&paths, 0);\n> +\n> +\tpacket_flush(1);\n> +\n> +\tfprintf(logfile, \" [OK]\\n\");\n\nI think it should be called out in the commit message that this is not\nwhat the Perl version is doing, i.e. it does things like:\n\n\tprint $debug \" [OK]\\n\";\n\t$debug->flush();\n\nAfter having previously printed the equivalent of your\nfor_each_string_list_item() to the log file.\n\nIn Perl anything that uses PerlIO is subject to internal buffering,\nwhich doesn't have the same semantics as stdio buffering.\n\nI think in this case it won't matter, since you're not expecting to have\nconcurrent writers. You could even use fputc() here.\n\nBut a faithful reproduction of the Perl version would be something like\nappending the output here to a \"struct strbuf\", and then \"flushing\" it\nat the end when the perl version does a \"$debug->flush()\".\n\nI don't think that's worth the effort here, and we should just say that\nit doesn't matter. I just think we should note it. Thanks!\n"},{"id":"460374","messageId":"xmqqr11zpuhr.fsf@gitster.g","threadId":"58212","inReplyTo":"5ec95c7e696a49104322d243bee1d5f137bc8222.1659291025.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 1/3] t0021: avoid grepping for a Perl-specific string at filter output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T20:41:04Z","receivedAt":"2022-08-01T20:41:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares <matheus.bernardino@usp.br> writes:\n\n> This test sets the t0021/rot13-filter.pl script as a long-running\n> process filter for a git checkout command. It then expects the filter to\n> fail producing a specific error message at stderr. In the following\n> commits we are going to replace the script with a C test-tool helper,\n> but the test currently expects the error message in a Perl-specific\n> format. That is, when you call `die <msg>` in Perl, it emits\n> \"<msg> at - line 1.\" In preparation for the conversion, let's avoid the\n> Perl-specific part and only grep for <msg> itself.\n\nSounds sane.  I am a bit surprised that we check for messages from\nthe external filter tool, actually, rather than messages we would\nemit in response to an error by the filter tool, which ought to be\nmore stable no matter how the external tool expresses its failures.\n\nBut the posted change gets the job done perfectly fine, so it is OK.\n\nThanks.\n\n> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>\n> ---\n>  t/t0021-conversion.sh | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\n> index 1c840348bd..963b66e08c 100755\n> --- a/t/t0021-conversion.sh\n> +++ b/t/t0021-conversion.sh\n> @@ -735,7 +735,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f\n>  \t\trm -f debug.log &&\n>  \t\tgit checkout --quiet --no-progress . 2>git-stderr.log &&\n>  \n> -\t\tgrep \"smudge write error at\" git-stderr.log &&\n> +\t\tgrep \"smudge write error\" git-stderr.log &&\n>  \t\ttest_i18ngrep \"error: external filter\" git-stderr.log &&\n>  \n>  \t\tcat >expected.log <<-EOF &&\n"},{"id":"460384","messageId":"xmqqr11zoe6i.fsf@gitster.g","threadId":"58212","inReplyTo":"86e6baba460f4d0fce353d1fb6a0e18b57ecadaa.1659291025.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T21:18:45Z","receivedAt":"2022-08-01T21:18:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares <matheus.bernardino@usp.br> writes:\n\n> +static char *get_value(char *buf, size_t size, const char *key)\n> +{\n> +\tconst char *orig_buf = buf;\n> +\tint orig_size = (int)size;\n> +\n> +\tif (!skip_prefix_mem((const char *)buf, size, key, (const char **)&buf, &size) ||\n> +\t    !skip_prefix_mem((const char *)buf, size, \"=\", (const char **)&buf, &size) ||\n> +\t    !size)\n\nSo, skip_prefix_mem(), when successfully parses the prefix out,\nadvances buf[] to skip the prefix and shortens size by the same\namount, so buf[size] is pointing at the same byte.  The code wants\nto make sure buf[] begins with the \"<key>=\", skip that part, so\npresumably buf[] after the above part moves to the beginning of\n<value> in the \"<key>=<value>\" string?  It also wants to reject\n\"<key>=\", i.e. an empty string as the <value>?\n\n> +\t\tdie(\"expected key '%s', got '%.*s'\",\n> +\t\t    key, orig_size, orig_buf);\n> +\n> +\tbuf[size] = '\\0';\n\nI find this assignment somewhat strange, but primarily because it\nuses the updated buf[size] that ought to be pointing at the same\nbyte as the original buf[size].  Is this necessary because buf[size]\nupon the entry to this function does not necessarily have NUL there?\n\nReading ahead,\n\n * packet_key_val_read() feeds the buffer taken from\n   packet_read_line_gently(), so buf[size] should be NUL terminated\n   already.\n\n * read_capabilities() feeds the buffer taken from\n   packet_read_line(), so buf[size] should be NUL terminated\n   already.\n\n> +\treturn buf;\n> +}\n\nAnd the caller gets the byte position that begins the <value> part.\n\n> +static char *packet_key_val_read(const char *key)\n> +{\n> +\tint size;\n> +\tchar *buf;\n> +\tif (packet_read_line_gently(0, &size, &buf) < 0)\n> +\t\treturn NULL;\n> +\treturn xstrdup(get_value(buf, size, key));\n> +}\n\nThe returned value from get_value() is pointing into\npkt-line.c::packet_buffer[], so we return a copy to the caller,\nwhich takes the ownership.  OK.\n\n> +static inline void assert_remote_capability(struct strset *caps, const char *cap)\n> +{\n> +\tif (!strset_contains(caps, cap))\n> +\t\tdie(\"required '%s' capability not available from remote\", cap);\n> +}\n> +\n> +static void read_capabilities(struct strset *remote_caps)\n> +{\n> +\tfor (;;) {\n> +\t\tint size;\n> +\t\tchar *buf = packet_read_line(0, &size);\n> +\t\tif (!buf)\n> +\t\t\tbreak;\n> +\t\tstrset_add(remote_caps, get_value(buf, size, \"capability\"));\n> +\t}\n\nstrset_add() creates a copy of what get_value() borrowed from\npkt-line.c::packet_buffer[] here, which is good.\n\n> +\tassert_remote_capability(remote_caps, \"clean\");\n> +\tassert_remote_capability(remote_caps, \"smudge\");\n> +\tassert_remote_capability(remote_caps, \"delay\");\n> +}\n\n> +static void command_loop(void)\n> +{\n> +\tfor (;;) {\n> +\t\tchar *buf, *output;\n> +\t\tint size;\n> +\t\tchar *pathname;\n> +\t\tstruct delay_entry *entry;\n> +\t\tstruct strbuf input = STRBUF_INIT;\n> +\t\tchar *command = packet_key_val_read(\"command\");\n> +\n> +\t\tif (!command) {\n> +\t\t\tfprintf(logfile, \"STOP\\n\");\n> +\t\t\tbreak;\n> +\t\t}\n> +\t\tfprintf(logfile, \"IN: %s\", command);\n> +\n> +\t\tif (!strcmp(command, \"list_available_blobs\")) {\n> +\t\t\treply_list_available_blobs_cmd();\n> +\t\t\tfree(command);\n> +\t\t\tcontinue;\n> +\t\t}\n\nOK.\n\n> +\t\tpathname = packet_key_val_read(\"pathname\");\n> +\t\tif (!pathname)\n> +\t\t\tdie(\"unexpected EOF while expecting pathname\");\n> +\t\tfprintf(logfile, \" %s\", pathname);\n> +\n> +\t\t/* Read until flush */\n> +\t\twhile ((buf = packet_read_line(0, &size))) {\n> +\t\t\tif (!strcmp(buf, \"can-delay=1\")) {\n> +\t\t\t\tentry = strmap_get(&delay, pathname);\n> +\t\t\t\tif (entry && !entry->requested) {\n> +\t\t\t\t\tentry->requested = 1;\n> +\t\t\t\t} else if (!entry && always_delay) {\n> +\t\t\t\t\tadd_delay_entry(pathname, 1, 1);\n> +\t\t\t\t}\n\nThese are unnecessary {} around single statement blocks, but let's\nlet it pass in a test helper.\n\n> +\t\t\t} else if (starts_with(buf, \"ref=\") ||\n> +\t\t\t\t   starts_with(buf, \"treeish=\") ||\n> +\t\t\t\t   starts_with(buf, \"blob=\")) {\n> +\t\t\t\tfprintf(logfile, \" %s\", buf);\n> +\t\t\t} else {\n> +\t\t\t\t/*\n> +\t\t\t\t * In general, filters need to be graceful about\n> +\t\t\t\t * new metadata, since it's documented that we\n> +\t\t\t\t * can pass any key-value pairs, but for tests,\n> +\t\t\t\t * let's be a little stricter.\n> +\t\t\t\t */\n> +\t\t\t\tdie(\"Unknown message '%s'\", buf);\n> +\t\t\t}\n> +\t\t}\n> +\n> +\n> +\t\tread_packetized_to_strbuf(0, &input, 0);\n\nI do not see a need for double blank lines above.\n\n> +\t\tfprintf(logfile, \" %\"PRIuMAX\" [OK] -- \", (uintmax_t)input.len);\n> +\n> +\t\tentry = strmap_get(&delay, pathname);\n> +\t\tif (entry && entry->output) {\n> +\t\t\toutput = entry->output;\n> +\t\t} else if (!strcmp(pathname, \"error.r\") || !strcmp(pathname, \"abort.r\")) {\n> +\t\t\toutput = \"\";\n> +\t\t} else if (!strcmp(command, \"clean\") && has_clean_cap) {\n> +\t\t\toutput = rot13(input.buf);\n> +\t\t} else if (!strcmp(command, \"smudge\") && has_smudge_cap) {\n> +\t\t\toutput = rot13(input.buf);\n> +\t\t} else {\n> +\t\t\tdie(\"bad command '%s'\", command);\n> +\t\t}\n\nGood.  At this point, output all points into something and itself\ndoes not own the memory it is pointing at.\n\n> +\t\tif (!strcmp(pathname, \"error.r\")) {\n> +\t\t\tfprintf(logfile, \"[ERROR]\\n\");\n> +\t\t\tpacket_write_fmt(1, \"status=error\");\n> +\t\t\tpacket_flush(1);\n> +\t\t} else if (!strcmp(pathname, \"abort.r\")) {\n> +\t\t\tfprintf(logfile, \"[ABORT]\\n\");\n> +\t\t\tpacket_write_fmt(1, \"status=abort\");\n> +\t\t\tpacket_flush(1);\n> +\t\t} else if (!strcmp(command, \"smudge\") &&\n> +\t\t\t   (entry = strmap_get(&delay, pathname)) &&\n> +\t\t\t   entry->requested == 1) {\n> +\t\t\tfprintf(logfile, \"[DELAYED]\\n\");\n> +\t\t\tpacket_write_fmt(1, \"status=delayed\");\n> +\t\t\tpacket_flush(1);\n> +\t\t\tentry->requested = 2;\n> +\t\t\tif (entry->output != output) {\n> +\t\t\t\tfree(entry->output);\n> +\t\t\t\tentry->output = xstrdup(output);\n> +\t\t\t}\n> +\t\t} else {\n> +\t\t\tint i, nr_packets = 0;\n> +\t\t\tsize_t output_len;\n> +\t\t\tconst char *p;\n> +\t\t\tpacket_write_fmt(1, \"status=success\");\n> +\t\t\tpacket_flush(1);\n> +\n> +\t\t\tif (skip_prefix(pathname, command, &p) &&\n> +\t\t\t    !strcmp(p, \"-write-fail.r\")) {\n> +\t\t\t\tfprintf(logfile, \"[WRITE FAIL]\\n\");\n> +\t\t\t\tdie(\"%s write error\", command);\n> +\t\t\t}\n> +\n> +\t\t\toutput_len = strlen(output);\n> +\t\t\tfprintf(logfile, \"OUT: %\"PRIuMAX\" \", (uintmax_t)output_len);\n> +\n> +\t\t\tif (write_packetized_from_buf_no_flush_count(output,\n> +\t\t\t\toutput_len, 1, &nr_packets))\n> +\t\t\t\tdie(\"failed to write buffer to stdout\");\n> +\t\t\tpacket_flush(1);\n> +\n> +\t\t\tfor (i = 0; i < nr_packets; i++)\n> +\t\t\t\tfprintf(logfile, \".\");\n> +\t\t\tfprintf(logfile, \" [OK]\\n\");\n> +\n> +\t\t\tpacket_flush(1);\n> +\t\t}\n> +\t\tfree(pathname);\n> +\t\tstrbuf_release(&input);\n> +\t\tfree(command);\n> +\t}\n> +}\n\nOK, at this point we are done with pathname and command so we can\nfree them for the next round.  input was used as a scratch buffer\nand we are done with it, too.\n\nLooking good.\n\nThanks.\n"},{"id":"460396","messageId":"CAHd-oW4OWLqMXwJPAeb_UN6Rxj-8KXTnRyGfm_tfLTrGqnZo-A@mail.gmail.com","threadId":"58212","inReplyTo":"xmqqr11zoe6i.fsf@gitster.g","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-08-02T00:13:09Z","receivedAt":"2022-08-02T00:13:28Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Mon, Aug 1, 2022 at 6:18 PM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Matheus Tavares <matheus.bernardino@usp.br> writes:\n>\n> > +             die(\"expected key '%s', got '%.*s'\",\n> > +                 key, orig_size, orig_buf);\n> > +\n> > +     buf[size] = '\\0';\n>\n> I find this assignment somewhat strange, but primarily because it\n> uses the updated buf[size] that ought to be pointing at the same\n> byte as the original buf[size].  Is this necessary because buf[size]\n> upon the entry to this function does not necessarily have NUL there?\n>\n> Reading ahead,\n>\n>  * packet_key_val_read() feeds the buffer taken from\n>    packet_read_line_gently(), so buf[size] should be NUL terminated\n>    already.\n>\n>  * read_capabilities() feeds the buffer taken from\n>    packet_read_line(), so buf[size] should be NUL terminated\n>    already.\n>\n> > +     return buf;\n> > +}\n>\n> And the caller gets the byte position that begins the <value> part.\n\nGood point. I'll remove the buf[size] = '\\0' assignment.\n\n> > +                             if (entry && !entry->requested) {\n> > +                                     entry->requested = 1;\n> > +                             } else if (!entry && always_delay) {\n> > +                                     add_delay_entry(pathname, 1, 1);\n> > +                             }\n>\n> These are unnecessary {} around single statement blocks, but let's\n> let it pass in a test helper.\n> > [...]\n> > +                             die(\"Unknown message '%s'\", buf);\n> > +                     }\n> > +             }\n> > +\n> > +\n> > +             read_packetized_to_strbuf(0, &input, 0);\n>\n> I do not see a need for double blank lines above.\n\nOops, I will fix these too. Thanks.\n"},{"id":"460397","messageId":"CAHd-oW6GLf=4VxAvMy6c9jrGx1zcSHbe_NKbAUg7wvNBPOmEXw@mail.gmail.com","threadId":"58212","inReplyTo":"220801.86les8i495.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-08-02T00:16:50Z","receivedAt":"2022-08-02T00:17:07Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Mon, Aug 1, 2022 at 8:37 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n>\n> On Sun, Jul 31 2022, Matheus Tavares wrote:\n> >\n> > +\n> > +     caps = argv + i;\n> > +     cap_count = argc - i;\n>\n> Since you need to change every single caller consider just starting out\n> with parse_options() here instead of rolling your own parsing. You could\n> use it for --always-delay in any case, but you could also just add a\n> --log-path and --capability (an OPT_STRING_LIST), so:\n>\n>         test-tool rot13-filter [--always-delay] --log-path=<path> [--capability <capbility]...\n\nAh, makes sense. Thanks\n\n> > +\n> > +     for (i = 0; i < cap_count; i++) {\n> > +             if (!strcmp(caps[i], \"clean\"))\n> > +                     has_clean_cap = 1;\n> > +             else if (!strcmp(caps[i], \"smudge\"))\n> > +                     has_smudge_cap = 1;\n>\n> In any case, maybe BUG() in an \"else\" here with \"unknown capability\"?\n\nYup, will do.\n\n> > +     fclose(logfile);\n>\n> Perhaps check the return value & die_errno() if we fail to fclose()\n> (happens e.g. if the disk fills up).\n\nSure. Thanks.\n"},{"id":"460882","messageId":"q7o86qo0-9618-p26p-q6q1-8n461qsqpq75@tzk.qr","threadId":"58212","inReplyTo":"CAHd-oW6LZay=MX2FdFjgTh1pjE=g-XTm63mGWuMhHd=-N=tXRA@mail.gmail.com","subject":"Re: [PATCH v2] t/t0021: convert the rot13-filter.pl script to C","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-09T09:36:16Z","receivedAt":"2022-08-09T09:36:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Sat, 30 Jul 2022, Matheus Tavares wrote:\n\n> On Thu, Jul 28, 2022 at 1:58 PM Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n> >\n> > > On Sun, 24 Jul 2022, Matheus Tavares wrote:\n> > >\n> > > +static void command_loop(void)\n> > > +{\n> > > +     while (1) {\n> > > +             char *command = packet_key_val_read(\"command\");\n> > > +             if (!command) {\n> > > +                     fprintf(logfile, \"STOP\\n\");\n> > > +                     break;\n> > > +             }\n> > > +             fprintf(logfile, \"IN: %s\", command);\n> >\n> > We will also need to `fflush(logfile)` here, to imitate the Perl script's\n> > behavior more precisely.\n>\n> I was somewhat intrigued as to why the flushes were needed in the Perl\n> script. But reading [1] and [2], now, it seems to have been an\n> oversight.\n>\n> That is, Eric suggested splictily flushing stdout because it is a\n> pipe, but the author ended up erroneously disabling autoflush for\n> stdout too, so that's why we needed the flushes there. They later\n> acknowledged that and said that they would re-enabled it (see [2]),\n> but it seems to have been forgotten. So I think we can safely drop the\n> flush calls.\n>\n> [1]: http://public-inbox.org/git/20160723072721.GA20875%40starla/\n> [2]: https://lore.kernel.org/git/7F1F1A0E-8FC3-4FBD-81AA-37786DE0EF50@gmail.com/\n\nI am somewhat weary of introducing a change of behavior while\nreimplementing a Perl script in C at the same time, but in this instance I\nthink that the benefit of _not_ touching the `pkt-line.c` code is a\nconvincing reason to do so.\n\n> > > +\n> > > +             if (!strcmp(command, \"list_available_blobs\")) {\n> > > +                     struct hashmap_iter iter;\n> > > +                     struct strmap_entry *ent;\n> > > +                     struct string_list_item *str_item;\n> > > +                     struct string_list paths = STRING_LIST_INIT_NODUP;\n> > > +\n> > > +                     /* flush */\n> > > +                     if (packet_read_line(0, NULL))\n> > > +                             die(\"bad list_available_blobs end\");\n> > > +\n> > > +                     strmap_for_each_entry(&delay, &iter, ent) {\n> > > +                             struct delay_entry *delay_entry = ent->value;\n> > > +                             if (!delay_entry->requested)\n> > > +                                     continue;\n> > > +                             delay_entry->count--;\n> > > +                             if (!strcmp(ent->key, \"invalid-delay.a\")) {\n> > > +                                     /* Send Git a pathname that was not delayed earlier */\n> > > +                                     packet_write_fmt(1, \"pathname=unfiltered\");\n> > > +                             }\n> > > +                             if (!strcmp(ent->key, \"missing-delay.a\")) {\n> > > +                                     /* Do not signal Git that this file is available */\n> > > +                             } else if (!delay_entry->count) {\n> > > +                                     string_list_insert(&paths, ent->key);\n> > > +                                     packet_write_fmt(1, \"pathname=%s\", ent->key);\n> > > +                             }\n> > > +                     }\n> > > +\n> > > +                     /* Print paths in sorted order. */\n> >\n> > The Perl script does not order them specifically. Do we really have to do\n> > that here?\n>\n> It actually prints them in sorted order:\n>\n>         foreach my $pathname ( sort keys %DELAY )\n\nWhoops, sorry for missing that!\n\n> > > +                             fprintf(logfile, \" [OK]\\n\");\n> > > +\n> > > +                             packet_flush(1);\n> > > +                             strbuf_release(&sb);\n> > > +                     }\n> > > +                     free(pathname);\n> > > +                     strbuf_release(&input);\n> > > +             }\n> > > +             free(command);\n> > > +     }\n> > > +}\n> > > [...]\n> > > +static void packet_initialize(const char *name, int version)\n> > > +{\n> > > +     struct strbuf sb = STRBUF_INIT;\n> > > +     int size;\n> > > +     char *pkt_buf = packet_read_line(0, &size);\n> > > +\n> > > +     strbuf_addf(&sb, \"%s-client\", name);\n> > > +     if (!pkt_buf || strncmp(pkt_buf, sb.buf, size))\n> >\n> > We do not need the flexibility of the Perl package, where `name` is a\n> > parameter. We can hard-code `git-filter-client` here. I.e. something like\n> > this:\n> >\n> >         if (!pkt_buf || size != 17 ||\n> >             strncmp(pkt_buf, \"git-filter-client\", 17))\n>\n> Good idea! Thanks. Perhaps, can't we do:\n>\n>         if (!pkt_buf || strncmp(pkt_buf, \"git-filter-client\", size))\n>\n> to avoid the hard-coded and possibly error-prone 17?\n\nI am afraid that this is not idempotent. If `pkt_buf` is \"git\" and `size`\nis 3, then the suggested `strncmp()` would return 0, but we would want it\nto be non-zero.\n\nThe best way to avoid the hard-coded 17 would be to introduce a local\nconstant and use `strlen()` on it (which modern compilers would evaluate\nalready at compile time).\n\n> > > +             die(\"bad initialize: '%s'\", xstrndup(pkt_buf, size));\n> > > +\n> > > +     strbuf_reset(&sb);\n> > > +     strbuf_addf(&sb, \"version=%d\", version);\n>\n> Thanks for a very detailed review and great suggestions!\n\nThank you for your contribution that is very much relevant to my\ninterests!\n\nCiao,\nDscho\n"},{"id":"460883","messageId":"psr5o1r8-ro70-24q1-7o01-8571n1802s18@tzk.qr","threadId":"58212","inReplyTo":"CAHd-oW6GLf=4VxAvMy6c9jrGx1zcSHbe_NKbAUg7wvNBPOmEXw@mail.gmail.com","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-09T09:45:16Z","receivedAt":"2022-08-09T09:45:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Mon, 1 Aug 2022, Matheus Tavares wrote:\n\n> On Mon, Aug 1, 2022 at 8:37 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> >\n> > On Sun, Jul 31 2022, Matheus Tavares wrote:\n> > >\n> > > +\n> > > +     for (i = 0; i < cap_count; i++) {\n> > > +             if (!strcmp(caps[i], \"clean\"))\n> > > +                     has_clean_cap = 1;\n> > > +             else if (!strcmp(caps[i], \"smudge\"))\n> > > +                     has_smudge_cap = 1;\n> >\n> > In any case, maybe BUG() in an \"else\" here with \"unknown capability\"?\n>\n> Yup, will do.\n\nPlease don't, the suggestion is unsound.\n\nThe idea here is to find out whether the command-line listed the \"clean\"\nand/or the \"smudge\" capabilities, ignoring all others for the moment.\n\nTo error out here with a BUG() would most likely break the invocation\nin t0021 where we also pass the `delay` capability.\n\nCiao,\nDscho\n"},{"id":"460884","messageId":"439p713r-32o4-5187-n8nn-r81n3007s4pp@tzk.qr","threadId":"58212","inReplyTo":"xmqqr11zoe6i.fsf@gitster.g","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-09T10:00:02Z","receivedAt":"2022-08-09T10:00:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 1 Aug 2022, Junio C Hamano wrote:\n\n> Matheus Tavares <matheus.bernardino@usp.br> writes:\n>\n> > +\t\t/* Read until flush */\n> > +\t\twhile ((buf = packet_read_line(0, &size))) {\n> > +\t\t\tif (!strcmp(buf, \"can-delay=1\")) {\n> > +\t\t\t\tentry = strmap_get(&delay, pathname);\n> > +\t\t\t\tif (entry && !entry->requested) {\n> > +\t\t\t\t\tentry->requested = 1;\n> > +\t\t\t\t} else if (!entry && always_delay) {\n> > +\t\t\t\t\tadd_delay_entry(pathname, 1, 1);\n> > +\t\t\t\t}\n>\n> These are unnecessary {} around single statement blocks, but let's\n> let it pass in a test helper.\n\nI would like to encourage you to think of ways how this project could\navoid the cost (mental space, reviewer time, back and forth between\ncontributor and reviewer) of such trivial code formatting issues.\n\nMy favored solution would be to adjust the code formatting rules in Git to\nsuch an extent that it can be completely automated, whether via a\n`clang-format-diff` rule [*1*] or via an adapted `checkpatch` [*2*] or via\nsomething that is modeled after cURL's `checksrc` script [*3*].\n\nIt costs us too much time, and is too annoying all around, having to spend\nso many brain cycles on code style (which people like me find much less\ninteresting than the actual, functional changes).\n\nI'd much rather focus on the implementation of the rot13 filter and\npotentially how this patch could give rise to even broader enhancements to\nGit's source code that eventually have a user-visible, positive impact.\n\nCiao,\nDscho\n\nFootnote *1*: https://lore.kernel.org/git/YstJl+5BPyR5RWnR@tapette.crustytoothpaste.net/\nFootnote *2*: https://lore.kernel.org/git/xmqqbktvl0s4.fsf@gitster.g/\nFootnote *3*: https://github.com/curl/curl/blob/master/scripts/checksrc.pl\n"},{"id":"460885","messageId":"663onqs0-465s-023o-9s25-p2193ss5so59@tzk.qr","threadId":"58212","inReplyTo":"xmqqr11zoe6i.fsf@gitster.g","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-09T10:37:01Z","receivedAt":"2022-08-09T10:37:17Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 1 Aug 2022, Junio C Hamano wrote:\n\n>  * read_capabilities() feeds the buffer taken from\n>    packet_read_line(), so buf[size] should be NUL terminated\n>    already.\n\nCould you help me agree?\n\nIn `packet_read_line()`, we call `packet_read()` with the\n`PACKET_READ_CHOMP_NEWLINE` option, but we do not NUL-terminate the\nbuffer.\n\nSee https://github.com/git/git/blob/v2.37.1/pkt-line.c#L488-L494\n\nIn `packet_read()`, we call `packet_read_with_status()`, but do not\nNUL-terminate the buffer.\n\nSee https://github.com/git/git/blob/v2.37.1/pkt-line.c#L478-L486\n\nIn `packet_read_with_status()`, I see that we call `get_packet_data()`\nwhich does not NUL-terminate the buffer. Then we parse the length via\n`packet_length()` which does not NUL-terminate the buffer.\n\nThen, crucially, if the packet length is smaller than 3, we set the length\nthat is returned to 0 and return early indicating the conditions\n`PACKET_READ_FLUSH`, `PACKET_READ_DELIM`, or `PACKET_READ_RESPONSE_END`,\nwhich are ignored by `packet_read()`.\n\nIn this instance, the buffer is not NUL-terminated, I think. But if you\nsee that I missed something, I would like to know.\n\nSee https://github.com/git/git/blob/v2.37.1/pkt-line.c#L399-L476\n\nAnd yes, in the case that there is a regular payload,\nhttps://github.com/git/git/blob/v2.37.1/pkt-line.c#L456 NUL-terminates the\nbuffer.\n\nAnd the proposed `get_value()` function would avoid returning a not\nNUL-terminated buffer by virtue of using the `skip_prefix_mem()` function\nwith a non-empty prefix but a zero length buffer.\n\nTherefore it is _still_ safe to skip the `buf[size] = '\\0';` assignment\ndespite what I wrote above, even if it adds yet another piece of code to\nGit's source code which is harder than necessary to reason about.\n\nAfter all, it took me half an hour to research and write up this mail,\nwhen reading `buf[size] = '\\0';` would have taken all of two seconds to\nverify that the code is safe.\n\nCiao,\nDscho\n"},{"id":"460886","messageId":"9239s8np-69ss-n035-53s9-869s42p9srno@tzk.qr","threadId":"58212","inReplyTo":"86e6baba460f4d0fce353d1fb6a0e18b57ecadaa.1659291025.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-09T10:47:03Z","receivedAt":"2022-08-09T10:47:17Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matheus,\n\nOn Sun, 31 Jul 2022, Matheus Tavares wrote:\n\n> diff --git a/pkt-line.c b/pkt-line.c\n> index 8e43c2def4..ce4e73b683 100644\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -309,7 +309,8 @@ int write_packetized_from_fd_no_flush(int fd_in, int fd_out)\n>  \treturn err;\n>  }\n>\n> -int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out)\n> +int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,\n> +\t\t\t\t\t     int fd_out, int *packet_counter)\n>  {\n>  \tint err = 0;\n>  \tsize_t bytes_written = 0;\n> @@ -324,6 +325,8 @@ int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_ou\n>  \t\t\tbreak;\n>  \t\terr = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);\n>  \t\tbytes_written += bytes_to_write;\n> +\t\tif (packet_counter)\n> +\t\t\t(*packet_counter)++;\n\nThe only reason why we do this here is to try to imitate the Perl script\nthat prints out a dot for every packet written, right?\n\nBut the Perl script wrote out those dots immediately and individually, not\nin one go after writing all the packets.\n\nUnless the tests rely on the dots in the output, I would therefore\nrecommend to simply scrap this functionality (and to write about it in the\ncommit message, with the rationale that it does not fit into the current C\ncode's paradigms and would require intrusive changes of questionable\nbenefit) and avoid touching `pkt-line.[ch]` altogether.\n\n> [...]\n> diff --git a/pkt-line.h b/pkt-line.h\n> [...]\n> +static void packet_initialize(void)\n> +{\n> +\tint size;\n> +\tchar *pkt_buf = packet_read_line(0, &size);\n> +\n> +\tif (!pkt_buf || strncmp(pkt_buf, \"git-filter-client\", size))\n> +\t\tdie(\"bad initialize: '%s'\", xstrndup(pkt_buf, size));\n> +\n> +\tpkt_buf = packet_read_line(0, &size);\n> +\tif (!pkt_buf || strncmp(pkt_buf, \"version=2\", size))\n> +\t\tdie(\"bad version: '%.*s'\", (int)size, pkt_buf);\n\nThis would mistake a packet `v` for being valid.\n\nJunio pointed out in his review that `packet_read_line()` already\nNUL-terminates the buffer (except when it returns `NULL`), therefore we\ncan write this instead:\n\n\tif (!pkt_buf || strcmp(pkt_buf, \"version=2\"))\n\nLikewise with `\"git-filter-client\"`.\n\n> +\n> +\tpkt_buf = packet_read_line(0, &size);\n> +\tif (pkt_buf)\n> +\t\tdie(\"bad version end: '%.*s'\", (int)size, pkt_buf);\n> +\n> +\tpacket_write_fmt(1, \"git-filter-server\");\n> +\tpacket_write_fmt(1, \"version=2\");\n> +\tpacket_flush(1);\n> +}\n> +\n> +static char *rot13_usage = \"test-tool rot13-filter [--always-delay] <log path> <capabilities>\";\n> +\n> +int cmd__rot13_filter(int argc, const char **argv)\n> +{\n> +\tconst char **caps;\n> +\tint cap_count, i = 1;\n> +\tstruct strset remote_caps = STRSET_INIT;\n> +\n> +\tif (argc > 1 && !strcmp(argv[1], \"--always-delay\")) {\n> +\t\talways_delay = 1;\n> +\t\ti++;\n> +\t}\n\nThis is so much simpler to read than if it used `parse_options()`,\ntherefore I think that this is good as-is.\n\nIt is probably obvious that I did not spend as much time on reviewing this\nround as I did the previous time (after all, if one spends three hours\nhere and three hours there, pretty soon one ends up having missed lunch\nbefore knowing it). However, it is equally obvious that you did a great\njob addressing my review of the previous round.\n\nThank you,\nDscho\n"},{"id":"461015","messageId":"xmqqtu6kkkr6.fsf@gitster.g","threadId":"58212","inReplyTo":"439p713r-32o4-5187-n8nn-r81n3007s4pp@tzk.qr","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-10T18:37:33Z","receivedAt":"2022-08-10T18:37:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> I would like to encourage you to think of ways how this project could\n> avoid the cost (mental space, reviewer time, back and forth between\n> contributor and reviewer) of such trivial code formatting issues.\n\nI do not need your encouragement.  I am sure the submitter could\nhave run clang-format or checkpatch.pl or whatever and noticed the\nissue.  Small style diversions in submitted patches are distracting\nenough to prevent me from concentrating on and noticing problems in\nthe more important aspects like correctness and leakiness.  That is\nwhy people get formatting issues pointed out and CodingGuidelines\ntalks about styles.\n\nCheckpatch is OK, but IIRC, you cannot ask to check \"only the code I\nchanged in this patch\" to clang-format, which may be the show\nstopper.  Otherwise, I would quite welcome an automated \"pre-flight\"\nautomation, like \"make\" target, that submitters can use and GGG can\nhelp them use.\n\nThanks.\n\n"},{"id":"461023","messageId":"xmqq4jyjlvl3.fsf@gitster.g","threadId":"58212","inReplyTo":"xmqqtu6kkkr6.fsf@gitster.g","subject":"Re: [PATCH v3 2/3] t0021: implementation the rot13-filter.pl script in C","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-10T19:58:16Z","receivedAt":"2022-08-10T19:58:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Checkpatch is OK, but IIRC, you cannot ask to check \"only the code I\n> changed in this patch\" to clang-format, which may be the show\n> stopper.  Otherwise, I would quite welcome an automated \"pre-flight\"\n> automation, like \"make\" target, that submitters can use and GGG can\n> help them use.\n\nLet me step a bit back.  I do not think any automated tool would be\nfree of false positives, so it is OK to configure the tool loose and\nhave \"judgement case\" still be dealt by human reviewer, but if the\nautomation is overly strict, that would probably waste submitters'\ntime too much.\n\nYou would need to accept that the new contributors are human and are\ncapable of learning and configuring editors on their end, and after\nthey get reminded of the style rules once or twice and they get used\nto the process, they would also help coaching yet even newer\ncontributors.\n\nI personally feel that the level of style issues that need to be\npointed out among the recent list traffic is not overly excessive.\n\nThanks.\n"}]}