{"thread":{"id":"57365","subject":"[PATCH 0/2] Add cat-file --batch-command flag","startedAt":"2022-02-03T19:08:14Z","lastAt":"2022-02-22T13:20:46Z","messageCount":97,"participants":["John Cai via GitGitGadget","Junio C Hamano","John Cai","Eric Sunshine","Ævar Arnfjörð Bjarmason","Phillip Wood","Jonathan Tan","Christian Couder"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"447640","messageId":"pull.1212.git.git.1643915286.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":null,"subject":"[PATCH 0/2] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-03T19:08:04Z","receivedAt":"2022-02-03T19:08:14Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThe latest change since [C] was replacing the flush command with a series of\ncommands to allow a read \"session\", so we can queue up a bunch of objects\nand then get either their info or contents, then end by flushing. This is to\naddress Phillip's concern about deadlocks if we allow the caller to call\nflush whenever they want.\n\nThis patch series has two parts:\n\n 1. preparation patch to rename a variable\n 2. logic to handle --batch-command flag, and adding different commands\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (2):\n cat-file.c: rename cmdmode to mode\n catfile.c: add --batch-command mode\n\n Documentation/git-cat-file.txt | 27 +++++\n builtin/cat-file.c | 203 +++++++++++++++++++++++++++++----\n t/t1006-cat-file.sh | 46 +++++++-\n 3 files changed, 251 insertions(+), 25 deletions(-)\n\n\nbase-commit: 5d01301f2b865aa8dba1654d3f447ce9d21db0b5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v1\nPull-Request: https://github.com/git/git/pull/1212\n-- \ngitgitgadget\n"},{"id":"447641","messageId":"86df0c9e4df34566c10870e06865af536504a6af.1643915286.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.git.git.1643915286.gitgitgadget@gmail.com","subject":"[PATCH 1/2] cat-file.c: rename cmdmode to mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-03T19:08:05Z","receivedAt":"2022-02-03T19:08:15Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nTo prepare for a new flag --batch-command, we will add a flag that\nindicates whether or not an interactive command mode will be used\nthat reads commands and arguments off of stdin.\n\nAn intuitive name for this flag would be \"command\", which can get\nconfusing with the already existing cmdmode.\n\nRename cmdmode->mode to prepare for this change.\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 19 +++++++++----------\n 1 file changed, 9 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex d94050e6c18..858bca208ff 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -306,19 +306,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -330,7 +330,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid mode: %c\", opt->mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -533,7 +533,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -695,10 +695,9 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \n \tbatch.buffer_output = -1;\n \targc = parse_options(argc, argv, prefix, options, cat_file_usage, 0);\n-\n \tif (opt) {\n \t\tif (batch.enabled && (opt == 'c' || opt == 'w'))\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.mode = opt;\n \t\telse if (argc == 1)\n \t\t\tobj_name = argv[0];\n \t\telse\n@@ -712,9 +711,9 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tusage_with_options(cat_file_usage, options);\n \t}\n \tif (batch.enabled) {\n-\t\tif (batch.cmdmode != opt || argc)\n+\t\tif (batch.mode != opt || argc)\n \t\t\tusage_with_options(cat_file_usage, options);\n-\t\tif (batch.cmdmode && batch.all_objects)\n+\t\tif (batch.mode && batch.all_objects)\n \t\t\tdie(\"--batch-all-objects cannot be combined with \"\n \t\t\t \"--textconv nor with --filters\");\n \t}\n-- \ngitgitgadget\n\n"},{"id":"447642","messageId":"ebd2a1356017905245744babf32c5b78a0af1639.1643915286.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.git.git.1643915286.gitgitgadget@gmail.com","subject":"[PATCH 2/2] catfile.c: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-03T19:08:06Z","receivedAt":"2022-02-03T19:08:17Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd new flag --batch-command that will accept commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldnt need to keep both\nprocesses around, and instead just have one process where we can flip\nbetween getting object info, and getting object contents. This means we\ncan get rid of roughly half of long lived git cat-file processes. This\ncan lead to huge savings since on a given server there could be hundreds\nof git cat-file processes running.\n\ngit cat-file --batch-command\n\nwould enter an interactive command mode whereby the user can enter in\ncommands and their arguments:\n\n [arg1] [arg2] NL\n\nThis patch adds the basic structure for add command which can be\nextended in the future to add more commands.\n\nThis patch also adds the following commands:\n\ncontents NL\ninfo NL\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes a argument and prints out the object\nmetadata.\n\nIn addition, we need a set of commands that enable a \"read session\".\n\nWhen a separate process (A) is connected to a git cat-file process (B)\nand is interactively writing to and reading from it in --buffer mode,\n(A) needs to be able to know when the buffer is flushed to stdout.\nCurrently, from (A)'s perspective, the only way is to either 1. exit\n(B)'s process or 2. send an invalid object to stdin. 1. is not ideal\nfrom a performance perspective as it will require spawning a new cat-file\nprocess each time, and 2. is hacky and not a good long term solution.\n\nWith the following commands, process (A) can begin a \"session\" and\nsend a list of object names over stdin. When \"get contents\" or \"get info\"\nis issued, this list of object names will be fed into batch_one_object()\nto retrieve either info or contents. Finally an fflush() will be called\nto end the session.\n\nbegin NL\nget contents NL\nget info NL\n\nThese can be used in the following way:\n\nbegin\n\n\n\n\nget info\n\nbegin\n\n\n\n\nget contents\n\nWith this mechanism, process (A) can be guaranteed to receive all of the\noutput even in --buffer mode.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 27 +++++\n builtin/cat-file.c | 184 ++++++++++++++++++++++++++++++---\n t/t1006-cat-file.sh | 46 ++++++++-\n 3 files changed, 242 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex 27b27e2b300..d1ba0b12e54 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -90,6 +90,33 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+\tEnter a command mode that reads commands and arguments from stdin.\n+\tMay not be combined with any other options or arguments except\n+\t`--textconv` or `--filters`, in which case the input lines also need to\n+\tspecify the path, separated by whitespace. See the section\n+\t`BATCH OUTPUT` below for details.\n+\n+contents ::\n+\tPrint object contents for object reference \n+\n+info ::\n+\tPrint object info for object reference \n+\n+begin::\n+\tBegins a session to read object names off of stdin. A session can be\n+\tterminated with `get contents` or `get info`.\n+\n+get contents::\n+\tAfter a read session is begun with the `begin` command, and object\n+\tnames have been fed into stdin, end the session and retrieve contents of\n+\tall the objects requested.\n+\n+get info::\n+\tAfter a read session is begun with the `begin` command, and object\n+\tnames have been fed into stdin, end the session and retrieve info of\n+\tall the objects requested.\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 858bca208ff..29d5cd6857b 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -26,6 +26,7 @@ struct batch_options {\n \tint unordered;\n \tint mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n+\tint command;\n };\n \n static const char *force_path;\n@@ -512,6 +513,151 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+static void parse_cmd_object(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data,\n+\t\t\t struct string_list revs)\n+{\n+\topt->print_contents = 1;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data,\n+\t\t\t struct string_list revs)\n+{\n+\topt->print_contents = 0;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_begin(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data,\n+\t\t\t struct string_list revs)\n+{\n+\t/* nothing needs to be done here */\n+}\n+\n+static void parse_cmd_get(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data,\n+\t\t\t struct string_list revs)\n+{\n+\tstruct string_list_item *item;\n+\tfor_each_string_list_item(item, &revs) {\n+\t\tbatch_one_object(item->string, output, opt, data);\n+\t}\n+\tif (opt->buffer_output)\n+\t\tfflush(stdout);\n+}\n+static void parse_cmd_get_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data,\n+\t\t\t struct string_list revs)\n+{\n+\topt->print_contents = 0;\n+\tparse_cmd_get(opt, line, output, data, revs);\n+}\n+\n+static void parse_cmd_get_objects(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data,\n+\t\t\t struct string_list revs)\n+{\n+\topt->print_contents = 1;\n+\tparse_cmd_get(opt, line, output, data, revs);\n+\tif (opt->buffer_output)\n+\t\tfflush(stdout);\n+}\n+\n+enum batch_state {\n+\tBATCH_STATE_COMMAND,\n+\tBATCH_STATE_INPUT,\n+};\n+\n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *,\n+\t\t\t struct string_list revs);\n+\n+static const struct parse_cmd {\n+\tconst char *prefix;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+\tenum batch_state next_state;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_object, 1, BATCH_STATE_COMMAND},\n+\t{ \"info\", parse_cmd_info, 1, BATCH_STATE_COMMAND},\n+\t{ \"begin\", parse_cmd_begin, 0, BATCH_STATE_INPUT},\n+\t{ \"get info\", parse_cmd_get_info, 0, BATCH_STATE_COMMAND},\n+\t{ \"get contents\", parse_cmd_get_objects, 0, BATCH_STATE_COMMAND},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tenum batch_state state = BATCH_STATE_COMMAND;\n+\tstruct string_list revs = STRING_LIST_INIT_DUP;\n+\n+\t/* Read each line dispatch its command */\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p, *cmd_end;\n+\n+\t\tif (state == BATCH_STATE_COMMAND) {\n+\t\t\tif (*input.buf == '\\n')\n+\t\t\t\tdie(\"empty command in input\");\n+\t\t\telse if (isspace(*input.buf))\n+\t\t\t\tdie(\"whitespace before command: %s\", input.buf);\n+\t\t}\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tconst char *prefix = commands[i].prefix;\n+\t\t\tchar c;\n+\t\t\tif (!skip_prefix(input.buf, prefix, &cmd_end))\n+\t\t\t\tcontinue;\n+\t\t\t/*\n+\t\t\t * If the command has arguments, verify that it's\n+\t\t\t * followed by a space. Otherwise, it shall be followed\n+\t\t\t * by a line terminator.\n+\t\t\t */\n+\t\t\tc = commands[i].takes_args ? ' ' : '\\n';\n+\t\t\tif (*cmd_end && *cmd_end != c)\n+\t\t\t\tdie(\"arguments invalid for command: %s\", commands[i].prefix);\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args)\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (input.buf[input.len - 1] == '\\n')\n+\t\t\tinput.buf[--input.len] = '\\0';\n+\n+\t\tif (state == BATCH_STATE_INPUT && !cmd){\n+\t\t\tstring_list_append(&revs, input.buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(\"unknown command: %s\", input.buf);\n+\n+\t\tstate = cmd->next_state;\n+\t\tcmd->fn(opt, p, output, data, revs);\n+\t}\n+\tstrbuf_release(&input);\n+\tstring_list_clear(&revs, 0);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -519,6 +665,7 @@ static int batch_objects(struct batch_options *opt)\n \tstruct expand_data data;\n \tint save_warning;\n \tint retval = 0;\n+\tconst int command = opt->command;\n \n \tif (!opt->format)\n \t\topt->format = \"%(objectname) %(objecttype) %(objectsize)\";\n@@ -594,22 +741,25 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n-\twhile (strbuf_getline(&input, stdin) != EOF) {\n-\t\tif (data.split_on_whitespace) {\n-\t\t\t/*\n-\t\t\t * Split at first whitespace, tying off the beginning\n-\t\t\t * of the string and saving the remainder (or NULL) in\n-\t\t\t * data.rest.\n-\t\t\t */\n-\t\t\tchar *p = strpbrk(input.buf, \" \\t\");\n-\t\t\tif (p) {\n-\t\t\t\twhile (*p && strchr(\" \\t\", *p))\n-\t\t\t\t\t*p++ = '\\0';\n+\tif (command)\n+\t\tbatch_objects_command(opt, &output, &data);\n+\telse {\n+\t\twhile (strbuf_getline(&input, stdin) != EOF) {\n+\t\t\tif (data.split_on_whitespace) {\n+\t\t\t\t/*\n+\t\t\t\t * Split at first whitespace, tying off the beginning\n+\t\t\t\t * of the string and saving the remainder (or NULL) in\n+\t\t\t\t * data.rest.\n+\t\t\t\t */\n+\t\t\t\tchar *p = strpbrk(input.buf, \" \\t\");\n+\t\t\t\tif (p) {\n+\t\t\t\t\twhile (*p && strchr(\" \\t\", *p))\n+\t\t\t\t\t\t*p++ = '\\0';\n+\t\t\t\t}\n+\t\t\t\tdata.rest = p;\n \t\t\t}\n-\t\t\tdata.rest = p;\n+\t\t\tbatch_one_object(input.buf, &output, opt, &data);\n \t\t}\n-\n-\t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n \tstrbuf_release(&input);\n@@ -646,6 +796,7 @@ static int batch_option_callback(const struct option *opt,\n \n \tbo->enabled = 1;\n \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\tbo->command = !strcmp(opt->long_name, \"batch-command\");\n \tbo->format = arg;\n \n \treturn 0;\n@@ -682,6 +833,11 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"show info about objects fed from the standard input\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"\"),\n+\t\t\t N_(\"enters batch mode that accepts commands\"),\n+\t\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\t batch_option_callback),\n+\n \t\tOPT_BOOL(0, \"follow-symlinks\", &batch.follow_symlinks,\n \t\t\t N_(\"follow in-tree symlinks (used with --batch or --batch-check)\")),\n \t\tOPT_BOOL(0, \"batch-all-objects\", &batch.all_objects,\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 39382fa1958..7360d049113 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -85,6 +85,34 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ test -z \"$content\" ||\n+ test_expect_success \"--batch-command output of $type content is correct\" '\n+\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\tmaybe_remove_timestamp \"$(echo contents $sha1 | git cat-file --batch-command)\" $no_ts >actual &&\n+\ttest_cmp expect actual\n+ '\n+\n+ test -z \"$content\" ||\n+ test_expect_success \"--batch-command session for $type content is correct\" '\n+\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\tmaybe_remove_timestamp \\\n+\t\t\"$(test_write_lines \"begin\" \"$sha1\" \"get contents\" | git cat-file --batch-command)\" \\\n+\t\t$no_ts >actual &&\n+\ttest_cmp expect actual\n+ '\n+\n+ test_expect_success \"--batch-command output of $type info is correct\" '\n+\techo \"$sha1 $type $size\" >expect &&\n+\techo \"info $sha1\" | git cat-file --batch-command >actual &&\n+\ttest_cmp expect actual\n+ '\n+\n+ test_expect_success \"--batch-command session for $type info is correct\" '\n+\techo \"$sha1 $type $size\" >expect &&\n+\ttest_write_lines \"begin\" \"$sha1\" \"get info\" | git cat-file --batch-command >actual &&\n+\ttest_cmp expect actual\n+ '\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -141,6 +169,7 @@ test_expect_success '--batch-check without %(rest) considers whole line' '\n '\n \n tree_sha1=$(git write-tree)\n+\n tree_size=$(($(test_oid rawsz) + 13))\n tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n \n@@ -175,7 +204,7 @@ test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -281,6 +310,15 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+test_expect_success \"--batch-command with multiple sha1s gives correct format\" '\n+ echo \"$batch_check_output\" >expect &&\n+ echo begin >input &&\n+ echo_without_newline \"$batch_check_input\" >>input &&\n+ echo \"get info\" >>input &&\n+ git cat-file --batch-command actual &&\n+ test_cmp expect actual\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -872,4 +910,10 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command < cmd 2>err &&\n+\tgrep -E \"^fatal:.*unknown command.*\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"447645","messageId":"xmqqfsozu4v8.fsf@gitster.g","threadId":"57365","inReplyTo":"86df0c9e4df34566c10870e06865af536504a6af.1643915286.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/2] cat-file.c: rename cmdmode to mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-03T19:28:27Z","receivedAt":"2022-02-03T19:28:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> From: John Cai \n>\n> To prepare for a new flag --batch-command, we will add a flag that\n> indicates whether or not an interactive command mode will be used\n> that reads commands and arguments off of stdin.\n>\n> An intuitive name for this flag would be \"command\", which can get\n> confusing with the already existing cmdmode.\n\nI am moderately against this change. \"mode\" is too vague a word\n(i.e. it does not answer \"mode of what?\" question with the name),\nand renaming it to \"mode\" is probably a regression in readability.\n\nThe original name \"cmdmode\" is not all that better, either, as I am\nsure that it was named after the irrelevant implementation detail\nthat it is originally read using the OPT_CMDMODE() feature of the\nparse-options machinery, and did not mean to express mode of doing\n\"what\" with the name.\n\nSo, let's think aloud what this \"mode\" is about. The feature lets\nthe batch operation to choose how the blob objects are mangled [*],\nso \"mangle-mode\", \"filter-mode\", etc. (or find and use a better verb\nthan these) would be an improvement over the original \"cmdmode\".\n\n\n[Footnote]\n\n* 'w' to apply the smudge filter to make them as if they were\n written to the working tree, 'c' to apply the textconv filter, or\n 0 to stream them as-is.\n\n"},{"id":"447646","messageId":"xmqqo83nsoxs.fsf@gitster.g","threadId":"57365","inReplyTo":"ebd2a1356017905245744babf32c5b78a0af1639.1643915286.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] catfile.c: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-03T19:57:51Z","receivedAt":"2022-02-03T19:57:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> Subject: Re: [PATCH 2/2] catfile.c: add --batch-command mode\n\n\"cat-file: add --batch-command mode\" perhaps. The patch touching\nthe file \"catfile.c\" (which does not even exist) is an irrelevant\nimplementation detail to spend 2 extra bytes in \"git shortlog\"\noutput.\n\n> From: John Cai \n>\n> Add new flag --batch-command that will accept commands and arguments\n> from stdin, similar to git-update-ref --stdin.\n>\n> At GitLab, we use a pair of long running cat-file processes when\n> accessing object content. One for iterating over object metadata with\n> --batch-check, and the other to grab object contents with --batch.\n>\n> However, if we had --batch-command, we wouldnt need to keep both\n> processes around, and instead just have one process where we can flip\n> between getting object info, and getting object contents. This means we\n> can get rid of roughly half of long lived git cat-file processes. This\n> can lead to huge savings since on a given server there could be hundreds\n> of git cat-file processes running.\n\nHmph, why hundreds, not two you listed?\n\nDo you mean \"we have two per repository, but by combining, we can do\nwith just one per repository, halving the number of processes\"?\n\n> git cat-file --batch-command\n>\n> would enter an interactive command mode whereby the user can enter in\n> commands and their arguments:\n>\n> [arg1] [arg2] NL\n>\n> This patch adds the basic structure for add command which can be\n> extended in the future to add more commands.\n>\n> This patch also adds the following commands:\n>\n> contents NL\n> info NL\n>\n> The contents command takes an argument and prints out the object\n> contents.\n>\n> The info command takes a argument and prints out the object\n> metadata.\n>\n> In addition, we need a set of commands that enable a \"read session\".\n>\n> When a separate process (A) is connected to a git cat-file process (B)\n\nThis description misleads readers into thinking as if we have just\ncreated a daemon process that is running, and an unrelated process\ncan connect to it, which obviously poses a question about securing\nthe connection. It is my understanding that what this creates is\njust a consumer process (A) starts the cat-file process (B) locally\non its behalf under process (A)'s privilege, and they talk over pipe\nwithout allowing any third-party to participate in the exchange, so\nwe should avoid misleading users by saying \"is connected to\" here.\n\n> and is interactively writing to and reading from it in --buffer mode,\n> (A) needs to be able to know when the buffer is flushed to stdout.\n\nIf A and B are talking over a pair pipes, in order to avoid\ndeadlocking, both ends need to be able to control whose turn it is\nto speak (and it is turn for the other side to listen). A needs to\nbe able to _control_ (not \"know\") when the buffer it uses to write\nto B gets flushed, in order to reliably say \"I am done for now, it\nis your turn to speak\" and be assured that it reaches B. The story\nis the same for the other side. When a request by A needs to be\nresponded with multiple lines of output, B needs to be able to say\n\"And that concludes my response, and I am ready to accept a new\nrequest from you\" and make sure it reaches A. \"know when...\" is\nprobably a wrong phrase here.\n\n> Currently, from (A)'s perspective, the only way is to either 1. exit\n> (B)'s process or 2. send an invalid object to stdin. 1. is not ideal\n> from a performance perspective as it will require spawning a new cat-file\n> process each time, and 2. is hacky and not a good long term solution.\n\nWriting enumeration as bulletted or enumerated list would make it\nmuch easier to read, I would think.\n\n From (A)'s perspective, the only way is to either \n\n 1. exit (B)'s process or\n 2. send an invalid object to stdin.\n\n 1. is not ideal from a performance perspective, as it will\n require spawning a new cat-file process each time, and 2. is\n hacky and not a good long term solution.\n\nI am not sure what you exactly mean by \"exit\" in the above. Do you\nmean \"kill\" instead?\n\n> With the following commands, process (A) can begin a \"session\" and\n> send a list of object names over stdin. When \"get contents\" or \"get info\"\n> is issued, this list of object names will be fed into batch_one_object()\n> to retrieve either info or contents. Finally an fflush() will be called\n> to end the session.\n>\n> begin NL\n> get contents NL\n> get info NL\n>\n> These can be used in the following way:\n>\n> begin\n> \n> \n> \n> \n> get info\n>\n> begin\n> \n> \n> \n> \n> get contents\n>\n> With this mechanism, process (A) can be guaranteed to receive all of the\n> output even in --buffer mode.\n\nOK, so do these \"get blah\" serve both as command and an implicit\n\"flush\"?\n\nWith an implicit \"flush\", do we really need \"begin\"?\n\nAlso, from the point of view of extensibility, not saying what kind\nof operation is started when given \"begin\" is probably not a good\nidea. \"get info\" and \"get contents\" may happen to be the only\ncommands that are supported right now, and the parameters to them\nmay happen to be just list of object names and nothing else, but\nwhat happens when a new \"git frotz\" command is added and its\noperation is extended with something other than object names and\npathnames? The way to parse these parameter lines for the \"get\"\nwould be different for different commands, and if \"cat-file\" knows\nupfront what is to be done to these parameters, it can even start\nprefetching and precomputing to reduce latency observed by the\nclient before the final \"get info\" command is given.\n\nSo, from that point of view,\n\n\tbegin \n\t\n\t\n\t...\n\tend\n\nmay be a better design, no?\n"},{"id":"447702","messageId":"74E01F50-D5B4-41F0-A6F0-BBCA462ED74A@gmail.com","threadId":"57365","inReplyTo":"xmqqo83nsoxs.fsf@gitster.g","subject":"Re: [PATCH 2/2] catfile.c: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-04T04:11:10Z","receivedAt":"2022-02-04T04:11:14Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Junio, thanks for the review!\n\nOn 3 Feb 2022, at 14:57, Junio C Hamano wrote:\n\n> \"John Cai via GitGitGadget\" writes:\n>\n>> Subject: Re: [PATCH 2/2] catfile.c: add --batch-command mode\n>\n> \"cat-file: add --batch-command mode\" perhaps. The patch touching\n> the file \"catfile.c\" (which does not even exist) is an irrelevant\n> implementation detail to spend 2 extra bytes in \"git shortlog\"\n> output.\n>\n>> From: John Cai \n>>\n>> Add new flag --batch-command that will accept commands and arguments\n>> from stdin, similar to git-update-ref --stdin.\n>>\n>> At GitLab, we use a pair of long running cat-file processes when\n>> accessing object content. One for iterating over object metadata with\n>> --batch-check, and the other to grab object contents with --batch.\n>>\n>> However, if we had --batch-command, we wouldnt need to keep both\n>> processes around, and instead just have one process where we can flip\n>> between getting object info, and getting object contents. This means we\n>> can get rid of roughly half of long lived git cat-file processes. This\n>> can lead to huge savings since on a given server there could be hundreds\n>> of git cat-file processes running.\n>\n> Hmph, why hundreds, not two you listed?\n>\n> Do you mean \"we have two per repository, but by combining, we can do\n> with just one per repository, halving the number of processes\"?\n\nYes, exactly. I'll reword this in the next version to be more clear.\n\n>\n>> git cat-file --batch-command\n>>\n>> would enter an interactive command mode whereby the user can enter in\n>> commands and their arguments:\n>>\n>> [arg1] [arg2] NL\n>>\n>> This patch adds the basic structure for add command which can be\n>> extended in the future to add more commands.\n>>\n>> This patch also adds the following commands:\n>>\n>> contents NL\n>> info NL\n>>\n>> The contents command takes an argument and prints out the object\n>> contents.\n>>\n>> The info command takes a argument and prints out the object\n>> metadata.\n>>\n>> In addition, we need a set of commands that enable a \"read session\".\n>>\n>> When a separate process (A) is connected to a git cat-file process (B)\n>\n> This description misleads readers into thinking as if we have just\n> created a daemon process that is running, and an unrelated process\n> can connect to it, which obviously poses a question about securing\n> the connection. It is my understanding that what this creates is\n> just a consumer process (A) starts the cat-file process (B) locally\n> on its behalf under process (A)'s privilege, and they talk over pipe\n> without allowing any third-party to participate in the exchange, so\n> we should avoid misleading users by saying \"is connected to\" here.\n\nYes this understanding is correct. Will fix wording in next version\n\n>\n>> and is interactively writing to and reading from it in --buffer mode,\n>> (A) needs to be able to know when the buffer is flushed to stdout.\n>\n> If A and B are talking over a pair pipes, in order to avoid\n> deadlocking, both ends need to be able to control whose turn it is\n> to speak (and it is turn for the other side to listen). A needs to\n> be able to _control_ (not \"know\") when the buffer it uses to write\n> to B gets flushed, in order to reliably say \"I am done for now, it\n> is your turn to speak\" and be assured that it reaches B. The story\n> is the same for the other side. When a request by A needs to be\n> responded with multiple lines of output, B needs to be able to say\n> \"And that concludes my response, and I am ready to accept a new\n> request from you\" and make sure it reaches A. \"know when...\" is\n> probably a wrong phrase here.\n\nCorrect, \"know\" is not exactly right. _control_ would be the more accurate description.\n>\n>> Currently, from (A)'s perspective, the only way is to either 1. exit\n>> (B)'s process or 2. send an invalid object to stdin. 1. is not ideal\n>> from a performance perspective as it will require spawning a new cat-file\n>> process each time, and 2. is hacky and not a good long term solution.\n>\n> Writing enumeration as bulletted or enumerated list would make it\n> much easier to read, I would think.\n>\n> From (A)'s perspective, the only way is to either\n>\n> 1. exit (B)'s process or\n> 2. send an invalid object to stdin.\n>\n> 1. is not ideal from a performance perspective, as it will\n> require spawning a new cat-file process each time, and 2. is\n> hacky and not a good long term solution.\n>\n> I am not sure what you exactly mean by \"exit\" in the above. Do you\n> mean \"kill\" instead?\n\nYes\n\n>\n>> With the following commands, process (A) can begin a \"session\" and\n>> send a list of object names over stdin. When \"get contents\" or \"get info\"\n>> is issued, this list of object names will be fed into batch_one_object()\n>> to retrieve either info or contents. Finally an fflush() will be called\n>> to end the session.\n>>\n>> begin NL\n>> get contents NL\n>> get info NL\n>>\n>> These can be used in the following way:\n>>\n>> begin\n>> \n>> \n>> \n>> \n>> get info\n>>\n>> begin\n>> \n>> \n>> \n>> \n>> get contents\n>>\n>> With this mechanism, process (A) can be guaranteed to receive all of the\n>> output even in --buffer mode.\n>\n> OK, so do these \"get blah\" serve both as command and an implicit\n> \"flush\"?\n\nyes, that's the idea.\n\n>\n> With an implicit \"flush\", do we really need \"begin\"?\n>\n> Also, from the point of view of extensibility, not saying what kind\n> of operation is started when given \"begin\" is probably not a good\n> idea. \"get info\" and \"get contents\" may happen to be the only\n> commands that are supported right now, and the parameters to them\n> may happen to be just list of object names and nothing else, but\n> what happens when a new \"git frotz\" command is added and its\n> operation is extended with something other than object names and\n> pathnames? The way to parse these parameter lines for the \"get\"\n> would be different for different commands, and if \"cat-file\" knows\n> upfront what is to be done to these parameters, it can even start\n> prefetching and precomputing to reduce latency observed by the\n> client before the final \"get info\" command is given.\n>\n> So, from that point of view,\n>\n> begin \n> \n> \n> ...\n> end\n>\n> may be a better design, no?\n\nGood point. Now I'm wondering if we can simplify where commands get queued up\nand a \"get\" will execute them along with an implicit flush.\n\n \n \n \nget\n"},{"id":"447709","messageId":"CAPig+cQHcstj28F-==oCakp3iMLHtd1SVQV+snzewYihHxNG6g@mail.gmail.com","threadId":"57365","inReplyTo":"ebd2a1356017905245744babf32c5b78a0af1639.1643915286.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] catfile.c: add --batch-command mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-04T06:45:52Z","receivedAt":"2022-02-04T06:46:07Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"()()On Thu, Feb 3, 2022 at 7:20 PM John Cai via GitGitGadget\n wrote:\n> Add new flag --batch-command that will accept commands and arguments\n> from stdin, similar to git-update-ref --stdin.\n>\n> contents NL\n> info NL\n>\n> With the following commands, process (A) can begin a \"session\" and\n> send a list of object names over stdin. When \"get contents\" or \"get info\"\n> is issued, this list of object names will be fed into batch_one_object()\n> to retrieve either info or contents. Finally an fflush() will be called\n> to end the session.\n>\n> begin\n> \n> get info\n>\n> begin\n> \n> get contents\n\nI had the same reaction to this new command set as Junio expressed\nupstream[1], and was prepared to suggest an alternative but Junio said\neverything I wanted to say, so I won't say anything more about it\nhere.\n\nThat aside, the implementation presented here is overly loose about\nmalformed input and accepts all sorts of bogus invocations which it\nshould be reporting as errors. For instance, a lone:\n\n get info\n\nwithout a preceding `begin` should be an error but such bogus input is\nnot diagnosed. `get contents` is likewise affected. Another example:\n\n begin\n \n \n\nwhich lacks a closing `get info` or `get contents` is silently\nignored. Similarly, malformed:\n\n begin\n begin\n ...\n\nshould be reported as an error but is not.\n\nThere is also a bug in which the accumulated list of OID's is never\ncleared. Thus:\n\n begin\n \n get info\n get info\n\nemits info about twice, once for each `get info` invocation. Similarly:\n\n begin\n \n get info\n \n get info\n\nemits information about twice and once. Invoking `begin`\nbetween the `get info` commands doesn't help because `begin` is a\nno-op. Thus:\n\n begin\n \n get info\n begin\n \n get info\n\nlikewise emits information about twice and once.\n\nIt also incorrectly accepts non-\"session\" commands in the middle of a\nsession. For instance:\n\n begin\n \n info \n \n\nimmediately emits information about and then bombs out claiming\nthat is an \"unknown command\" because the `info` command --\nwhich should not have been allowed within a session -- prematurely\nended the \"session\".\n\nThe `info` and `contents` commands neglect to do any sort of\nvalidation of their arguments, thus any and all bogus invocations are\naccepted. Thus:\n\n info\n info \n info \n\nare all accepted as valid invocations, misleadingly producing \"\nmissing\" messages, rather than erroring out as they should. The \"\nmissing\" message should be reserved for the case when the lone\nargument to `info` or `contents` is something which looks like a\nlegitimate OID.\n\nThe above is a long-winded way of saying that it is important not only\nto check the obvious \"expect success\" cases when implementing a new\nfeature, but it's also important to add checks for the \"expect\nfailure\" cases, such as all the above malformed inputs.\n\nIt's subjective, but it feels like this implementation is trying to be\ntoo clever by handling all these cases via a single strbuf_getline()\nloop in batch_objects_command(). It would be easier to reason about,\nand would have avoided some of the above problems, for instance, if\nhandling of `begin` was split out to its own function which looped\nover strbuf_getline() itself, thus could easily have detected\npremature EOF (lacking `get info` or `get contents`) and likewise\nwould not have allowed `info` or `contents` commands to be executed\nwithin a \"session\".\n\nSimilarly (again subjective), the generic command dispatcher seems\nover-engineered and perhaps contributed to the oversight of `info` and\n`contents` failing to perform validation of their arguments. A simpler\nhand-rolled command-response loop is more common on this project and\noften easier to reason about. Perhaps something like this\n(pseudo-code):\n\n while (strbuf_getline(&input, stdin)) {\n char *end_cmd = strchrnul(&input.buf, ' ');\n const char *argstart = *end_cmd ? end_cmd + 1 : end_cmd;\n *end_cmd = '\\0';\n if (strcmp(input.buf, \"info\"))\n show_info(argstart);\n else if (strcmp(input.buf, \"contents\"))\n show_contents(argstart);\n else if (strcmp(input.buf, \"begin\"))\n begin_session(argstart);\n else\n die(...);\n }\n\nand each of the command-handler functions would perform its own\nargument validation (including the case when no argument should be\npresent).\n\n[1]: https://lore.kernel.org/git/xmqqo83nsoxs.fsf@gitster.g/\n\n> +static void parse_cmd_begin(struct batch_options *opt,\n> + const char *line,\n> + struct strbuf *output,\n> + struct expand_data *data,\n> + struct string_list revs)\n> +{\n> + /* nothing needs to be done here */\n> +}\n\nEither this function should be clearing the list of collected OID's or...\n\n> +static void parse_cmd_get(struct batch_options *opt,\n> + const char *line,\n> + struct strbuf *output,\n> + struct expand_data *data,\n> + struct string_list revs)\n> +{\n> + struct string_list_item *item;\n> + for_each_string_list_item(item, &revs) {\n> + batch_one_object(item->string, output, opt, data);\n> + }\n> + if (opt->buffer_output)\n> + fflush(stdout);\n\n... this function should do so after it's done processing the OID list.\n\n> +}\n> +static void parse_cmd_get_info(struct batch_options *opt,\n> + const char *line,\n> + struct strbuf *output,\n> + struct expand_data *data,\n> + struct string_list revs)\n\nMissing blank line before the function definition.\n\n> +static void parse_cmd_get_objects(struct batch_options *opt,\n> + const char *line,\n> + struct strbuf *output,\n> + struct expand_data *data,\n> + struct string_list revs)\n> +{\n> + opt->print_contents = 1;\n> + parse_cmd_get(opt, line, output, data, revs);\n> + if (opt->buffer_output)\n> + fflush(stdout);\n> +}\n\nWhy does this function duplicate the flushing logic of parse_cmd_get()\nimmediately after calling parse_cmd_get()?\n\n> +static void batch_objects_command(struct batch_options *opt,\n> + struct strbuf *output,\n> + struct expand_data *data)\n> +{\n> + /* Read each line dispatch its command */\n\nMissing \"and\".\n\n> + while (!strbuf_getline(&input, stdin)) {\n> + if (state == BATCH_STATE_COMMAND) {\n> + if (*input.buf == '\\n')\n> + die(\"empty command in input\");\n\nThis never triggers because strbuf_getline() removes the line\nterminator before returning.\n\n> + else if (isspace(*input.buf))\n> + die(\"whitespace before command: %s\", input.buf);\n> + }\n> +\n> + for (i = 0; i < ARRAY_SIZE(commands); i++) {\n> + if (!skip_prefix(input.buf, prefix, &cmd_end))\n> + continue;\n> + /*\n> + * If the command has arguments, verify that it's\n> + * followed by a space. Otherwise, it shall be followed\n> + * by a line terminator.\n> + */\n> + c = commands[i].takes_args ? ' ' : '\\n';\n> + if (*cmd_end && *cmd_end != c)\n> + die(\"arguments invalid for command: %s\", commands[i].prefix);\n\nDitto regarding strbuf_getline() removing line-terminator.\n\n> + cmd = &commands[i];\n> + if (cmd->takes_args)\n> + p = cmd_end + 1;\n> + break;\n> + }\n> +\n> + if (input.buf[input.len - 1] == '\\n')\n> + input.buf[--input.len] = '\\0';\n\nDitto again. This is especially scary since it's accessing element\n`input.len-1` without even checking if `input.len` is greater than\nzero.\n\nAlso, it's better to use published strbuf API rather than mucking with\nthe internals:\n\n strbuf_setlen(&input, input.len - 1);\n\n> + if (state == BATCH_STATE_INPUT && !cmd){\n> + string_list_append(&revs, input.buf);\n> + continue;\n> + }\n> +\n> + if (!cmd)\n> + die(\"unknown command: %s\", input.buf);\n> +\n> + state = cmd->next_state;\n> + cmd->fn(opt, p, output, data, revs);\n> + }\n> + strbuf_release(&input);\n> + string_list_clear(&revs, 0);\n> +}\n"},{"id":"447715","messageId":"220204.86zgn63k8n.gmgdl@evledraar.gmail.com","threadId":"57365","inReplyTo":"86df0c9e4df34566c10870e06865af536504a6af.1643915286.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/2] cat-file.c: rename cmdmode to mode","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-04T12:10:16Z","receivedAt":"2022-02-04T12:10:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 03 2022, John Cai via GitGitGadget wrote:\n\n> From: John Cai \n>\n> To prepare for a new flag --batch-command, we will add a flag that\n> indicates whether or not an interactive command mode will be used\n> that reads commands and arguments off of stdin.\n>\n> An intuitive name for this flag would be \"command\", which can get\n> confusing with the already existing cmdmode.\n>\n> Rename cmdmode->mode to prepare for this change.\n>\n> Signed-off-by: John Cai \n> ---\n> builtin/cat-file.c | 19 +++++++++----------\n> 1 file changed, 9 insertions(+), 10 deletions(-)\n>\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index d94050e6c18..858bca208ff 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -24,7 +24,7 @@ struct batch_options {\n> \tint buffer_output;\n> \tint all_objects;\n> \tint unordered;\n> -\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n> +\tint mode; /* may be 'w' or 'c' for --filters or --textconv */\n> \tconst char *format;\n> };\n> \n> @@ -306,19 +306,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n> \tif (data->type == OBJ_BLOB) {\n> \t\tif (opt->buffer_output)\n> \t\t\tfflush(stdout);\n> -\t\tif (opt->cmdmode) {\n> +\t\tif (opt->mode) {\n> \t\t\tchar *contents;\n> \t\t\tunsigned long size;\n> \n> \t\t\tif (!data->rest)\n> \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n> \n> -\t\t\tif (opt->cmdmode == 'w') {\n> +\t\t\tif (opt->mode == 'w') {\n> \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n> \t\t\t\t\t\t &contents, &size))\n> \t\t\t\t\tdie(\"could not convert '%s' %s\",\n> \t\t\t\t\t oid_to_hex(oid), data->rest);\n> -\t\t\t} else if (opt->cmdmode == 'c') {\n> +\t\t\t} else if (opt->mode == 'c') {\n> \t\t\t\tenum object_type type;\n> \t\t\t\tif (!textconv_object(the_repository,\n> \t\t\t\t\t\t data->rest, 0100644, oid,\n> @@ -330,7 +330,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n> \t\t\t\t\tdie(\"could not convert '%s' %s\",\n> \t\t\t\t\t oid_to_hex(oid), data->rest);\n> \t\t\t} else\n> -\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n> +\t\t\t\tBUG(\"invalid mode: %c\", opt->mode);\n> \t\t\tbatch_write(opt, contents, size);\n> \t\t\tfree(contents);\n> \t\t} else {\n> @@ -533,7 +533,7 @@ static int batch_objects(struct batch_options *opt)\n> \tstrbuf_expand(&output, opt->format, expand_format, &data);\n> \tdata.mark_query = 0;\n> \tstrbuf_release(&output);\n> -\tif (opt->cmdmode)\n> +\tif (opt->mode)\n> \t\tdata.split_on_whitespace = 1;\n> \n> \t/*\n> @@ -695,10 +695,9 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n> \n> \tbatch.buffer_output = -1;\n> \targc = parse_options(argc, argv, prefix, options, cat_file_usage, 0);\n> -\n> \tif (opt) {\n> \t\tif (batch.enabled && (opt == 'c' || opt == 'w'))\n> -\t\t\tbatch.cmdmode = opt;\n> +\t\t\tbatch.mode = opt;\n> \t\telse if (argc == 1)\n> \t\t\tobj_name = argv[0];\n> \t\telse\n> @@ -712,9 +711,9 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n> \t\t\tusage_with_options(cat_file_usage, options);\n> \t}\n> \tif (batch.enabled) {\n> -\t\tif (batch.cmdmode != opt || argc)\n> +\t\tif (batch.mode != opt || argc)\n> \t\t\tusage_with_options(cat_file_usage, options);\n> -\t\tif (batch.cmdmode && batch.all_objects)\n> +\t\tif (batch.mode && batch.all_objects)\n> \t\t\tdie(\"--batch-all-objects cannot be combined with \"\n> \t\t\t \"--textconv nor with --filters\");\n> \t}\n\nThere's a rather major rewrite in-flight to this codepath in\nab/cat-file. I think it would be better to base this on top of that. Per\nJunio's latest \"What's Cooking\" that may be getting merged down today or\ntomorrow.\n"},{"id":"447716","messageId":"220204.86v8xu3jou.gmgdl@evledraar.gmail.com","threadId":"57365","inReplyTo":"ebd2a1356017905245744babf32c5b78a0af1639.1643915286.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] catfile.c: add --batch-command mode","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-04T12:11:34Z","receivedAt":"2022-02-04T12:22:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 03 2022, John Cai via GitGitGadget wrote:\n\n> From: John Cai \n\n[Trying not to pile on and mentioning some things others haven't, but\nmaybe there'll be duplications]\n\n> \trequested batch operation on all objects in the repository and\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 858bca208ff..29d5cd6857b 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -26,6 +26,7 @@ struct batch_options {\n> \tint unordered;\n> \tint mode; /* may be 'w' or 'c' for --filters or --textconv */\n> \tconst char *format;\n> +\tint command;\n> };\n\nNot sure why it's not added to \"int mode\", but in any case\npost-ab/cat-file that might be clearer...\n\n> +\t/* Read each line dispatch its command */\n> +\twhile (!strbuf_getline(&input, stdin)) {\n\nI think comments that are obvious from the code are probably best\ndropped. We're just doing a fairly obvious consume stdin pattern here.\n\n> +\t\tint i;\n> +\t\tconst struct parse_cmd *cmd = NULL;\n> +\t\tconst char *p, *cmd_end;\n> +\n> +\t\tif (state == BATCH_STATE_COMMAND) {\n> +\t\t\tif (*input.buf == '\\n')\n> +\t\t\t\tdie(\"empty command in input\");\n> +\t\t\telse if (isspace(*input.buf))\n> +\t\t\t\tdie(\"whitespace before command: %s\", input.buf);\n\nThese should use _() to mark strings for translation, and let's quote %s\nlike '%s'\n\n> +\t\t}\n> +\n> +\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n> +\t\t\tconst char *prefix = commands[i].prefix;\n> +\t\t\tchar c;\n\nnit: add a \\n between variable decls and code.\n\n> +\t\t\tif (!skip_prefix(input.buf, prefix, &cmd_end))\n> +\t\t\t\tcontinue;\n> +\t\t\t/*\n> +\t\t\t * If the command has arguments, verify that it's\n> +\t\t\t * followed by a space. Otherwise, it shall be followed\n> +\t\t\t * by a line terminator.\n> +\t\t\t */\n\nI'd say ditto on the \"the code says that\" comments...\n\n> +\t\t\tc = commands[i].takes_args ? ' ' : '\\n';\n> +\t\t\tif (*cmd_end && *cmd_end != c)\n> +\t\t\t\tdie(\"arguments invalid for command: %s\", commands[i].prefix);\n> +\n> +\t\t\tcmd = &commands[i];\n> +\t\t\tif (cmd->takes_args)\n> +\t\t\t\tp = cmd_end + 1;\n> +\t\t\tbreak;\n> +\t\t}\n> +\n> +\t\tif (input.buf[input.len - 1] == '\\n')\n> +\t\t\tinput.buf[--input.len] = '\\0';\n\nDon't we mean strbuf_trim_trailing_newline() here, or do we not want\nWindows newlines to be accepted?\n\nBut more generally doesn't one of the strbuf_getline_*() functions do\nthe right thing here already?\n\n> +\n> +\t\tif (state == BATCH_STATE_INPUT && !cmd){\n> +\t\t\tstring_list_append(&revs, input.buf);\n\nNit: You can save yourself some malloc() churn here with:\n\n string_list_append_nodup(..., strbuf_detach(&input, NULL));\n\nI.e. we're looping over the input, here we're done, so we might as well\nsteal the already alloc'd string....\n\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (!cmd)\n> +\t\t\tdie(\"unknown command: %s\", input.buf);\n> +\n> +\t\tstate = cmd->next_state;\n> +\t\tcmd->fn(opt, p, output, data, revs);\n> +\t}\n> +\tstrbuf_release(&input);\n> +\tstring_list_clear(&revs, 0);\n\n...and these will do the right thing, as strbuf will notice the string\nis stolen (it'll be the slopbuf again), and due to the combination of\n*_DUP and *_nodup() we'll properly free it here too.\n\n> +}\n> +\n> static int batch_objects(struct batch_options *opt)\n> {\n> \tstruct strbuf input = STRBUF_INIT;\n> @@ -519,6 +665,7 @@ static int batch_objects(struct batch_options *opt)\n> \tstruct expand_data data;\n> \tint save_warning;\n> \tint retval = 0;\n> +\tconst int command = opt->command;\n> \n> \tif (!opt->format)\n> \t\topt->format = \"%(objectname) %(objecttype) %(objectsize)\";\n> @@ -594,22 +741,25 @@ static int batch_objects(struct batch_options *opt)\n> \tsave_warning = warn_on_object_refname_ambiguity;\n> \twarn_on_object_refname_ambiguity = 0;\n> \n> -\twhile (strbuf_getline(&input, stdin) != EOF) {\n> -\t\tif (data.split_on_whitespace) {\n> -\t\t\t/*\n> -\t\t\t * Split at first whitespace, tying off the beginning\n> -\t\t\t * of the string and saving the remainder (or NULL) in\n> -\t\t\t * data.rest.\n> -\t\t\t */\n> -\t\t\tchar *p = strpbrk(input.buf, \" \\t\");\n> -\t\t\tif (p) {\n> -\t\t\t\twhile (*p && strchr(\" \\t\", *p))\n> -\t\t\t\t\t*p++ = '\\0';\n> +\tif (command)\n> +\t\tbatch_objects_command(opt, &output, &data);\n> +\telse {\n\nStyle: {} braces for all arms if one requires it.\n\n> +\t\twhile (strbuf_getline(&input, stdin) != EOF) {\n> +\t\t\tif (data.split_on_whitespace) {\n\ndiff nit: maybe we can find some way to not require re-indenting the existing code. E.g.:\n\t\n\tif (command) {\n\t\tbatch_objects_command(...);\n\t goto cleanup;\n\t}\n\n...\n\n> +\t\t\t\t/*\n> +\t\t\t\t * Split at first whitespace, tying off the beginning\n> +\t\t\t\t * of the string and saving the remainder (or NULL) in\n> +\t\t\t\t * data.rest.\n> +\t\t\t\t */\n> +\t\t\t\tchar *p = strpbrk(input.buf, \" \\t\");\n> +\t\t\t\tif (p) {\n> +\t\t\t\t\twhile (*p && strchr(\" \\t\", *p))\n> +\t\t\t\t\t\t*p++ = '\\0';\n> +\t\t\t\t}\n> +\t\t\t\tdata.rest = p;\n> \t\t\t}\n> -\t\t\tdata.rest = p;\n> +\t\t\tbatch_one_object(input.buf, &output, opt, &data);\n> \t\t}\n> -\n> -\t\tbatch_one_object(input.buf, &output, opt, &data);\n> \t}\n> \n\n...and then just add a \"cleanup:\" label here.\n\n> \tstrbuf_release(&input);\n> @@ -646,6 +796,7 @@ static int batch_option_callback(const struct option *opt,\n> \n> \tbo->enabled = 1;\n> \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n> +\tbo->command = !strcmp(opt->long_name, \"batch-command\");\n> \tbo->format = arg;\n> \n> \treturn 0;\n> @@ -682,6 +833,11 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n> \t\t\tN_(\"show info about objects fed from the standard input\"),\n> \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> \t\t\tbatch_option_callback),\n> +\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"\"),\n\nYou're either missing a string here in \"\", or we don't need N_() to mark\nit for translation.\n\n> +\t\t\t N_(\"enters batch mode that accepts commands\"),\n> +\t\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> +\t\t\t batch_option_callback),\n> +\n> \t\tOPT_BOOL(0, \"follow-symlinks\", &batch.follow_symlinks,\n> \t\t\t N_(\"follow in-tree symlinks (used with --batch or --batch-check)\")),\n> \t\tOPT_BOOL(0, \"batch-all-objects\", &batch.all_objects,\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 39382fa1958..7360d049113 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -85,6 +85,34 @@ $content\"\n> \ttest_cmp expect actual\n> '\n> \n> + test -z \"$content\" ||\n> + test_expect_success \"--batch-command output of $type content is correct\" '\n> +\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n> +\tmaybe_remove_timestamp \"$(echo contents $sha1 | git cat-file --batch-command)\" $no_ts >actual &&\n> +\ttest_cmp expect actual\n> + '\n> +\n> + test -z \"$content\" ||\n> + test_expect_success \"--batch-command session for $type content is correct\" '\n> +\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n> +\tmaybe_remove_timestamp \\\n> +\t\t\"$(test_write_lines \"begin\" \"$sha1\" \"get contents\" | git cat-file --batch-command)\" \\\n> +\t\t$no_ts >actual &&\n> +\ttest_cmp expect actual\n> + '\n> +\n> + test_expect_success \"--batch-command output of $type info is correct\" '\n> +\techo \"$sha1 $type $size\" >expect &&\n> +\techo \"info $sha1\" | git cat-file --batch-command >actual &&\n> +\ttest_cmp expect actual\n> + '\n> +\n> + test_expect_success \"--batch-command session for $type info is correct\" '\n> +\techo \"$sha1 $type $size\" >expect &&\n> +\ttest_write_lines \"begin\" \"$sha1\" \"get info\" | git cat-file --batch-command >actual &&\n> +\ttest_cmp expect actual\n> + '\n> +\n> test_expect_success \"custom --batch-check format\" '\n> \techo \"$type $sha1\" >expect &&\n> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n> @@ -141,6 +169,7 @@ test_expect_success '--batch-check without %(rest) considers whole line' '\n> '\n> \n> tree_sha1=$(git write-tree)\n> +\n\nstray newline addition.\n\n> tree_size=$(($(test_oid rawsz) + 13))\n> tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n> \n> @@ -175,7 +204,7 @@ test_expect_success \\\n> \"Reach a blob from a tag pointing to it\" \\\n> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n> \n> -for batch in batch batch-check\n> +for batch in batch batch-check batch-command\n> do\n> for opt in t s e p\n> do\n> @@ -281,6 +310,15 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n> '\n> \n> +test_expect_success \"--batch-command with multiple sha1s gives correct format\" '\n> + echo \"$batch_check_output\" >expect &&\n> + echo begin >input &&\n> + echo_without_newline \"$batch_check_input\" >>input &&\n> + echo \"get info\" >>input &&\n> + git cat-file --batch-command actual &&\n> + test_cmp expect actual\n> +'\n\nindentation with spaces, \\t correctly used for the rest.\n"},{"id":"447750","messageId":"767d5f5a-8395-78bc-865f-a39acc39e061@gmail.com","threadId":"57365","inReplyTo":"74E01F50-D5B4-41F0-A6F0-BBCA462ED74A@gmail.com","subject":"Re: [PATCH 2/2] catfile.c: add --batch-command mode","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-04T16:46:54Z","receivedAt":"2022-02-04T16:47:00Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John\n\nOn 04/02/2022 04:11, John Cai wrote:\n> Hi Junio, thanks for the review!\n> [...]\n>> So, from that point of view,\n>>\n>> begin \n>> \n>> \n>> ...\n>> end\n>>\n>> may be a better design, no?\n> \n> Good point. Now I'm wondering if we can simplify where commands get queued up\n> and a \"get\" will execute them along with an implicit flush.\n> \n> \n> \n> \n> get\n\nI think that would be an improvement (I'd suggest calling \"get\" \"flush\" \ninstead), the begin ... get sequence seems unnecessarily complex \ncompared the the RFC of this series. If the user gives --buffer on the \ncommand line then we can buffer the input until we see a \"flush\" command \nand if the user does not give --buffer on the command line then we \nshould flush the output after each command.\n\nBest Wishes\n\nPhillip\n"},{"id":"447751","messageId":"21e8af90-3217-782a-2ce5-3a99fd37cfbc@gmail.com","threadId":"57365","inReplyTo":"220204.86v8xu3jou.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] catfile.c: add --batch-command mode","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-04T16:51:25Z","receivedAt":"2022-02-04T16:51:33Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John/Ævar\n\nOn 04/02/2022 12:11, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Feb 03 2022, John Cai via GitGitGadget wrote:\n> \n>> From: John Cai \n>> +\t\tif (state == BATCH_STATE_INPUT && !cmd){\n>> +\t\t\tstring_list_append(&revs, input.buf);\n> \n> Nit: You can save yourself some malloc() churn here with:\n> \n> string_list_append_nodup(..., strbuf_detach(&input, NULL));\n> \n> I.e. we're looping over the input, here we're done, so we might as well\n> steal the already alloc'd string....\n\nIf we do that then the strbuf will have to reallocate its buffer when it \nreads the next input line on the next iteration of the loop. As the \nstrbuf will have likely over allocated the buffer using the \nstring_list_append_nodup() + strbuf_detach() will will be less efficient \noverall than using string_list_append().\n\nBest Wishes\n\nPhillip\n\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\tif (!cmd)\n>> +\t\t\tdie(\"unknown command: %s\", input.buf);\n>> +\n>> +\t\tstate = cmd->next_state;\n>> +\t\tcmd->fn(opt, p, output, data, revs);\n>> +\t}\n>> +\tstrbuf_release(&input);\n>> +\tstring_list_clear(&revs, 0);\n> \n> ...and these will do the right thing, as strbuf will notice the string\n> is stolen (it'll be the slopbuf again), and due to the combination of\n> *_DUP and *_nodup() we'll properly free it here too.\n> \n>> +}\n>> +\n>> static int batch_objects(struct batch_options *opt)\n>> {\n>> \tstruct strbuf input = STRBUF_INIT;\n>> @@ -519,6 +665,7 @@ static int batch_objects(struct batch_options *opt)\n>> \tstruct expand_data data;\n>> \tint save_warning;\n>> \tint retval = 0;\n>> +\tconst int command = opt->command;\n>> \n>> \tif (!opt->format)\n>> \t\topt->format = \"%(objectname) %(objecttype) %(objectsize)\";\n>> @@ -594,22 +741,25 @@ static int batch_objects(struct batch_options *opt)\n>> \tsave_warning = warn_on_object_refname_ambiguity;\n>> \twarn_on_object_refname_ambiguity = 0;\n>> \n>> -\twhile (strbuf_getline(&input, stdin) != EOF) {\n>> -\t\tif (data.split_on_whitespace) {\n>> -\t\t\t/*\n>> -\t\t\t * Split at first whitespace, tying off the beginning\n>> -\t\t\t * of the string and saving the remainder (or NULL) in\n>> -\t\t\t * data.rest.\n>> -\t\t\t */\n>> -\t\t\tchar *p = strpbrk(input.buf, \" \\t\");\n>> -\t\t\tif (p) {\n>> -\t\t\t\twhile (*p && strchr(\" \\t\", *p))\n>> -\t\t\t\t\t*p++ = '\\0';\n>> +\tif (command)\n>> +\t\tbatch_objects_command(opt, &output, &data);\n>> +\telse {\n> \n> Style: {} braces for all arms if one requires it.\n> \n>> +\t\twhile (strbuf_getline(&input, stdin) != EOF) {\n>> +\t\t\tif (data.split_on_whitespace) {\n> \n> diff nit: maybe we can find some way to not require re-indenting the existing code. E.g.:\n> \t\n> \tif (command) {\n> \t\tbatch_objects_command(...);\n> \t goto cleanup;\n> \t}\n> \n> ...\n> \n>> +\t\t\t\t/*\n>> +\t\t\t\t * Split at first whitespace, tying off the beginning\n>> +\t\t\t\t * of the string and saving the remainder (or NULL) in\n>> +\t\t\t\t * data.rest.\n>> +\t\t\t\t */\n>> +\t\t\t\tchar *p = strpbrk(input.buf, \" \\t\");\n>> +\t\t\t\tif (p) {\n>> +\t\t\t\t\twhile (*p && strchr(\" \\t\", *p))\n>> +\t\t\t\t\t\t*p++ = '\\0';\n>> +\t\t\t\t}\n>> +\t\t\t\tdata.rest = p;\n>> \t\t\t}\n>> -\t\t\tdata.rest = p;\n>> +\t\t\tbatch_one_object(input.buf, &output, opt, &data);\n>> \t\t}\n>> -\n>> -\t\tbatch_one_object(input.buf, &output, opt, &data);\n>> \t}\n>> \n> \n> ...and then just add a \"cleanup:\" label here.\n> \n>> \tstrbuf_release(&input);\n>> @@ -646,6 +796,7 @@ static int batch_option_callback(const struct option *opt,\n>> \n>> \tbo->enabled = 1;\n>> \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n>> +\tbo->command = !strcmp(opt->long_name, \"batch-command\");\n>> \tbo->format = arg;\n>> \n>> \treturn 0;\n>> @@ -682,6 +833,11 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n>> \t\t\tN_(\"show info about objects fed from the standard input\"),\n>> \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n>> \t\t\tbatch_option_callback),\n>> +\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"\"),\n> \n> You're either missing a string here in \"\", or we don't need N_() to mark\n> it for translation.\n> \n>> +\t\t\t N_(\"enters batch mode that accepts commands\"),\n>> +\t\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n>> +\t\t\t batch_option_callback),\n>> +\n>> \t\tOPT_BOOL(0, \"follow-symlinks\", &batch.follow_symlinks,\n>> \t\t\t N_(\"follow in-tree symlinks (used with --batch or --batch-check)\")),\n>> \t\tOPT_BOOL(0, \"batch-all-objects\", &batch.all_objects,\n>> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n>> index 39382fa1958..7360d049113 100755\n>> --- a/t/t1006-cat-file.sh\n>> +++ b/t/t1006-cat-file.sh\n>> @@ -85,6 +85,34 @@ $content\"\n>> \ttest_cmp expect actual\n>> '\n>> \n>> + test -z \"$content\" ||\n>> + test_expect_success \"--batch-command output of $type content is correct\" '\n>> +\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n>> +\tmaybe_remove_timestamp \"$(echo contents $sha1 | git cat-file --batch-command)\" $no_ts >actual &&\n>> +\ttest_cmp expect actual\n>> + '\n>> +\n>> + test -z \"$content\" ||\n>> + test_expect_success \"--batch-command session for $type content is correct\" '\n>> +\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n>> +\tmaybe_remove_timestamp \\\n>> +\t\t\"$(test_write_lines \"begin\" \"$sha1\" \"get contents\" | git cat-file --batch-command)\" \\\n>> +\t\t$no_ts >actual &&\n>> +\ttest_cmp expect actual\n>> + '\n>> +\n>> + test_expect_success \"--batch-command output of $type info is correct\" '\n>> +\techo \"$sha1 $type $size\" >expect &&\n>> +\techo \"info $sha1\" | git cat-file --batch-command >actual &&\n>> +\ttest_cmp expect actual\n>> + '\n>> +\n>> + test_expect_success \"--batch-command session for $type info is correct\" '\n>> +\techo \"$sha1 $type $size\" >expect &&\n>> +\ttest_write_lines \"begin\" \"$sha1\" \"get info\" | git cat-file --batch-command >actual &&\n>> +\ttest_cmp expect actual\n>> + '\n>> +\n>> test_expect_success \"custom --batch-check format\" '\n>> \techo \"$type $sha1\" >expect &&\n>> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n>> @@ -141,6 +169,7 @@ test_expect_success '--batch-check without %(rest) considers whole line' '\n>> '\n>> \n>> tree_sha1=$(git write-tree)\n>> +\n> \n> stray newline addition.\n> \n>> tree_size=$(($(test_oid rawsz) + 13))\n>> tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n>> \n>> @@ -175,7 +204,7 @@ test_expect_success \\\n>> \"Reach a blob from a tag pointing to it\" \\\n>> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n>> \n>> -for batch in batch batch-check\n>> +for batch in batch batch-check batch-command\n>> do\n>> for opt in t s e p\n>> do\n>> @@ -281,6 +310,15 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n>> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n>> '\n>> \n>> +test_expect_success \"--batch-command with multiple sha1s gives correct format\" '\n>> + echo \"$batch_check_output\" >expect &&\n>> + echo begin >input &&\n>> + echo_without_newline \"$batch_check_input\" >>input &&\n>> + echo \"get info\" >>input &&\n>> + git cat-file --batch-command actual &&\n>> + test_cmp expect actual\n>> +'\n> \n> indentation with spaces, \\t correctly used for the rest.\n"},{"id":"447779","messageId":"3F83E252-D99C-47CB-B47F-11B8EA554A4F@gmail.com","threadId":"57365","inReplyTo":"CAPig+cQHcstj28F-==oCakp3iMLHtd1SVQV+snzewYihHxNG6g@mail.gmail.com","subject":"Re: [PATCH 2/2] catfile.c: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-04T21:41:56Z","receivedAt":"2022-02-04T21:42:03Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Eric,\n\nI appreciate the feedback and the thorough review!\n\nOn 4 Feb 2022, at 1:45, Eric Sunshine wrote:\n\n> ()()On Thu, Feb 3, 2022 at 7:20 PM John Cai via GitGitGadget\n> wrote:\n>> Add new flag --batch-command that will accept commands and arguments\n>> from stdin, similar to git-update-ref --stdin.\n>>\n>> contents NL\n>> info NL\n>>\n>> With the following commands, process (A) can begin a \"session\" and\n>> send a list of object names over stdin. When \"get contents\" or \"get info\"\n>> is issued, this list of object names will be fed into batch_one_object()\n>> to retrieve either info or contents. Finally an fflush() will be called\n>> to end the session.\n>>\n>> begin\n>> \n>> get info\n>>\n>> begin\n>> \n>> get contents\n>\n> I had the same reaction to this new command set as Junio expressed\n> upstream[1], and was prepared to suggest an alternative but Junio said\n> everything I wanted to say, so I won't say anything more about it\n> here.\n>\n> That aside, the implementation presented here is overly loose about\n> malformed input and accepts all sorts of bogus invocations which it\n> should be reporting as errors. For instance, a lone:\n>\n> get info\n>\n> without a preceding `begin` should be an error but such bogus input is\n> not diagnosed. `get contents` is likewise affected. Another example:\n>\n> begin\n> \n> \n>\n> which lacks a closing `get info` or `get contents` is silently\n> ignored. Similarly, malformed:\n>\n> begin\n> begin\n> ...\n>\n> should be reported as an error but is not.\n>\n> There is also a bug in which the accumulated list of OID's is never\n> cleared. Thus:\n>\n> begin\n> \n> get info\n> get info\n>\n> emits info about twice, once for each `get info` invocation. Similarly:\n>\n> begin\n> \n> get info\n> \n> get info\n>\n> emits information about twice and once. Invoking `begin`\n> between the `get info` commands doesn't help because `begin` is a\n> no-op. Thus:\n>\n> begin\n> \n> get info\n> begin\n> \n> get info\n>\n> likewise emits information about twice and once.\n>\n> It also incorrectly accepts non-\"session\" commands in the middle of a\n> session. For instance:\n>\n> begin\n> \n> info \n> \n>\n> immediately emits information about and then bombs out claiming\n> that is an \"unknown command\" because the `info` command --\n> which should not have been allowed within a session -- prematurely\n> ended the \"session\".\n>\n> The `info` and `contents` commands neglect to do any sort of\n> validation of their arguments, thus any and all bogus invocations are\n> accepted. Thus:\n>\n> info\n> info \n> info \n>\n> are all accepted as valid invocations, misleadingly producing \"\n> missing\" messages, rather than erroring out as they should. The \"\n> missing\" message should be reserved for the case when the lone\n> argument to `info` or `contents` is something which looks like a\n> legitimate OID.\n\nSo actually the argument to info and contents doesn't have to be an OID but an object name that\neventually gets passed to get_oid_with_context() to resolve to an actual oid. This is the\nsame behavior as git cat-file --batch and --batch-check, neither of which throws an error. My\ngoal was to maintain this behavior in --batch-command.\n>\n> The above is a long-winded way of saying that it is important not only\n> to check the obvious \"expect success\" cases when implementing a new\n> feature, but it's also important to add checks for the \"expect\n> failure\" cases, such as all the above malformed inputs.\n\nI overall agree and will add test coverage for invalid input.\n\n> It's subjective, but it feels like this implementation is trying to be\n> too clever by handling all these cases via a single strbuf_getline()\n> loop in batch_objects_command(). It would be easier to reason about,\n> and would have avoided some of the above problems, for instance, if\n> handling of `begin` was split out to its own function which looped\n> over strbuf_getline() itself, thus could easily have detected\n> premature EOF (lacking `get info` or `get contents`) and likewise\n> would not have allowed `info` or `contents` commands to be executed\n> within a \"session\".\n>\n> Similarly (again subjective), the generic command dispatcher seems\n> over-engineered and perhaps contributed to the oversight of `info` and\n> `contents` failing to perform validation of their arguments. A simpler\n> hand-rolled command-response loop is more common on this project and\n> often easier to reason about. Perhaps something like this\n> (pseudo-code):\n>\n> while (strbuf_getline(&input, stdin)) {\n> char *end_cmd = strchrnul(&input.buf, ' ');\n> const char *argstart = *end_cmd ? end_cmd + 1 : end_cmd;\n> *end_cmd = '\\0';\n> if (strcmp(input.buf, \"info\"))\n> show_info(argstart);\n> else if (strcmp(input.buf, \"contents\"))\n> show_contents(argstart);\n> else if (strcmp(input.buf, \"begin\"))\n> begin_session(argstart);\n> else\n> die(...);\n> }\n\nI think I agree that the code in this patch is a little hard to follow,\nbut now I'm planning to simplify the design by getting rid of \"begin\"[2].\n\nI'm hoping this change will make the code easier to reason about.\n\n2. https://lore.kernel.org/git/767d5f5a-8395-78bc-865f-a39acc39e061@gmail.com/\n\n>\n> and each of the command-handler functions would perform its own\n> argument validation (including the case when no argument should be\n> present).\n>\n> [1]: https://lore.kernel.org/git/xmqqo83nsoxs.fsf@gitster.g/\n>\n>> +static void parse_cmd_begin(struct batch_options *opt,\n>> + const char *line,\n>> + struct strbuf *output,\n>> + struct expand_data *data,\n>> + struct string_list revs)\n>> +{\n>> + /* nothing needs to be done here */\n>> +}\n>\n> Either this function should be clearing the list of collected OID's or...\n>\n>> +static void parse_cmd_get(struct batch_options *opt,\n>> + const char *line,\n>> + struct strbuf *output,\n>> + struct expand_data *data,\n>> + struct string_list revs)\n>> +{\n>> + struct string_list_item *item;\n>> + for_each_string_list_item(item, &revs) {\n>> + batch_one_object(item->string, output, opt, data);\n>> + }\n>> + if (opt->buffer_output)\n>> + fflush(stdout);\n>\n> ... this function should do so after it's done processing the OID list.\n>\n>> +}\n>> +static void parse_cmd_get_info(struct batch_options *opt,\n>> + const char *line,\n>> + struct strbuf *output,\n>> + struct expand_data *data,\n>> + struct string_list revs)\n>\n> Missing blank line before the function definition.\n>\n>> +static void parse_cmd_get_objects(struct batch_options *opt,\n>> + const char *line,\n>> + struct strbuf *output,\n>> + struct expand_data *data,\n>> + struct string_list revs)\n>> +{\n>> + opt->print_contents = 1;\n>> + parse_cmd_get(opt, line, output, data, revs);\n>> + if (opt->buffer_output)\n>> + fflush(stdout);\n>> +}\n>\n> Why does this function duplicate the flushing logic of parse_cmd_get()\n> immediately after calling parse_cmd_get()?\n>\n>> +static void batch_objects_command(struct batch_options *opt,\n>> + struct strbuf *output,\n>> + struct expand_data *data)\n>> +{\n>> + /* Read each line dispatch its command */\n>\n> Missing \"and\".\n>\n>> + while (!strbuf_getline(&input, stdin)) {\n>> + if (state == BATCH_STATE_COMMAND) {\n>> + if (*input.buf == '\\n')\n>> + die(\"empty command in input\");\n>\n> This never triggers because strbuf_getline() removes the line\n> terminator before returning.\n\nyes, this was an oversight. thanks for catching it!\n\n>\n>> + else if (isspace(*input.buf))\n>> + die(\"whitespace before command: %s\", input.buf);\n>> + }\n>> +\n>> + for (i = 0; i < ARRAY_SIZE(commands); i++) {\n>> + if (!skip_prefix(input.buf, prefix, &cmd_end))\n>> + continue;\n>> + /*\n>> + * If the command has arguments, verify that it's\n>> + * followed by a space. Otherwise, it shall be followed\n>> + * by a line terminator.\n>> + */\n>> + c = commands[i].takes_args ? ' ' : '\\n';\n>> + if (*cmd_end && *cmd_end != c)\n>> + die(\"arguments invalid for command: %s\", commands[i].prefix);\n>\n> Ditto regarding strbuf_getline() removing line-terminator.\n>\n>> + cmd = &commands[i];\n>> + if (cmd->takes_args)\n>> + p = cmd_end + 1;\n>> + break;\n>> + }\n>> +\n>> + if (input.buf[input.len - 1] == '\\n')\n>> + input.buf[--input.len] = '\\0';\n>\n> Ditto again. This is especially scary since it's accessing element\n> `input.len-1` without even checking if `input.len` is greater than\n> zero.\n\nI realize we actually don't need this if block at all because strubf_getline()\nstrips the line terminator for us already.\n\n>\n> Also, it's better to use published strbuf API rather than mucking with\n> the internals:\n>\n> strbuf_setlen(&input, input.len - 1);\n>\n>> + if (state == BATCH_STATE_INPUT && !cmd){\n>> + string_list_append(&revs, input.buf);\n>> + continue;\n>> + }\n>> +\n>> + if (!cmd)\n>> + die(\"unknown command: %s\", input.buf);\n>> +\n>> + state = cmd->next_state;\n>> + cmd->fn(opt, p, output, data, revs);\n>> + }\n>> + strbuf_release(&input);\n>> + string_list_clear(&revs, 0);\n>> +}\n"},{"id":"447814","messageId":"CAPig+cTeqhOYTu9WBiY=LnZtt35hAp3Qa5RduC2yLut6p01_1w@mail.gmail.com","threadId":"57365","inReplyTo":"3F83E252-D99C-47CB-B47F-11B8EA554A4F@gmail.com","subject":"Re: [PATCH 2/2] catfile.c: add --batch-command mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-05T06:52:26Z","receivedAt":"2022-02-05T06:52:40Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Feb 4, 2022 at 4:42 PM John Cai wrote:\n> On 4 Feb 2022, at 1:45, Eric Sunshine wrote:\n> > The `info` and `contents` commands neglect to do any sort of\n> > validation of their arguments, thus any and all bogus invocations are\n> > accepted. Thus:\n> >\n> > info\n> > info \n> > info \n> >\n> > are all accepted as valid invocations, misleadingly producing \"\n> > missing\" messages, rather than erroring out as they should. The \"\n> > missing\" message should be reserved for the case when the lone\n> > argument to `info` or `contents` is something which looks like a\n> > legitimate OID.\n>\n> So actually the argument to info and contents doesn't have to be an OID but an object name that\n> eventually gets passed to get_oid_with_context() to resolve to an actual oid. This is the\n> same behavior as git cat-file --batch and --batch-check, neither of which throws an error. My\n> goal was to maintain this behavior in --batch-command.\n\nOkay, that makes sense. I was misled by the commit message talking\nabout \"\" in its examples, and didn't do my due diligence by\ndigging more deeply into the existing documentation and code.\n"},{"id":"447890","messageId":"pull.1212.v2.git.git.1644251611.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.git.git.1643915286.gitgitgadget@gmail.com","subject":"[PATCH v2 0/2] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-07T16:33:29Z","receivedAt":"2022-02-07T16:46:31Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has two parts:\n\n 1. preparation patch to rename a variable\n 2. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (2):\n cat-file: rename cmdmode to transform_mode\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 19 ++++\n builtin/cat-file.c | 138 +++++++++++++++++++++--\n t/t1006-cat-file.sh | 197 ++++++++++++++++++++++++++++++++-\n 3 files changed, 346 insertions(+), 8 deletions(-)\n\n\nbase-commit: 38062e73e009f27ea192d50481fcb5e7b0e9d6eb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v2\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v1:\n\n 1: 86df0c9e4df ! 1: 2d9a0b06ce5 cat-file.c: rename cmdmode to mode\n @@ Metadata\n Author: John Cai \n \n ## Commit message ##\n - cat-file.c: rename cmdmode to mode\n + cat-file: rename cmdmode to transform_mode\n \n - To prepare for a new flag --batch-command, we will add a flag that\n - indicates whether or not an interactive command mode will be used\n - that reads commands and arguments off of stdin.\n + When introducing a new flag --batch-command, we will add a flag on the\n + batch_options struct that indicates whether or not an interactive\n + command mode will be used that reads commands and arguments off of\n + stdin.\n \n An intuitive name for this flag would be \"command\", which can get\n confusing with the already existing cmdmode.\n \n - Rename cmdmode->mode to prepare for this change.\n + cmdmode refers to how the result output of the blob will be transformed,\n + either according to --filter or --textconv. So transform_mode is a more\n + descriptive name for the flag, and will not get confused with the new\n + command flag to be added in the next commit.\n +\n + Rename cmdmode to transform_mode in cat-file.c\n \n Signed-off-by: John Cai \n \n @@ builtin/cat-file.c: struct batch_options {\n \tint all_objects;\n \tint unordered;\n -\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n -+\tint mode; /* may be 'w' or 'c' for --filters or --textconv */\n ++\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n @@ builtin/cat-file.c: static void print_object_or_die(struct batch_options *opt, s\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n -\t\tif (opt->cmdmode) {\n -+\t\tif (opt->mode) {\n ++\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n @@ builtin/cat-file.c: static void print_object_or_die(struct batch_options *opt, s\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n -\t\t\tif (opt->cmdmode == 'w') {\n -+\t\t\tif (opt->mode == 'w') {\n ++\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n -\t\t\t} else if (opt->cmdmode == 'c') {\n -+\t\t\t} else if (opt->mode == 'c') {\n ++\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n @@ builtin/cat-file.c: static void print_object_or_die(struct batch_options *opt, s\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n -\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n -+\t\t\t\tBUG(\"invalid mode: %c\", opt->mode);\n ++\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n -\tif (opt->cmdmode)\n -+\tif (opt->mode)\n ++\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *prefix)\n - \n - \tbatch.buffer_output = -1;\n - \targc = parse_options(argc, argv, prefix, options, cat_file_usage, 0);\n --\n - \tif (opt) {\n - \t\tif (batch.enabled && (opt == 'c' || opt == 'w'))\n --\t\t\tbatch.cmdmode = opt;\n -+\t\t\tbatch.mode = opt;\n - \t\telse if (argc == 1)\n - \t\t\tobj_name = argv[0];\n - \t\telse\n -@@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *prefix)\n - \t\t\tusage_with_options(cat_file_usage, options);\n - \t}\n + \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n --\t\tif (batch.cmdmode != opt || argc)\n -+\t\tif (batch.mode != opt || argc)\n - \t\t\tusage_with_options(cat_file_usage, options);\n --\t\tif (batch.cmdmode && batch.all_objects)\n -+\t\tif (batch.mode && batch.all_objects)\n - \t\t\tdie(\"--batch-all-objects cannot be combined with \"\n - \t\t\t \"--textconv nor with --filters\");\n - \t}\n + \t\tif (opt_cw)\n +-\t\t\tbatch.cmdmode = opt;\n ++\t\t\tbatch.transform_mode = opt;\n + \t\telse if (opt && opt != 'b')\n + \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n + \t\t\t\t usage, options, opt);\n 2: ebd2a135601 ! 2: 1b63164ad4d catfile.c: add --batch-command mode\n @@ Metadata\n Author: John Cai \n \n ## Commit message ##\n - catfile.c: add --batch-command mode\n + cat-file: add --batch-command mode\n \n - Add new flag --batch-command that will accept commands and arguments\n + Add a new flag --batch-command that accepts commands and arguments\n from stdin, similar to git-update-ref --stdin.\n \n At GitLab, we use a pair of long running cat-file processes when\n accessing object content. One for iterating over object metadata with\n --batch-check, and the other to grab object contents with --batch.\n \n - However, if we had --batch-command, we wouldnt need to keep both\n - processes around, and instead just have one process where we can flip\n - between getting object info, and getting object contents. This means we\n - can get rid of roughly half of long lived git cat-file processes. This\n - can lead to huge savings since on a given server there could be hundreds\n - of git cat-file processes running.\n + However, if we had --batch-command, we wouldn't need to keep both\n + processes around, and instead just have one --batch-command process\n + where we can flip between getting object info, and getting object\n + contents. Since we have a pair of cat-file processes per repository,\n + this means we can get rid of roughly half of long lived git cat-file\n + processes. Given there are many repositories being accessed at any given\n + time, this can lead to huge savings since on a given server.\n \n git cat-file --batch-command\n \n - would enter an interactive command mode whereby the user can enter in\n - commands and their arguments:\n + will enter an interactive command mode whereby the user can enter in\n + commands and their arguments that get queued in memory:\n \n - [arg1] [arg2] NL\n + [arg1] [arg2] NL\n + [arg1] [arg2] NL\n \n - This patch adds the basic structure for add command which can be\n - extended in the future to add more commands.\n + When --buffer mode is used, commands will be queued in memory until a\n + flush command is issued that execute them:\n \n - This patch also adds the following commands:\n + flush NL\n +\n + The reason for a flush command is that when a consumer process (A)\n + talks to a git cat-file process (B) and interactively writes to and\n + reads from it in --buffer mode, (A) needs to be able to control when\n + the buffer is flushed to stdout.\n +\n + Currently, from (A)'s perspective, the only way is to either\n +\n + 1. kill (B)'s process\n + 2. send an invalid object to stdin.\n +\n + 1. is not ideal from a performance perspective as it will require\n + spawning a new cat-file process each time, and 2. is hacky and not a\n + good long term solution.\n +\n + With this mechanism of queueing up commands and letting (A) issue a\n + flush command, process (A) can control when the buffer is flushed and\n + can guarantee it will receive all of the output when in --buffer mode.\n +\n + This patch adds the basic structure for adding command which can be\n + extended in the future to add more commands. It also adds the following\n + two commands (on top of the flush command):\n \n contents NL\n info NL\n @@ Commit message\n The info command takes a argument and prints out the object\n metadata.\n \n - In addition, we need a set of commands that enable a \"read session\".\n -\n - When a separate process (A) is connected to a git cat-file process (B)\n - and is interactively writing to and reading from it in --buffer mode,\n - (A) needs to be able to know when the buffer is flushed to stdout.\n - Currently, from (A)'s perspective, the only way is to either 1. exit\n - (B)'s process or 2. send an invalid object to stdin. 1. is not ideal\n - from a performance perspective as it will require spawning a new cat-file\n - process each time, and 2. is hacky and not a good long term solution.\n + These can be used in the following way with --buffer:\n \n - With the following commands, process (A) can begin a \"session\" and\n - send a list of object names over stdin. When \"get contents\" or \"get info\"\n - is issued, this list of object names will be fed into batch_one_object()\n - to retrieve either info or contents. Finally an fflush() will be called\n - to end the session.\n + contents NL\n + object NL\n + object NL\n + contents NL\n + flush\n + contents NL\n + flush\n \n - begin NL\n - get contents NL\n - get info NL\n + When used without --buffer:\n \n - These can be used in the following way:\n -\n - begin\n - \n - \n - \n - \n - get info\n -\n - begin\n - \n - \n - \n - \n - get contents\n -\n - With this mechanism, process (A) can be guaranteed to receive all of the\n - output even in --buffer mode.\n + contents NL\n + object NL\n + object NL\n + contents NL\n + contents NL\n \n Helped-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: John Cai \n @@ Documentation/git-cat-file.txt: OPTIONS\n +info ::\n +\tPrint object info for object reference \n +\n -+begin::\n -+\tBegins a session to read object names off of stdin. A session can be\n -+\tterminated with `get contents` or `get info`.\n -+\n -+get contents::\n -+\tAfter a read session is begun with the `begin` command, and object\n -+\tnames have been fed into stdin, end the session and retrieve contents of\n -+\tall the objects requested.\n -+\n -+get info::\n -+\tAfter a read session is begun with the `begin` command, and object\n -+\tnames have been fed into stdin, end the session and retrieve info of\n -+\tall the objects requested.\n ++flush::\n ++\tExecute all preceding commands that were issued since the beginning or\n ++\tsince the last flush command was issued. Only used with --buffer. When\n ++\t--buffer is not used, commands are flushed each time without issuing\n ++\t`flush`.\n +\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n @@ Documentation/git-cat-file.txt: OPTIONS\n ## builtin/cat-file.c ##\n @@ builtin/cat-file.c: struct batch_options {\n \tint unordered;\n - \tint mode; /* may be 'w' or 'c' for --filters or --textconv */\n + \tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n +\tint command;\n };\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n \t\t\t\t data);\n }\n \n -+static void parse_cmd_object(struct batch_options *opt,\n ++typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n ++\t\t\t struct strbuf *, struct expand_data *);\n ++\n ++struct queued_cmd {\n ++\tparse_cmd_fn_t fn;\n ++\tconst char *line;\n ++};\n ++\n ++static void parse_cmd_contents(struct batch_options *opt,\n +\t\t\t const char *line,\n +\t\t\t struct strbuf *output,\n -+\t\t\t struct expand_data *data,\n -+\t\t\t struct string_list revs)\n ++\t\t\t struct expand_data *data)\n +{\n +\topt->print_contents = 1;\n +\tbatch_one_object(line, output, opt, data);\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +static void parse_cmd_info(struct batch_options *opt,\n +\t\t\t const char *line,\n +\t\t\t struct strbuf *output,\n -+\t\t\t struct expand_data *data,\n -+\t\t\t struct string_list revs)\n ++\t\t\t struct expand_data *data)\n +{\n +\topt->print_contents = 0;\n +\tbatch_one_object(line, output, opt, data);\n +}\n +\n -+static void parse_cmd_begin(struct batch_options *opt,\n -+\t\t\t const char *line,\n -+\t\t\t struct strbuf *output,\n -+\t\t\t struct expand_data *data,\n -+\t\t\t struct string_list revs)\n ++static void flush_batch_calls(struct batch_options *opt,\n ++\t\tstruct strbuf *output,\n ++\t\tstruct expand_data *data,\n ++\t\tstruct queued_cmd *cmds,\n ++\t\tint queued)\n +{\n -+\t/* nothing needs to be done here */\n -+}\n -+\n -+static void parse_cmd_get(struct batch_options *opt,\n -+\t\t\t const char *line,\n -+\t\t\t struct strbuf *output,\n -+\t\t\t struct expand_data *data,\n -+\t\t\t struct string_list revs)\n -+{\n -+\tstruct string_list_item *item;\n -+\tfor_each_string_list_item(item, &revs) {\n -+\t\tbatch_one_object(item->string, output, opt, data);\n ++\tint i;\n ++\tfor(i = 0; i < queued; i++){\n ++\t\tcmds[i].fn(opt, cmds[i].line, output, data);\n +\t}\n -+\tif (opt->buffer_output)\n -+\t\tfflush(stdout);\n -+}\n -+static void parse_cmd_get_info(struct batch_options *opt,\n -+\t\t\t const char *line,\n -+\t\t\t struct strbuf *output,\n -+\t\t\t struct expand_data *data,\n -+\t\t\t struct string_list revs)\n -+{\n -+\topt->print_contents = 0;\n -+\tparse_cmd_get(opt, line, output, data, revs);\n -+}\n -+\n -+static void parse_cmd_get_objects(struct batch_options *opt,\n -+\t\t\t const char *line,\n -+\t\t\t struct strbuf *output,\n -+\t\t\t struct expand_data *data,\n -+\t\t\t struct string_list revs)\n -+{\n -+\topt->print_contents = 1;\n -+\tparse_cmd_get(opt, line, output, data, revs);\n -+\tif (opt->buffer_output)\n -+\t\tfflush(stdout);\n ++\tfflush(stdout);\n +}\n +\n -+enum batch_state {\n -+\tBATCH_STATE_COMMAND,\n -+\tBATCH_STATE_INPUT,\n -+};\n -+\n -+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n -+\t\t\t struct strbuf *, struct expand_data *,\n -+\t\t\t struct string_list revs);\n -+\n +static const struct parse_cmd {\n +\tconst char *prefix;\n +\tparse_cmd_fn_t fn;\n +\tunsigned takes_args;\n -+\tenum batch_state next_state;\n +} commands[] = {\n -+\t{ \"contents\", parse_cmd_object, 1, BATCH_STATE_COMMAND},\n -+\t{ \"info\", parse_cmd_info, 1, BATCH_STATE_COMMAND},\n -+\t{ \"begin\", parse_cmd_begin, 0, BATCH_STATE_INPUT},\n -+\t{ \"get info\", parse_cmd_get_info, 0, BATCH_STATE_COMMAND},\n -+\t{ \"get contents\", parse_cmd_get_objects, 0, BATCH_STATE_COMMAND},\n ++\t{ \"contents\", parse_cmd_contents, 1},\n ++\t{ \"info\", parse_cmd_info, 1},\n +};\n +\n +static void batch_objects_command(struct batch_options *opt,\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\t\t struct expand_data *data)\n +{\n +\tstruct strbuf input = STRBUF_INIT;\n -+\tenum batch_state state = BATCH_STATE_COMMAND;\n -+\tstruct string_list revs = STRING_LIST_INIT_DUP;\n ++\tstruct queued_cmd *cmds = NULL;\n ++\tsize_t alloc = 0, nr = 0;\n ++\tint queued = 0;\n +\n -+\t/* Read each line dispatch its command */\n +\twhile (!strbuf_getline(&input, stdin)) {\n +\t\tint i;\n +\t\tconst struct parse_cmd *cmd = NULL;\n +\t\tconst char *p, *cmd_end;\n ++\t\tstruct queued_cmd call = {0};\n ++\n ++\t\tif (!input.len)\n ++\t\t\tdie(_(\"empty command in input\"));\n ++\t\tif (isspace(*input.buf))\n ++\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n +\n -+\t\tif (state == BATCH_STATE_COMMAND) {\n -+\t\t\tif (*input.buf == '\\n')\n -+\t\t\t\tdie(\"empty command in input\");\n -+\t\t\telse if (isspace(*input.buf))\n -+\t\t\t\tdie(\"whitespace before command: %s\", input.buf);\n ++\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n ++\t\t\tif (!opt->buffer_output)\n ++\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n ++\t\t\tif (*cmd_end)\n ++\t\t\t\tdie(_(\"flush takes no arguments\"));\n ++\t\t\tif (!queued)\n ++\t\t\t\tdie(_(\"nothing to flush\"));\n ++\t\t\tflush_batch_calls(opt, output, data, cmds, queued);\n ++\t\t\tqueued = 0;\n ++\t\t\tcontinue;\n +\t\t}\n +\n +\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n -+\t\t\tconst char *prefix = commands[i].prefix;\n -+\t\t\tchar c;\n -+\t\t\tif (!skip_prefix(input.buf, prefix, &cmd_end))\n ++\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n +\t\t\t\tcontinue;\n -+\t\t\t/*\n -+\t\t\t * If the command has arguments, verify that it's\n -+\t\t\t * followed by a space. Otherwise, it shall be followed\n -+\t\t\t * by a line terminator.\n -+\t\t\t */\n -+\t\t\tc = commands[i].takes_args ? ' ' : '\\n';\n -+\t\t\tif (*cmd_end && *cmd_end != c)\n -+\t\t\t\tdie(\"arguments invalid for command: %s\", commands[i].prefix);\n +\n +\t\t\tcmd = &commands[i];\n +\t\t\tif (cmd->takes_args)\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\tbreak;\n +\t\t}\n +\n -+\t\tif (input.buf[input.len - 1] == '\\n')\n -+\t\t\tinput.buf[--input.len] = '\\0';\n ++\t\tif (!cmd)\n ++\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n +\n -+\t\tif (state == BATCH_STATE_INPUT && !cmd){\n -+\t\t\tstring_list_append(&revs, input.buf);\n ++\t\tif (!opt->buffer_output) {\n ++\t\t\tcmd->fn(opt, p, output, data);\n +\t\t\tcontinue;\n +\t\t}\n +\n -+\t\tif (!cmd)\n -+\t\t\tdie(\"unknown command: %s\", input.buf);\n ++\t\tqueued++;\n ++\t\tif (queued > nr) {\n ++\t\t\tALLOC_GROW(cmds, nr+1, alloc);\n ++\t\t\tnr++;\n ++\t\t}\n +\n -+\t\tstate = cmd->next_state;\n -+\t\tcmd->fn(opt, p, output, data, revs);\n ++\t\tcall.fn = cmd->fn;\n ++\t\tcall.line = xstrdup(p);\n ++\t\tcmds[queued-1] = call;\n +\t}\n ++\tfree(cmds);\n +\tstrbuf_release(&input);\n -+\tstring_list_clear(&revs, 0);\n +}\n +\n static int batch_objects(struct batch_options *opt)\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n --\twhile (strbuf_getline(&input, stdin) != EOF) {\n --\t\tif (data.split_on_whitespace) {\n --\t\t\t/*\n --\t\t\t * Split at first whitespace, tying off the beginning\n --\t\t\t * of the string and saving the remainder (or NULL) in\n --\t\t\t * data.rest.\n --\t\t\t */\n --\t\t\tchar *p = strpbrk(input.buf, \" \\t\");\n --\t\t\tif (p) {\n --\t\t\t\twhile (*p && strchr(\" \\t\", *p))\n --\t\t\t\t\t*p++ = '\\0';\n -+\tif (command)\n ++\tif (command) {\n +\t\tbatch_objects_command(opt, &output, &data);\n -+\telse {\n -+\t\twhile (strbuf_getline(&input, stdin) != EOF) {\n -+\t\t\tif (data.split_on_whitespace) {\n -+\t\t\t\t/*\n -+\t\t\t\t * Split at first whitespace, tying off the beginning\n -+\t\t\t\t * of the string and saving the remainder (or NULL) in\n -+\t\t\t\t * data.rest.\n -+\t\t\t\t */\n -+\t\t\t\tchar *p = strpbrk(input.buf, \" \\t\");\n -+\t\t\t\tif (p) {\n -+\t\t\t\t\twhile (*p && strchr(\" \\t\", *p))\n -+\t\t\t\t\t\t*p++ = '\\0';\n -+\t\t\t\t}\n -+\t\t\t\tdata.rest = p;\n - \t\t\t}\n --\t\t\tdata.rest = p;\n -+\t\t\tbatch_one_object(input.buf, &output, opt, &data);\n - \t\t}\n --\n --\t\tbatch_one_object(input.buf, &output, opt, &data);\n ++\t\tgoto cleanup;\n ++\t}\n + \twhile (strbuf_getline(&input, stdin) != EOF) {\n + \t\tif (data.split_on_whitespace) {\n + \t\t\t/*\n +@@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n + \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n ++ cleanup:\n \tstrbuf_release(&input);\n + \tstrbuf_release(&output);\n + \twarn_on_object_refname_ambiguity = save_warning;\n @@ builtin/cat-file.c: static int batch_option_callback(const struct option *opt,\n \n \tbo->enabled = 1;\n @@ builtin/cat-file.c: static int batch_option_callback(const struct option *opt,\n \n \treturn 0;\n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *prefix)\n - \t\t\tN_(\"show info about objects fed from the standard input\"),\n + \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n -+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"\"),\n -+\t\t\t N_(\"enters batch mode that accepts commands\"),\n -+\t\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n -+\t\t\t batch_option_callback),\n -+\n - \t\tOPT_BOOL(0, \"follow-symlinks\", &batch.follow_symlinks,\n - \t\t\t N_(\"follow in-tree symlinks (used with --batch or --batch-check)\")),\n - \t\tOPT_BOOL(0, \"batch-all-objects\", &batch.all_objects,\n ++\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n ++\t\t\tN_(\"read commands from stdin\"),\n ++\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n ++\t\t\tbatch_option_callback),\n + \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n + \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n + \t\t/* Batch-specific options */\n \n ## t/t1006-cat-file.sh ##\n @@ t/t1006-cat-file.sh: $content\"\n @@ t/t1006-cat-file.sh: $content\"\n + test -z \"$content\" ||\n + test_expect_success \"--batch-command output of $type content is correct\" '\n +\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n -+\tmaybe_remove_timestamp \"$(echo contents $sha1 | git cat-file --batch-command)\" $no_ts >actual &&\n -+\ttest_cmp expect actual\n -+ '\n -+\n -+ test -z \"$content\" ||\n -+ test_expect_success \"--batch-command session for $type content is correct\" '\n -+\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n -+\tmaybe_remove_timestamp \\\n -+\t\t\"$(test_write_lines \"begin\" \"$sha1\" \"get contents\" | git cat-file --batch-command)\" \\\n -+\t\t$no_ts >actual &&\n ++\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n ++\t| git cat-file --batch-command)\" $no_ts >actual &&\n +\ttest_cmp expect actual\n + '\n +\n + test_expect_success \"--batch-command output of $type info is correct\" '\n +\techo \"$sha1 $type $size\" >expect &&\n -+\techo \"info $sha1\" | git cat-file --batch-command >actual &&\n -+\ttest_cmp expect actual\n -+ '\n -+\n -+ test_expect_success \"--batch-command session for $type info is correct\" '\n -+\techo \"$sha1 $type $size\" >expect &&\n -+\ttest_write_lines \"begin\" \"$sha1\" \"get info\" | git cat-file --batch-command >actual &&\n ++\ttest_write_lines \"info $sha1\" | git cat-file --batch-command >actual &&\n +\ttest_cmp expect actual\n + '\n +\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n -@@ t/t1006-cat-file.sh: test_expect_success '--batch-check without %(rest) considers whole line' '\n - '\n +@@ t/t1006-cat-file.sh: $content\"\n + '\n + }\n \n - tree_sha1=$(git write-tree)\n ++run_buffer_test_flush () {\n ++\ttype=$1\n ++\tsha1=$2\n ++\tsize=$3\n ++\n ++\tmkfifo input &&\n ++\ttest_when_finished 'rm input; exec 8<&-' &&\n ++\tmkfifo output &&\n ++\texec 9<>output &&\n ++\ttest_when_finished 'rm output; exec 9<&-'\n ++\t(\n ++\t\tgit cat-file --buffer --batch-command err &\n ++\t\techo $! &&\n ++\t\twait $!\n ++\t) >&9 &\n ++\tsh_pid=$! &&\n ++\tread cat_file_pid <&9 &&\n ++\ttest_when_finished \"kill $cat_file_pid\n ++\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n ++\techo \"$sha1 $type $size\" >expect &&\n ++\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n ++\t# TODO - consume all available input, not just one\n ++\t# line (see above).\n ++\t# check output is flushed on exit\n ++\tread actual <&9 &&\n ++\techo \"$actual\" >actual &&\n ++\ttest_cmp expect actual &&\n ++\ttest_must_be_empty err\n ++}\n ++\n ++run_buffer_test_no_flush () {\n ++\ttype=$1\n ++\tsha1=$2\n ++\tsize=$3\n ++\n ++\ttouch output &&\n ++\ttest_when_finished 'rm output' &&\n ++\tmkfifo input &&\n ++\ttest_when_finished 'rm input' &&\n ++\tmkfifo pid &&\n ++\texec 9<>pid &&\n ++\ttest_when_finished 'rm pid; exec 9<&-'\n ++\t(\n ++\t\tgit cat-file --buffer --batch-command output &\n ++\t\techo $! &&\n ++\t\twait $!\n ++\t\techo $?\n ++\t) >&9 &\n ++\tsh_pid=$! &&\n ++\tread cat_file_pid <&9 &&\n ++\ttest_when_finished \"kill $cat_file_pid\n ++\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n ++\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n ++\tkill $cat_file_pid &&\n ++\tread status <&9 &&\n ++\ttest_must_be_empty output\n ++}\n +\n - tree_size=$(($(test_oid rawsz) + 13))\n - tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n + hello_content=\"Hello World\"\n + hello_size=$(strlen \"$hello_content\")\n + hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n +@@ t/t1006-cat-file.sh: test_expect_success \"setup\" '\n \n -@@ t/t1006-cat-file.sh: test_expect_success \\\n + run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n + \n ++test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n ++ run_buffer_test_flush blob $hello_sha1 $hello_size\n ++'\n ++\n ++test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n ++ run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n ++'\n ++\n + test_expect_success '--batch-check without %(rest) considers whole line' '\n + \techo \"$hello_sha1 blob $hello_size\" >expect &&\n + \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n +@@ t/t1006-cat-file.sh: tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n + \n + run_tests 'tree' $tree_sha1 $tree_size \"\" \"$tree_pretty_content\"\n + \n ++test_expect_success PIPE '--batch-command --buffer with flush for tree info' '\n ++ run_buffer_test_flush tree $tree_sha1 $tree_size\n ++'\n ++\n ++test_expect_success PIPE '--batch-command --buffer without flush for tree info' '\n ++ run_buffer_test_no_flush tree $tree_sha1 $tree_size false\n ++'\n ++\n + commit_message=\"Initial commit\"\n + commit_sha1=$(echo_without_newline \"$commit_message\" | git commit-tree $tree_sha1)\n + commit_size=$(($(test_oid hexsz) + 137))\n +@@ t/t1006-cat-file.sh: $commit_message\"\n + \n + run_tests 'commit' $commit_sha1 $commit_size \"$commit_content\" \"$commit_content\" 1\n + \n ++test_expect_success PIPE '--batch-command --buffer with flush for commit info' '\n ++ run_buffer_test_flush commit $commit_sha1 $commit_size\n ++'\n ++\n ++test_expect_success PIPE '--batch-command --buffer without flush for commit info' '\n ++ run_buffer_test_no_flush commit $commit_sha1 $commit_size false\n ++'\n ++\n + tag_header_without_timestamp=\"object $hello_sha1\n + type blob\n + tag hellotag\n +@@ t/t1006-cat-file.sh: tag_size=$(strlen \"$tag_content\")\n + \n + run_tests 'tag' $tag_sha1 $tag_size \"$tag_content\" \"$tag_content\" 1\n + \n ++test_expect_success PIPE '--batch-command --buffer with flush for tag info' '\n ++ run_buffer_test_flush tag $tag_sha1 $tag_size\n ++'\n ++\n ++test_expect_success PIPE '--batch-command --buffer without flush for tag info' '\n ++ run_buffer_test_no_flush tag $tag_sha1 $tag_size false\n ++'\n ++\n + test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n -+test_expect_success \"--batch-command with multiple sha1s gives correct format\" '\n -+ echo \"$batch_check_output\" >expect &&\n -+ echo begin >input &&\n -+ echo_without_newline \"$batch_check_input\" >>input &&\n -+ echo \"get info\" >>input &&\n -+ git cat-file --batch-command actual &&\n -+ test_cmp expect actual\n ++batch_command_info_input=\"info $hello_sha1\n ++info $tree_sha1\n ++info $commit_sha1\n ++info $tag_sha1\n ++info deadbeef\n ++info \n ++flush\n ++\"\n ++\n ++test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n ++\ttest \"$batch_check_output\" = \"$(echo_without_newline \\\n ++\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n ++'\n ++\n ++batch_command_contents_input=\"contents $hello_sha1\n ++contents $commit_sha1\n ++contents $tag_sha1\n ++contents deadbeef\n ++contents \n ++flush\n ++\"\n ++\n ++test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n ++\ttest \"$(maybe_remove_timestamp \"$batch_output\" 1)\" = \\\n ++\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n ++'\n ++\n ++batch_command_mixed_input=\"info $hello_sha1\n ++contents $hello_sha1\n ++info $commit_sha1\n ++contents $commit_sha1\n ++info $tag_sha1\n ++contents $tag_sha1\n ++contents deadbeef\n ++info \n ++flush\n ++\"\n ++\n ++batch_command_mixed_output=\"$hello_sha1 blob $hello_size\n ++$hello_sha1 blob $hello_size\n ++$hello_content\n ++$commit_sha1 commit $commit_size\n ++$commit_sha1 commit $commit_size\n ++$commit_content\n ++$tag_sha1 tag $tag_size\n ++$tag_sha1 tag $tag_size\n ++$tag_content\n ++deadbeef missing\n ++ missing\"\n ++\n ++test_expect_success \"--batch-command with mixed calls gives correct format\" '\n ++\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n ++\t\"$(maybe_remove_timestamp \"$(echo_without_newline \\\n ++\t\"$batch_command_mixed_input\" | git cat-file --batch-command --buffer)\" 1)\"\n +'\n +\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n + \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n - \n ++test_expect_success 'batch-command empty command' '\n ++\techo \"\" >cmd &&\n ++\ttest_expect_code 128 git cat-file --batch-command err &&\n ++\tgrep -E \"^fatal:.*empty command in input.*\" err\n ++'\n ++\n ++test_expect_success 'batch-command whitespace before command' '\n ++\techo \" info deadbeef\" >cmd &&\n ++\ttest_expect_code 128 git cat-file --batch-command err &&\n ++\tgrep -E \"^fatal:.*whitespace before command.*\" err\n ++'\n ++\n +test_expect_success 'batch-command unknown command' '\n +\techo unknown_command >cmd &&\n -+\ttest_expect_code 128 git cat-file --batch-command < cmd 2>err &&\n ++\ttest_expect_code 128 git cat-file --batch-command err &&\n +\tgrep -E \"^fatal:.*unknown command.*\" err\n +'\n +\n ++test_expect_success 'batch-command flush with arguments' '\n ++\techo \"flush arg\" >cmd &&\n ++\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n ++\tgrep -E \"^fatal:.*flush takes no arguments.*\" err\n ++'\n ++\n ++test_expect_success 'batch-command flush without --buffer' '\n ++\techo \"flush arg\" >cmd &&\n ++\ttest_expect_code 128 git cat-file --batch-command err &&\n ++\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n ++'\n ++\n ++test_expect_success 'batch-command flush empty queue' '\n ++\techo flush >cmd &&\n ++\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n ++\tgrep -E \"^fatal:.*nothing to flush.*\" err\n ++'\n + \n test_done\n\n-- \ngitgitgadget\n"},{"id":"447892","messageId":"2d9a0b06ce5ab62095ae9a91d8d56441e9bb8c54.1644251611.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v2.git.git.1644251611.gitgitgadget@gmail.com","subject":"[PATCH v2 1/2] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-07T16:33:30Z","receivedAt":"2022-02-07T16:48:48Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nWhen introducing a new flag --batch-command, we will add a flag on the\nbatch_options struct that indicates whether or not an interactive\ncommand mode will be used that reads commands and arguments off of\nstdin.\n\nAn intuitive name for this flag would be \"command\", which can get\nconfusing with the already existing cmdmode.\n\ncmdmode refers to how the result output of the blob will be transformed,\neither according to --filter or --textconv. So transform_mode is a more\ndescriptive name for the flag, and will not get confused with the new\ncommand flag to be added in the next commit.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"447893","messageId":"1b63164ad4d9ec6b5fa6f733b6095b2779298b36.1644251611.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v2.git.git.1644251611.gitgitgadget@gmail.com","subject":"[PATCH v2 2/2] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-07T16:33:31Z","receivedAt":"2022-02-07T16:48:48Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings since on a given server.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] NL\n [arg1] [arg2] NL\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush NL\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents NL\ninfo NL\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes a argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ncontents NL\nobject NL\nobject NL\ncontents NL\nflush\ncontents NL\nflush\n\nWhen used without --buffer:\n\ncontents NL\nobject NL\nobject NL\ncontents NL\ncontents NL\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 19 ++++\n builtin/cat-file.c | 124 +++++++++++++++++++++\n t/t1006-cat-file.sh | 197 ++++++++++++++++++++++++++++++++-\n 3 files changed, 339 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..618dbd15338 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,25 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+\tEnter a command mode that reads commands and arguments from stdin.\n+\tMay not be combined with any other options or arguments except\n+\t`--textconv` or `--filters`, in which case the input lines also need to\n+\tspecify the path, separated by whitespace. See the section\n+\t`BATCH OUTPUT` below for details.\n+\n+contents ::\n+\tPrint object contents for object reference \n+\n+info ::\n+\tPrint object info for object reference \n+\n+flush::\n+\tExecute all preceding commands that were issued since the beginning or\n+\tsince the last flush command was issued. Only used with --buffer. When\n+\t--buffer is not used, commands are flushed each time without issuing\n+\t`flush`.\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..6bfab74b58a 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -26,6 +26,7 @@ struct batch_options {\n \tint unordered;\n \tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n+\tint command;\n };\n \n static const char *force_path;\n@@ -508,6 +509,118 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tconst char *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->print_contents = 1;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->print_contents = 0;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void flush_batch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmds,\n+\t\tint queued)\n+{\n+\tint i;\n+\tfor(i = 0; i < queued; i++){\n+\t\tcmds[i].fn(opt, cmds[i].line, output, data);\n+\t}\n+\tfflush(stdout);\n+}\n+\n+static const struct parse_cmd {\n+\tconst char *prefix;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *cmds = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\tint queued = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n+\t\t\tif (!opt->buffer_output)\n+\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n+\t\t\tif (*cmd_end)\n+\t\t\t\tdie(_(\"flush takes no arguments\"));\n+\t\t\tif (!queued)\n+\t\t\t\tdie(_(\"nothing to flush\"));\n+\t\t\tflush_batch_calls(opt, output, data, cmds, queued);\n+\t\t\tqueued = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args)\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tqueued++;\n+\t\tif (queued > nr) {\n+\t\t\tALLOC_GROW(cmds, nr+1, alloc);\n+\t\t\tnr++;\n+\t\t}\n+\n+\t\tcall.fn = cmd->fn;\n+\t\tcall.line = xstrdup(p);\n+\t\tcmds[queued-1] = call;\n+\t}\n+\tfree(cmds);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -515,6 +628,7 @@ static int batch_objects(struct batch_options *opt)\n \tstruct expand_data data;\n \tint save_warning;\n \tint retval = 0;\n+\tconst int command = opt->command;\n \n \tif (!opt->format)\n \t\topt->format = \"%(objectname) %(objecttype) %(objectsize)\";\n@@ -590,6 +704,10 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (command) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -608,6 +726,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -636,6 +755,7 @@ static int batch_option_callback(const struct option *opt,\n \n \tbo->enabled = 1;\n \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\tbo->command = !strcmp(opt->long_name, \"batch-command\");\n \tbo->format = arg;\n \n \treturn 0;\n@@ -683,6 +803,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..c57a35ea20a 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -177,6 +177,20 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ test -z \"$content\" ||\n+ test_expect_success \"--batch-command output of $type content is correct\" '\n+\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n+\t| git cat-file --batch-command)\" $no_ts >actual &&\n+\ttest_cmp expect actual\n+ '\n+\n+ test_expect_success \"--batch-command output of $type info is correct\" '\n+\techo \"$sha1 $type $size\" >expect &&\n+\ttest_write_lines \"info $sha1\" | git cat-file --batch-command >actual &&\n+\ttest_cmp expect actual\n+ '\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -213,6 +227,64 @@ $content\"\n '\n }\n \n+run_buffer_test_flush () {\n+\ttype=$1\n+\tsha1=$2\n+\tsize=$3\n+\n+\tmkfifo input &&\n+\ttest_when_finished 'rm input; exec 8<&-' &&\n+\tmkfifo output &&\n+\texec 9<>output &&\n+\ttest_when_finished 'rm output; exec 9<&-'\n+\t(\n+\t\tgit cat-file --buffer --batch-command err &\n+\t\techo $! &&\n+\t\twait $!\n+\t) >&9 &\n+\tsh_pid=$! &&\n+\tread cat_file_pid <&9 &&\n+\ttest_when_finished \"kill $cat_file_pid\n+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n+\techo \"$sha1 $type $size\" >expect &&\n+\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n+\t# TODO - consume all available input, not just one\n+\t# line (see above).\n+\t# check output is flushed on exit\n+\tread actual <&9 &&\n+\techo \"$actual\" >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_be_empty err\n+}\n+\n+run_buffer_test_no_flush () {\n+\ttype=$1\n+\tsha1=$2\n+\tsize=$3\n+\n+\ttouch output &&\n+\ttest_when_finished 'rm output' &&\n+\tmkfifo input &&\n+\ttest_when_finished 'rm input' &&\n+\tmkfifo pid &&\n+\texec 9<>pid &&\n+\ttest_when_finished 'rm pid; exec 9<&-'\n+\t(\n+\t\tgit cat-file --buffer --batch-command output &\n+\t\techo $! &&\n+\t\twait $!\n+\t\techo $?\n+\t) >&9 &\n+\tsh_pid=$! &&\n+\tread cat_file_pid <&9 &&\n+\ttest_when_finished \"kill $cat_file_pid\n+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n+\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n+\tkill $cat_file_pid &&\n+\tread status <&9 &&\n+\ttest_must_be_empty output\n+}\n+\n hello_content=\"Hello World\"\n hello_size=$(strlen \"$hello_content\")\n hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n@@ -224,6 +296,14 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n+ run_buffer_test_flush blob $hello_sha1 $hello_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n+ run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -238,6 +318,14 @@ tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n \n run_tests 'tree' $tree_sha1 $tree_size \"\" \"$tree_pretty_content\"\n \n+test_expect_success PIPE '--batch-command --buffer with flush for tree info' '\n+ run_buffer_test_flush tree $tree_sha1 $tree_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for tree info' '\n+ run_buffer_test_no_flush tree $tree_sha1 $tree_size false\n+'\n+\n commit_message=\"Initial commit\"\n commit_sha1=$(echo_without_newline \"$commit_message\" | git commit-tree $tree_sha1)\n commit_size=$(($(test_oid hexsz) + 137))\n@@ -249,6 +337,14 @@ $commit_message\"\n \n run_tests 'commit' $commit_sha1 $commit_size \"$commit_content\" \"$commit_content\" 1\n \n+test_expect_success PIPE '--batch-command --buffer with flush for commit info' '\n+ run_buffer_test_flush commit $commit_sha1 $commit_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for commit info' '\n+ run_buffer_test_no_flush commit $commit_sha1 $commit_size false\n+'\n+\n tag_header_without_timestamp=\"object $hello_sha1\n type blob\n tag hellotag\n@@ -263,11 +359,19 @@ tag_size=$(strlen \"$tag_content\")\n \n run_tests 'tag' $tag_sha1 $tag_size \"$tag_content\" \"$tag_content\" 1\n \n+test_expect_success PIPE '--batch-command --buffer with flush for tag info' '\n+ run_buffer_test_flush tag $tag_sha1 $tag_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for tag info' '\n+ run_buffer_test_no_flush tag $tag_sha1 $tag_size false\n+'\n+\n test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -373,6 +477,62 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+batch_command_info_input=\"info $hello_sha1\n+info $tree_sha1\n+info $commit_sha1\n+info $tag_sha1\n+info deadbeef\n+info \n+flush\n+\"\n+\n+test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n+\ttest \"$batch_check_output\" = \"$(echo_without_newline \\\n+\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n+'\n+\n+batch_command_contents_input=\"contents $hello_sha1\n+contents $commit_sha1\n+contents $tag_sha1\n+contents deadbeef\n+contents \n+flush\n+\"\n+\n+test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n+\ttest \"$(maybe_remove_timestamp \"$batch_output\" 1)\" = \\\n+\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n+'\n+\n+batch_command_mixed_input=\"info $hello_sha1\n+contents $hello_sha1\n+info $commit_sha1\n+contents $commit_sha1\n+info $tag_sha1\n+contents $tag_sha1\n+contents deadbeef\n+info \n+flush\n+\"\n+\n+batch_command_mixed_output=\"$hello_sha1 blob $hello_size\n+$hello_sha1 blob $hello_size\n+$hello_content\n+$commit_sha1 commit $commit_size\n+$commit_sha1 commit $commit_size\n+$commit_content\n+$tag_sha1 tag $tag_size\n+$tag_sha1 tag $tag_size\n+$tag_content\n+deadbeef missing\n+ missing\"\n+\n+test_expect_success \"--batch-command with mixed calls gives correct format\" '\n+\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n+\t\"$(maybe_remove_timestamp \"$(echo_without_newline \\\n+\t\"$batch_command_mixed_input\" | git cat-file --batch-command --buffer)\" 1)\"\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -963,5 +1123,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep -E \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n+\n+test_expect_success 'batch-command flush empty queue' '\n+\techo flush >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep -E \"^fatal:.*nothing to flush.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"447922","messageId":"20220207233422.14094-1-jonathantanmy@google.com","threadId":"57365","inReplyTo":"1b63164ad4d9ec6b5fa6f733b6095b2779298b36.1644251611.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/2] cat-file: add --batch-command mode","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-07T23:34:22Z","receivedAt":"2022-02-08T01:06:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"John Cai via GitGitGadget\" writes:\n> However, if we had --batch-command, we wouldn't need to keep both\n> processes around, and instead just have one --batch-command process\n> where we can flip between getting object info, and getting object\n> contents. Since we have a pair of cat-file processes per repository,\n> this means we can get rid of roughly half of long lived git cat-file\n> processes. Given there are many repositories being accessed at any given\n> time, this can lead to huge savings since on a given server.\n\nOne other benefit is that with explicit flushes, in a partial clone,\nthis makes it possible to batch prefetch objects.\n\n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> index bef76f4dd06..618dbd15338 100644\n> --- a/Documentation/git-cat-file.txt\n> +++ b/Documentation/git-cat-file.txt\n> @@ -96,6 +96,25 @@ OPTIONS\n> \tneed to specify the path, separated by whitespace. See the\n> \tsection `BATCH OUTPUT` below for details.\n> \n> +--batch-command::\n> +\tEnter a command mode that reads commands and arguments from stdin.\n> +\tMay not be combined with any other options or arguments except\n> +\t`--textconv` or `--filters`, in which case the input lines also need to\n> +\tspecify the path, separated by whitespace. See the section\n> +\t`BATCH OUTPUT` below for details.\n> +\n> +contents ::\n> +\tPrint object contents for object reference \n> +\n> +info ::\n> +\tPrint object info for object reference \n> +\n> +flush::\n> +\tExecute all preceding commands that were issued since the beginning or\n> +\tsince the last flush command was issued. Only used with --buffer. When\n> +\t--buffer is not used, commands are flushed each time without issuing\n> +\t`flush`.\n\nThe way this is formatted leads me to think that \"contents\", etc. are\nCLI arguments, not things written to stdin. Some of the commit message\nprobably needs to go here.\n\nI just looked at the commit message and documentation for now.\n\nIf you have time and are interested, we at Google are thinking of a more\ncomprehensive \"batch\" process [1].\n\n[1] https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n"},{"id":"447928","messageId":"xmqq35kufahb.fsf@gitster.g","threadId":"57365","inReplyTo":"1b63164ad4d9ec6b5fa6f733b6095b2779298b36.1644251611.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/2] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-08T00:49:52Z","receivedAt":"2022-02-08T01:06:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> From: John Cai \n>\n> Add a new flag --batch-command that accepts commands and arguments\n> from stdin, similar to git-update-ref --stdin.\n>\n> At GitLab, we use a pair of long running cat-file processes when\n> accessing object content. One for iterating over object metadata with\n> --batch-check, and the other to grab object contents with --batch.\n>\n> However, if we had --batch-command, we wouldn't need to keep both\n> processes around, and instead just have one --batch-command process\n> where we can flip between getting object info, and getting object\n> contents. Since we have a pair of cat-file processes per repository,\n> this means we can get rid of roughly half of long lived git cat-file\n> processes. Given there are many repositories being accessed at any given\n> time, this can lead to huge savings since on a given server.\n>\n> git cat-file --batch-command\n>\n> will enter an interactive command mode whereby the user can enter in\n> commands and their arguments that get queued in memory:\n>\n> [arg1] [arg2] NL\n> [arg1] [arg2] NL\n\nIf you mean you take one command with its args per line, use LF not\nNL.\n\n $ git grep '\\' Documentation\n $ git grep '\\' Documentation\n\nWe may want to fix the sole offender in Documentation/config.txt\nfile (#leftoverbits).\n\n> With this mechanism of queueing up commands and letting (A) issue a\n> flush command, process (A) can control when the buffer is flushed and\n> can guarantee it will receive all of the output when in --buffer mode.\n\nAre we giving them guarantee when output will *not* come? If (B) is\nallowed to flush when it thinks it has too much in-core, it would\nmean that (A) cannot keep issuing commands forever without reading\nthe response from (B), because (B) will eventually be blocked when\nit tries to flush to a pipe that (A) is not reading. I think there\nshould be some discussion on that, too. IOW, --batch-command does\nnot allow (B) to flush until it gets \"flush\", or something like that.\n\n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> index bef76f4dd06..618dbd15338 100644\n> --- a/Documentation/git-cat-file.txt\n> +++ b/Documentation/git-cat-file.txt\n> @@ -96,6 +96,25 @@ OPTIONS\n> \tneed to specify the path, separated by whitespace. See the\n> \tsection `BATCH OUTPUT` below for details.\n> \n> +--batch-command::\n> +\tEnter a command mode that reads commands and arguments from stdin.\n> +\tMay not be combined with any other options or arguments except\n> +\t`--textconv` or `--filters`, in which case the input lines also need to\n> +\tspecify the path, separated by whitespace. See the section\n> +\t`BATCH OUTPUT` below for details.\n> +\n> +contents ::\n> +\tPrint object contents for object reference \n\nPresumably this corresponds to what you get out of \"--batch\"?\n\n> +info ::\n> +\tPrint object info for object reference \n\nand this one \"--batch-check\"?\n\nI expect that future readers will ask this same question because it\nis not clear how \"object contents\" and \"object info\" are exactly\nprinted. These two paragraphs may want to anticipate it and reduce\nthe need for readers to ask such a question. \n\n> +flush::\n> +\tExecute all preceding commands that were issued since the beginning or\n> +\tsince the last flush command was issued. Only used with --buffer. When\n> +\t--buffer is not used, commands are flushed each time without issuing\n> +\t`flush`.\n\nHere is a good place to also say \"When --buffer is used, no output\nwill come until this is issued\" or something.\n\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 5f015e71096..6bfab74b58a 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -26,6 +26,7 @@ struct batch_options {\n> \tint unordered;\n> \tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n> \tconst char *format;\n> +\tint command;\n\nI am not sure if \"command\" is a good name. Does it answer this\nquestion clearly? \"'command' as opposed to what?\"\n\n> @@ -508,6 +509,118 @@ static int batch_unordered_packed(const struct object_id *oid,\n> \t\t\t\t data);\n> }\n> \n> +typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n> +\t\t\t struct strbuf *, struct expand_data *);\n> +\n> +struct queued_cmd {\n> +\tparse_cmd_fn_t fn;\n> +\tconst char *line;\n> +};\n> +\n> +static void parse_cmd_contents(struct batch_options *opt,\n> +\t\t\t const char *line,\n> +\t\t\t struct strbuf *output,\n> +\t\t\t struct expand_data *data)\n> +{\n> +\topt->print_contents = 1;\n> +\tbatch_one_object(line, output, opt, data);\n> +}\n> +\n> +static void parse_cmd_info(struct batch_options *opt,\n> +\t\t\t const char *line,\n> +\t\t\t struct strbuf *output,\n> +\t\t\t struct expand_data *data)\n> +{\n> +\topt->print_contents = 0;\n> +\tbatch_one_object(line, output, opt, data);\n> +}\n\nOK, these are as simple as expected.\n\n> +static void flush_batch_calls(struct batch_options *opt,\n> +\t\tstruct strbuf *output,\n> +\t\tstruct expand_data *data,\n> +\t\tstruct queued_cmd *cmds,\n> +\t\tint queued)\n> +{\n> +\tint i;\n> +\tfor(i = 0; i < queued; i++){\n\nMissing SP around parentheses.\n\nExcess brace pair wround a single-statement block.\n\n> +\t\tcmds[i].fn(opt, cmds[i].line, output, data);\n> +\t}\n> +\tfflush(stdout);\n> +}\n> +\n> +static const struct parse_cmd {\n> +\tconst char *prefix;\n> +\tparse_cmd_fn_t fn;\n> +\tunsigned takes_args;\n> +} commands[] = {\n> +\t{ \"contents\", parse_cmd_contents, 1},\n> +\t{ \"info\", parse_cmd_info, 1},\n> +};\n> +\n> +static void batch_objects_command(struct batch_options *opt,\n> +\t\t\t\t struct strbuf *output,\n> +\t\t\t\t struct expand_data *data)\n> +{\n> +\tstruct strbuf input = STRBUF_INIT;\n> +\tstruct queued_cmd *cmds = NULL;\n> +\tsize_t alloc = 0, nr = 0;\n> +\tint queued = 0;\n> +\n> +\twhile (!strbuf_getline(&input, stdin)) {\n> +\t\tint i;\n> +\t\tconst struct parse_cmd *cmd = NULL;\n> +\t\tconst char *p, *cmd_end;\n> +\t\tstruct queued_cmd call = {0};\n> +\n> +\t\tif (!input.len)\n> +\t\t\tdie(_(\"empty command in input\"));\n> +\t\tif (isspace(*input.buf))\n> +\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n> +\n> +\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n> +\t\t\tif (!opt->buffer_output)\n> +\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n> +\t\t\tif (*cmd_end)\n> +\t\t\t\tdie(_(\"flush takes no arguments\"));\n> +\t\t\tif (!queued)\n> +\t\t\t\tdie(_(\"nothing to flush\"));\n\nI am not sure if this is a good idea at all. What do we gain from\npunishing an automated stupid loop that issues flush every once in a\nwhile after issuing a handful real commands and issues another flush\nafter running out of the real commands for a good measure?\n\n> +\t\t\tflush_batch_calls(opt, output, data, cmds, queued);\n> +\t\t\tqueued = 0;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n> +\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\tcmd = &commands[i];\n> +\t\t\tif (cmd->takes_args)\n> +\t\t\t\tp = cmd_end + 1;\n\nIf somebody adds an entry with .takes_args==false, p will stay\nuninitialized and used in the call ->fn() below, or passed to\nxstrdup(p). It probabloy should be initialized to NULL, and\nxstrdup(p) below replaced with xstrdup_or_null(p).\n\n> +\t\t\tbreak;\n> +\t\t}\n> +\n> +\t\tif (!cmd)\n> +\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n> +\n> +\t\tif (!opt->buffer_output) {\n> +\t\t\tcmd->fn(opt, p, output, data);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n\n\n> +\t\tqueued++;\n> +\t\tif (queued > nr) {\n> +\t\t\tALLOC_GROW(cmds, nr+1, alloc);\n> +\t\t\tnr++;\n> +\t\t}\n> +\n> +\t\tcall.fn = cmd->fn;\n> +\t\tcall.line = xstrdup(p);\n> +\t\tcmds[queued-1] = call;\n\nCan nr and queued ever go out of sync?\n\nIf cmds is the usual tuple we let ALLOC_GROW() to\nmanage, alloc keeps track of how physically large the array is, and\nnr indicates how many slots are filled. Holding onto the block of\nmemory we used when discarding the accumulated items and reusing\nthat block without having to reallocate until we use the slots that\nwe have allocated is done by using only pair.\n\nBut the above code seems that it does not understand that, and\ninstead thinks it has to use \"nr\" for the \"we have made the array\nthis big, so we do not have to realloc up to that point\" pointer,\nhence its use of a separate \"queued\". IOW, the array growing code\nabove seems confused.\n\nIt is more customery (hence easier to follow by readers who work on\nour code base) to lose queued and say\n\n\t\tALLOC_GROW(cmds, nr + 1, alloc);\n\t\tcmds[nr++] = call;\n\t\tcall.fn = cmd->fn;\n\t\tcall.line = xstrdup_or_null(p);\n\ninstead of the above 9 lines.\n\n> @@ -515,6 +628,7 @@ static int batch_objects(struct batch_options *opt)\n> \tstruct expand_data data;\n> \tint save_warning;\n> \tint retval = 0;\n> +\tconst int command = opt->command;\n\nThis tells me that it is quite a misnomer. This single bit is used\nto differentiate between \"other batch modes\" and \"--batch-command\"\nmode, which already smells like a misdesign, because we have, from\nan end-user's point of view, three modes:\n\n --batch\n --batch-check\n --batch-command\n\nso it would be far cleaner to have a single batch_mode enum that can\nrepresent these three \"batch modes\", no?\n\n> \tif (!opt->format)\n> \t\topt->format = \"%(objectname) %(objecttype) %(objectsize)\";\n> @@ -590,6 +704,10 @@ static int batch_objects(struct batch_options *opt)\n> \tsave_warning = warn_on_object_refname_ambiguity;\n> \twarn_on_object_refname_ambiguity = 0;\n> \n> +\tif (command) {\n> +\t\tbatch_objects_command(opt, &output, &data);\n> +\t\tgoto cleanup;\n> +\t}\n> \twhile (strbuf_getline(&input, stdin) != EOF) {\n> \t\tif (data.split_on_whitespace) {\n> \t\t\t/*\n> @@ -608,6 +726,7 @@ static int batch_objects(struct batch_options *opt)\n> \t\tbatch_one_object(input.buf, &output, opt, &data);\n> \t}\n> \n> + cleanup:\n> \tstrbuf_release(&input);\n> \tstrbuf_release(&output);\n> \twarn_on_object_refname_ambiguity = save_warning;\n> @@ -636,6 +755,7 @@ static int batch_option_callback(const struct option *opt,\n> \n> \tbo->enabled = 1;\n> \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n> +\tbo->command = !strcmp(opt->long_name, \"batch-command\");\n\nAnd this part needs fixing. The original used to say\n\n\twe supoprt \"batch\" and something else (it turns out that\n\t\"batch-check\" is that something else, but the above code is\n\tso sloppy that it does not even tell readers that), and\n\tthe .print_contents member is how you can tell them apart.\n\nThis patch is making it worse by introducing another member that can\nbe independently set or unset, pretending that the four combinations\nall can make sense, but that is not the case, right?\n\nSo, perhaps a good first step would be to lose .print_contents\nmember and add .batch_command member that is used to more explicitly\nname one of the three possibilities?\n"},{"id":"447930","messageId":"xmqqo83ifcv0.fsf@gitster.g","threadId":"57365","inReplyTo":"2d9a0b06ce5ab62095ae9a91d8d56441e9bb8c54.1644251611.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/2] cat-file: rename cmdmode to transform_mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-07T23:58:27Z","receivedAt":"2022-02-08T01:06:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> From: John Cai \n>\n> When introducing a new flag --batch-command, we will add a flag on the\n> batch_options struct that indicates whether or not an interactive\n> command mode will be used that reads commands and arguments off of\n> stdin.\n>\n> An intuitive name for this flag would be \"command\", which can get\n> confusing with the already existing cmdmode.\n\nIs \"command\" truly an intuitive name for \"were we given '--batch-command'\noption?\" bit? I dunno.\n\nBut I agree that \"transform_mode\" is much better name than \"cmdmode\"\nfor what this member is about.\n"},{"id":"447967","messageId":"20a951f4-ac68-639e-0a4c-24f8f7668d47@gmail.com","threadId":"57365","inReplyTo":"1b63164ad4d9ec6b5fa6f733b6095b2779298b36.1644251611.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/2] cat-file: add --batch-command mode","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-08T11:06:27Z","receivedAt":"2022-02-08T11:32:20Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John\n\nOn 07/02/2022 16:33, John Cai via GitGitGadget wrote:\n> From: John Cai \n> \n> Add a new flag --batch-command that accepts commands and arguments\n> from stdin, similar to git-update-ref --stdin.\n> \n> At GitLab, we use a pair of long running cat-file processes when\n> accessing object content. One for iterating over object metadata with\n> --batch-check, and the other to grab object contents with --batch.\n> \n> However, if we had --batch-command, we wouldn't need to keep both\n> processes around, and instead just have one --batch-command process\n> where we can flip between getting object info, and getting object\n> contents. Since we have a pair of cat-file processes per repository,\n> this means we can get rid of roughly half of long lived git cat-file\n> processes. Given there are many repositories being accessed at any given\n> time, this can lead to huge savings since on a given server.\n> \n> git cat-file --batch-command\n> \n> will enter an interactive command mode whereby the user can enter in\n> commands and their arguments that get queued in memory:\n> \n> [arg1] [arg2] NL\n> [arg1] [arg2] NL\n> \n> When --buffer mode is used, commands will be queued in memory until a\n> flush command is issued that execute them:\n> \n> flush NL\n> \n> The reason for a flush command is that when a consumer process (A)\n> talks to a git cat-file process (B) and interactively writes to and\n> reads from it in --buffer mode, (A) needs to be able to control when\n> the buffer is flushed to stdout.\n> \n> Currently, from (A)'s perspective, the only way is to either\n> \n> 1. kill (B)'s process\n> 2. send an invalid object to stdin.\n> \n> 1. is not ideal from a performance perspective as it will require\n> spawning a new cat-file process each time, and 2. is hacky and not a\n> good long term solution.\n> \n> With this mechanism of queueing up commands and letting (A) issue a\n> flush command, process (A) can control when the buffer is flushed and\n> can guarantee it will receive all of the output when in --buffer mode.\n> \n> This patch adds the basic structure for adding command which can be\n> extended in the future to add more commands. It also adds the following\n> two commands (on top of the flush command):\n> \n> contents NL\n> info NL\n> \n> The contents command takes an argument and prints out the object\n> contents.\n> \n> The info command takes a argument and prints out the object\n> metadata.\n> \n> These can be used in the following way with --buffer:\n> \n> contents NL\n> object NL\n\nThere is no object command\n\n >[...]\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 145eee11df9..c57a35ea20a 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -177,6 +177,20 @@ $content\"\n> \ttest_cmp expect actual\n> '\n> \n> + test -z \"$content\" ||\n> + test_expect_success \"--batch-command output of $type content is correct\" '\n> +\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n> +\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n> +\t| git cat-file --batch-command)\" $no_ts >actual &&\n> +\ttest_cmp expect actual\n> + '\n> +\n> + test_expect_success \"--batch-command output of $type info is correct\" '\n> +\techo \"$sha1 $type $size\" >expect &&\n> +\ttest_write_lines \"info $sha1\" | git cat-file --batch-command >actual &&\n> +\ttest_cmp expect actual\n> + '\n> +\n> test_expect_success \"custom --batch-check format\" '\n> \techo \"$type $sha1\" >expect &&\n> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n> @@ -213,6 +227,64 @@ $content\"\n> '\n> }\n> \n> +run_buffer_test_flush () {\n> +\ttype=$1\n> +\tsha1=$2\n> +\tsize=$3\n> +\n> +\tmkfifo input &&\n> +\ttest_when_finished 'rm input; exec 8<&-' &&\n> +\tmkfifo output &&\n> +\texec 9<>output &&\n> +\ttest_when_finished 'rm output; exec 9<&-'\n> +\t(\n> +\t\tgit cat-file --buffer --batch-command err &\n> +\t\techo $! &&\n> +\t\twait $!\n> +\t) >&9 &\n> +\tsh_pid=$! &&\n> +\tread cat_file_pid <&9 &&\n> +\ttest_when_finished \"kill $cat_file_pid\n> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n> +\techo \"$sha1 $type $size\" >expect &&\n> +\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n> +\t# TODO - consume all available input, not just one\n> +\t# line (see above).\n> +\t# check output is flushed on exit\n\nThis test seems to have lost some code above this comment so the comment \nis no longer correct - we do not test if the output is flushed on exit \nand looking at the implementation I don't think it is.\n\nBest Wishes\n\nPhillip\n\n> +\tread actual <&9 &&\n> +\techo \"$actual\" >actual &&\n> +\ttest_cmp expect actual &&\n> +\ttest_must_be_empty err\n> +}\n> +\n> +run_buffer_test_no_flush () {\n> +\ttype=$1\n> +\tsha1=$2\n> +\tsize=$3\n> +\n> +\ttouch output &&\n> +\ttest_when_finished 'rm output' &&\n> +\tmkfifo input &&\n> +\ttest_when_finished 'rm input' &&\n> +\tmkfifo pid &&\n> +\texec 9<>pid &&\n> +\ttest_when_finished 'rm pid; exec 9<&-'\n> +\t(\n> +\t\tgit cat-file --buffer --batch-command output &\n> +\t\techo $! &&\n> +\t\twait $!\n> +\t\techo $?\n> +\t) >&9 &\n> +\tsh_pid=$! &&\n> +\tread cat_file_pid <&9 &&\n> +\ttest_when_finished \"kill $cat_file_pid\n> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n> +\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n> +\tkill $cat_file_pid &&\n> +\tread status <&9 &&\n> +\ttest_must_be_empty output\n> +}\n> +\n> hello_content=\"Hello World\"\n> hello_size=$(strlen \"$hello_content\")\n> hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n> @@ -224,6 +296,14 @@ test_expect_success \"setup\" '\n> \n> run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n> + run_buffer_test_flush blob $hello_sha1 $hello_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n> + run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n> +'\n> +\n> test_expect_success '--batch-check without %(rest) considers whole line' '\n> \techo \"$hello_sha1 blob $hello_size\" >expect &&\n> \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n> @@ -238,6 +318,14 @@ tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n> \n> run_tests 'tree' $tree_sha1 $tree_size \"\" \"$tree_pretty_content\"\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for tree info' '\n> + run_buffer_test_flush tree $tree_sha1 $tree_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for tree info' '\n> + run_buffer_test_no_flush tree $tree_sha1 $tree_size false\n> +'\n> +\n> commit_message=\"Initial commit\"\n> commit_sha1=$(echo_without_newline \"$commit_message\" | git commit-tree $tree_sha1)\n> commit_size=$(($(test_oid hexsz) + 137))\n> @@ -249,6 +337,14 @@ $commit_message\"\n> \n> run_tests 'commit' $commit_sha1 $commit_size \"$commit_content\" \"$commit_content\" 1\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for commit info' '\n> + run_buffer_test_flush commit $commit_sha1 $commit_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for commit info' '\n> + run_buffer_test_no_flush commit $commit_sha1 $commit_size false\n> +'\n> +\n> tag_header_without_timestamp=\"object $hello_sha1\n> type blob\n> tag hellotag\n> @@ -263,11 +359,19 @@ tag_size=$(strlen \"$tag_content\")\n> \n> run_tests 'tag' $tag_sha1 $tag_size \"$tag_content\" \"$tag_content\" 1\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for tag info' '\n> + run_buffer_test_flush tag $tag_sha1 $tag_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for tag info' '\n> + run_buffer_test_no_flush tag $tag_sha1 $tag_size false\n> +'\n> +\n> test_expect_success \\\n> \"Reach a blob from a tag pointing to it\" \\\n> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n> \n> -for batch in batch batch-check\n> +for batch in batch batch-check batch-command\n> do\n> for opt in t s e p\n> do\n> @@ -373,6 +477,62 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n> '\n> \n> +batch_command_info_input=\"info $hello_sha1\n> +info $tree_sha1\n> +info $commit_sha1\n> +info $tag_sha1\n> +info deadbeef\n> +info\n> +flush\n> +\"\n> +\n> +test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n> +\ttest \"$batch_check_output\" = \"$(echo_without_newline \\\n> +\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n> +'\n> +\n> +batch_command_contents_input=\"contents $hello_sha1\n> +contents $commit_sha1\n> +contents $tag_sha1\n> +contents deadbeef\n> +contents\n> +flush\n> +\"\n> +\n> +test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n> +\ttest \"$(maybe_remove_timestamp \"$batch_output\" 1)\" = \\\n> +\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n> +'\n> +\n> +batch_command_mixed_input=\"info $hello_sha1\n> +contents $hello_sha1\n> +info $commit_sha1\n> +contents $commit_sha1\n> +info $tag_sha1\n> +contents $tag_sha1\n> +contents deadbeef\n> +info\n> +flush\n> +\"\n> +\n> +batch_command_mixed_output=\"$hello_sha1 blob $hello_size\n> +$hello_sha1 blob $hello_size\n> +$hello_content\n> +$commit_sha1 commit $commit_size\n> +$commit_sha1 commit $commit_size\n> +$commit_content\n> +$tag_sha1 tag $tag_size\n> +$tag_sha1 tag $tag_size\n> +$tag_content\n> +deadbeef missing\n> + missing\"\n> +\n> +test_expect_success \"--batch-command with mixed calls gives correct format\" '\n> +\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n> +\t\"$(maybe_remove_timestamp \"$(echo_without_newline \\\n> +\t\"$batch_command_mixed_input\" | git cat-file --batch-command --buffer)\" 1)\"\n> +'\n> +\n> test_expect_success 'setup blobs which are likely to delta' '\n> \ttest-tool genrandom foo 10240 >foo &&\n> \t{ cat foo && echo plus; } >foo-plus &&\n> @@ -963,5 +1123,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n> \techo \"$orig commit $orig_size\" >expect &&\n> \ttest_cmp expect actual\n> '\n> +test_expect_success 'batch-command empty command' '\n> +\techo \"\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep -E \"^fatal:.*empty command in input.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command whitespace before command' '\n> +\techo \" info deadbeef\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep -E \"^fatal:.*whitespace before command.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command unknown command' '\n> +\techo unknown_command >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep -E \"^fatal:.*unknown command.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush with arguments' '\n> +\techo \"flush arg\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n> +\tgrep -E \"^fatal:.*flush takes no arguments.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush without --buffer' '\n> +\techo \"flush arg\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush empty queue' '\n> +\techo flush >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n> +\tgrep -E \"^fatal:.*nothing to flush.*\" err\n> +'\n> \n> test_done\n\n"},{"id":"447971","messageId":"83ab0f7d-96d5-8730-edd7-693f10b09002@gmail.com","threadId":"57365","inReplyTo":"20220207233422.14094-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 2/2] cat-file: add --batch-command mode","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-08T11:00:21Z","receivedAt":"2022-02-08T11:32:44Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Jonathan and John\n\nOn 07/02/2022 23:34, Jonathan Tan wrote:\n> \"John Cai via GitGitGadget\" writes:\n>> However, if we had --batch-command, we wouldn't need to keep both\n>> processes around, and instead just have one --batch-command process\n>> where we can flip between getting object info, and getting object\n>> contents. Since we have a pair of cat-file processes per repository,\n>> this means we can get rid of roughly half of long lived git cat-file\n>> processes. Given there are many repositories being accessed at any given\n>> time, this can lead to huge savings since on a given server.\n> \n> One other benefit is that with explicit flushes, in a partial clone,\n> this makes it possible to batch prefetch objects.\n\nJonathan is there any overlap between what this series is trying to do \nand your proposal for a batch command[1]? For example would extending \nthis series to get blob sizes be useful to you?\n\nBest Wishes\n\nPhillip\n\n[1] \nhttps://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n\n>> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n>> index bef76f4dd06..618dbd15338 100644\n>> --- a/Documentation/git-cat-file.txt\n>> +++ b/Documentation/git-cat-file.txt\n>> @@ -96,6 +96,25 @@ OPTIONS\n>> \tneed to specify the path, separated by whitespace. See the\n>> \tsection `BATCH OUTPUT` below for details.\n>> \n>> +--batch-command::\n>> +\tEnter a command mode that reads commands and arguments from stdin.\n>> +\tMay not be combined with any other options or arguments except\n>> +\t`--textconv` or `--filters`, in which case the input lines also need to\n>> +\tspecify the path, separated by whitespace. See the section\n>> +\t`BATCH OUTPUT` below for details.\n>> +\n>> +contents ::\n>> +\tPrint object contents for object reference \n>> +\n>> +info ::\n>> +\tPrint object info for object reference \n>> +\n>> +flush::\n>> +\tExecute all preceding commands that were issued since the beginning or\n>> +\tsince the last flush command was issued. Only used with --buffer. When\n>> +\t--buffer is not used, commands are flushed each time without issuing\n>> +\t`flush`.\n> \n> The way this is formatted leads me to think that \"contents\", etc. are\n> CLI arguments, not things written to stdin. Some of the commit message\n> probably needs to go here.\n> \n> I just looked at the commit message and documentation for now.\n> \n> If you have time and are interested, we at Google are thinking of a more\n> comprehensive \"batch\" process [1].\n> \n> [1] https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n\n"},{"id":"447993","messageId":"20220208175651.155629-1-jonathantanmy@google.com","threadId":"57365","inReplyTo":"83ab0f7d-96d5-8730-edd7-693f10b09002@gmail.com","subject":"Re: [PATCH v2 2/2] cat-file: add --batch-command mode","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-08T17:56:51Z","receivedAt":"2022-02-08T17:57:56Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Phillip Wood writes:\n> Jonathan is there any overlap between what this series is trying to do \n> and your proposal for a batch command[1]? For example would extending \n> this series to get blob sizes be useful to you?\n> \n> Best Wishes\n> \n> Phillip\n> \n> [1] \n> https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n\nThere is overlap, yes. I'm not sure of the best way to resolve it,\nthough. John mentions a substantial reduction (\"roughly half\") of Git\nprocesses [1], and if they foresee needing to access things other than\nobject info and contents, it might be better to start with something\nmore extensible, like my proposal for a specific batch command. (If not,\nthey will encounter another increase in the number of processes.) If\nthey think that they can make do with this patch for the time being, I\nthink that's fine too: once this is merged (which will be earlier than\nany extensible batch command, for sure), they (and anyone else who needs\nbatched object info and contents without the overhead of initializing\nall the data structures in Git) can make use of this improvement.\n\nAs for getting blob sizes, I think that --batch-check can already give\nit to us. If that is the case, the series is fine as-is (at least in\nthat aspect).\n\n[1] https://lore.kernel.org/git/1b63164ad4d9ec6b5fa6f733b6095b2779298b36.1644251611.git.gitgitgadget@gmail.com/\n"},{"id":"447994","messageId":"xmqq5yppdyc0.fsf@gitster.g","threadId":"57365","inReplyTo":"20220208175651.155629-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 2/2] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-08T18:09:51Z","receivedAt":"2022-02-08T18:09:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> There is overlap, yes. I'm not sure of the best way to resolve it,\n> though. John mentions a substantial reduction (\"roughly half\") of Git\n> processes [1], and if they foresee needing to access things other than\n> object info and contents, it might be better to start with something\n> more extensible, like my proposal for a specific batch command.\n\nI agree that it would be ideal to have just one way generic and\nextensible enough. I do not know if there are much difference in\nthat area between the two approaches, though. The RFC I saw did\nlook more complex and rigidly specified with framing and such, but\nthat is only the syntax part---in the way in which interaction\nbetween two processes happen, I didn't quite see fundamental\ndifferences. I'd expect it wouldn't be too much trouble to add new\ncommands to code written using either approach (although I haven't\nseen yours yet ;-).\n\nThanks.\n"},{"id":"447997","messageId":"fa6294387ab223d56e6ab66448f1ebe1ec370cd7.1644353884.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v3.git.git.1644353884.gitgitgadget@gmail.com","subject":"[PATCH v3 1/3] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-08T20:58:02Z","receivedAt":"2022-02-08T22:24:53Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIn the next patch, we will add an enum on the batch_options struct that\nindicates which type of batch operation will be used: --batch,\n--batch-check and the soon to be --batch-command that will read\ncommands from stdin. --batch-command mode might get confused with\nthe cmdmode flag.\n\nThere is value in renaming cmdmode in any case. cmdmode refers to how\nthe result output of the blob will be transformed, either according to\n--filter or --textconv. So transform_mode is a more descriptive name\nfor the flag.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"448001","messageId":"1ab5524ee874fc7af65efaca37fb7cd933ea8913.1644353884.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v3.git.git.1644353884.gitgitgadget@gmail.com","subject":"[PATCH v3 3/3] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-08T20:58:04Z","receivedAt":"2022-02-08T22:25:10Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] LF\n [arg1] [arg2] LF\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush LF\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n--batch-command also will not allow (B) to flush to stdout until a flush\nis received.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents LF\ninfo LF\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes a argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\nflush\ninfo LF\nflush\n\nWhen used without --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\ninfo LF\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 24 ++++\n builtin/cat-file.c | 140 ++++++++++++++++++++--\n t/t1006-cat-file.sh | 207 ++++++++++++++++++++++++++++++++-\n 3 files changed, 361 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..d77a61c47de 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,30 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+\tEnter a command mode that reads commands and arguments from stdin.\n+\tMay not be combined with any other options or arguments except\n+\t`--textconv` or `--filters`, in which case the input lines also need to\n+\tspecify the path, separated by whitespace. See the section\n+\t`BATCH OUTPUT` below for details.\n++\n+--\n+contents ::\n+\tPrint object contents for object reference . This corresponds to\n+\tthe output of --batch.\n+\n+info ::\n+\tPrint object info for object reference . This corresponds to the\n+\toutput of --batch-check.\n+\n+flush::\n+\tUsed in --buffer mode to execute all preceding commands that were issued\n+\tsince the beginning or since the last flush was issued. When --buffer\n+\tis used, no output will come until flush is issued. When --buffer is not\n+\tused, commands are flushed each time without issuing `flush`.\n+--\n++\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 1c673385868..ec266ff95e9 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,15 +17,16 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n-enum batch_command {\n-\tBATCH_COMMAND_CONTENTS,\n-\tBATCH_COMMAND_INFO,\n+enum batch_mode {\n+\tBATCH_MODE_CONTENTS,\n+\tBATCH_MODE_INFO,\n+\tBATCH_MODE_PARSE_CMDS,\n };\n \n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tenum batch_command command_mode;\n+\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -391,7 +392,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->command_mode == BATCH_COMMAND_CONTENTS) {\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -513,6 +514,117 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tconst char *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_CONTENTS;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_INFO;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void flush_batch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmds,\n+\t\tint nr)\n+{\n+\tint i;\n+\tfor (i = 0; i < nr; i++)\n+\t\tcmds[i].fn(opt, cmds[i].line, output, data);\n+\n+\tfflush(stdout);\n+}\n+\n+static const struct parse_cmd {\n+\tconst char *prefix;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *cmds = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p = NULL, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n+\t\t\tif (!opt->buffer_output)\n+\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n+\t\t\tif (*cmd_end)\n+\t\t\t\tdie(_(\"flush takes no arguments\"));\n+\t\t\tif (!nr)\n+\t\t\t\terror(_(\"nothing to flush\"));\n+\n+\t\t\tflush_batch_calls(opt, output, data, cmds, nr);\n+\t\t\tnr = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args)\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tcontinue;\n+\t\t}\n+\t\t\n+\t\tALLOC_GROW(cmds, nr + 1, alloc);\n+\t\tcall.fn = cmd->fn;\n+\t\tcall.line = xstrdup(p);\n+\t\tcmds[nr++] = call;\n+\t}\n+\n+\tif (opt->buffer_output && nr)\n+\t\tflush_batch_calls(opt, output, data, cmds, nr);\n+\n+\tfree(cmds);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -541,7 +653,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->command_mode == BATCH_COMMAND_CONTENTS)\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -595,6 +707,10 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (opt->batch_mode == BATCH_MODE_PARSE_CMDS) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -613,6 +729,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -640,11 +757,12 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\n \tif (!strcmp(opt->long_name, \"batch\"))\n-\t\tbo->command_mode = BATCH_COMMAND_CONTENTS;\n+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n \tif (!strcmp(opt->long_name, \"batch-check\"))\n-\t\tbo->command_mode = BATCH_COMMAND_INFO;\n+\t\tbo->batch_mode = BATCH_MODE_INFO;\n+\tif (!strcmp(opt->long_name, \"batch-command\"))\n+\t\tbo->batch_mode = BATCH_MODE_PARSE_CMDS;\n \n \tbo->format = arg;\n \n@@ -693,6 +811,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..635667f8168 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -177,6 +177,24 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ for opt in --buffer --no-buffer\n+ do\n+\ttest -z \"$content\" ||\n+\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n+\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n+\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_write_lines \"info $sha1\" \\\n+\t\t| git cat-file --batch-command $opt >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+ done\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -213,6 +231,70 @@ $content\"\n '\n }\n \n+run_buffer_test_flush () {\n+\ttype=$1\n+\tsha1=$2\n+\tsize=$3\n+\n+\tmkfifo input &&\n+\ttest_when_finished 'rm input; exec 8<&-' &&\n+\tmkfifo output &&\n+\texec 9<>output &&\n+\ttest_when_finished 'rm output; exec 9<&-'\n+\t(\n+\t\t# TODO - Ideally we'd pipe the output of cat-file\n+\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n+\t\t# would consume all the available\n+\t\t# output. Unfortunately we cannot do this as we cannot\n+\t\t# control when sed flushes its output. We could write\n+\t\t# a test helper in C that appended a '\\' to the end of\n+\t\t# each line and flushes its output after every line.\n+\t\tgit cat-file --buffer --batch-command err &\n+\t\techo $! &&\n+\t\twait $!\n+\t) >&9 &\n+\tsh_pid=$! &&\n+\tread cat_file_pid <&9 &&\n+\ttest_when_finished \"kill $cat_file_pid\n+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n+\techo \"$sha1 $type $size\" >expect &&\n+\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n+\t# TODO - consume all available input, not just one\n+\t# line (see above).\n+\tread actual <&9 &&\n+\techo \"$actual\" >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_be_empty err\n+}\n+\n+run_buffer_test_no_flush () {\n+\ttype=$1\n+\tsha1=$2\n+\tsize=$3\n+\n+\ttouch output &&\n+\ttest_when_finished 'rm output' &&\n+\tmkfifo input &&\n+\ttest_when_finished 'rm input' &&\n+\tmkfifo pid &&\n+\texec 9<>pid &&\n+\ttest_when_finished 'rm pid; exec 9<&-'\n+\t(\n+\t\tgit cat-file --buffer --batch-command output &\n+\t\techo $! &&\n+\t\twait $!\n+\t\techo $?\n+\t) >&9 &\n+\tsh_pid=$! &&\n+\tread cat_file_pid <&9 &&\n+\ttest_when_finished \"kill $cat_file_pid\n+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n+\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n+\tkill $cat_file_pid &&\n+\tread status <&9 &&\n+\ttest_must_be_empty output\n+}\n+\n hello_content=\"Hello World\"\n hello_size=$(strlen \"$hello_content\")\n hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n@@ -224,6 +306,14 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n+ run_buffer_test_flush blob $hello_sha1 $hello_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n+ run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -238,6 +328,14 @@ tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n \n run_tests 'tree' $tree_sha1 $tree_size \"\" \"$tree_pretty_content\"\n \n+test_expect_success PIPE '--batch-command --buffer with flush for tree info' '\n+ run_buffer_test_flush tree $tree_sha1 $tree_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for tree info' '\n+ run_buffer_test_no_flush tree $tree_sha1 $tree_size false\n+'\n+\n commit_message=\"Initial commit\"\n commit_sha1=$(echo_without_newline \"$commit_message\" | git commit-tree $tree_sha1)\n commit_size=$(($(test_oid hexsz) + 137))\n@@ -249,6 +347,14 @@ $commit_message\"\n \n run_tests 'commit' $commit_sha1 $commit_size \"$commit_content\" \"$commit_content\" 1\n \n+test_expect_success PIPE '--batch-command --buffer with flush for commit info' '\n+ run_buffer_test_flush commit $commit_sha1 $commit_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for commit info' '\n+ run_buffer_test_no_flush commit $commit_sha1 $commit_size false\n+'\n+\n tag_header_without_timestamp=\"object $hello_sha1\n type blob\n tag hellotag\n@@ -263,11 +369,19 @@ tag_size=$(strlen \"$tag_content\")\n \n run_tests 'tag' $tag_sha1 $tag_size \"$tag_content\" \"$tag_content\" 1\n \n+test_expect_success PIPE '--batch-command --buffer with flush for tag info' '\n+ run_buffer_test_flush tag $tag_sha1 $tag_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for tag info' '\n+ run_buffer_test_no_flush tag $tag_sha1 $tag_size false\n+'\n+\n test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -373,6 +487,62 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+batch_command_info_input=\"info $hello_sha1\n+info $tree_sha1\n+info $commit_sha1\n+info $tag_sha1\n+info deadbeef\n+info \n+flush\n+\"\n+\n+test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n+\ttest \"$batch_check_output\" = \"$(echo_without_newline \\\n+\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n+'\n+\n+batch_command_contents_input=\"contents $hello_sha1\n+contents $commit_sha1\n+contents $tag_sha1\n+contents deadbeef\n+contents \n+flush\n+\"\n+\n+test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n+\ttest \"$(maybe_remove_timestamp \"$batch_output\" 1)\" = \\\n+\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n+'\n+\n+batch_command_mixed_input=\"info $hello_sha1\n+contents $hello_sha1\n+info $commit_sha1\n+contents $commit_sha1\n+info $tag_sha1\n+contents $tag_sha1\n+contents deadbeef\n+info \n+flush\n+\"\n+\n+batch_command_mixed_output=\"$hello_sha1 blob $hello_size\n+$hello_sha1 blob $hello_size\n+$hello_content\n+$commit_sha1 commit $commit_size\n+$commit_sha1 commit $commit_size\n+$commit_content\n+$tag_sha1 tag $tag_size\n+$tag_sha1 tag $tag_size\n+$tag_content\n+deadbeef missing\n+ missing\"\n+\n+test_expect_success \"--batch-command with mixed calls gives correct format\" '\n+\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n+\t\"$(maybe_remove_timestamp \"$(echo_without_newline \\\n+\t\"$batch_command_mixed_input\" | git cat-file --batch-command --buffer)\" 1)\"\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -963,5 +1133,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep -E \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n+\n+test_expect_success 'batch-command flush empty queue' '\n+\techo flush >cmd &&\n+\ttest_expect_code 0 git cat-file --batch-command --buffer err &&\n+\tgrep -E \"^error:.*nothing to flush.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"448007","messageId":"pull.1212.v3.git.git.1644353884.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v2.git.git.1644251611.gitgitgadget@gmail.com","subject":"[PATCH v3 0/3] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-08T20:58:01Z","receivedAt":"2022-02-08T22:25:42Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has three parts:\n\n 1. preparation patch to rename a variable\n 2. adding an enum to keep track of batch modes\n 3. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v2:\n\n * added enum to keep track of which batch mode we are in (thanks to Junio's\n feedback)\n * fixed array allocation logic (thanks to Junio's feedback)\n * added code to flush commands when --batch-commands receives an EOF and\n exits (thanks to Phillip's feedback)\n * fixed docs formatting (thanks to Jonathan's feedback)\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (3):\n cat-file: rename cmdmode to transform_mode\n cat-file: introduce batch_command enum to replace print_contents\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 24 ++++\n builtin/cat-file.c | 154 ++++++++++++++++++++++--\n t/t1006-cat-file.sh | 207 ++++++++++++++++++++++++++++++++-\n 3 files changed, 373 insertions(+), 12 deletions(-)\n\n\nbase-commit: 38062e73e009f27ea192d50481fcb5e7b0e9d6eb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v3\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v2:\n\n 1: 2d9a0b06ce5 ! 1: fa6294387ab cat-file: rename cmdmode to transform_mode\n @@ Metadata\n ## Commit message ##\n cat-file: rename cmdmode to transform_mode\n \n - When introducing a new flag --batch-command, we will add a flag on the\n - batch_options struct that indicates whether or not an interactive\n - command mode will be used that reads commands and arguments off of\n - stdin.\n + In the next patch, we will add an enum on the batch_options struct that\n + indicates which type of batch operation will be used: --batch,\n + --batch-check and the soon to be --batch-command that will read\n + commands from stdin. --batch-command mode might get confused with\n + the cmdmode flag.\n \n - An intuitive name for this flag would be \"command\", which can get\n - confusing with the already existing cmdmode.\n -\n - cmdmode refers to how the result output of the blob will be transformed,\n - either according to --filter or --textconv. So transform_mode is a more\n - descriptive name for the flag, and will not get confused with the new\n - command flag to be added in the next commit.\n + There is value in renaming cmdmode in any case. cmdmode refers to how\n + the result output of the blob will be transformed, either according to\n + --filter or --textconv. So transform_mode is a more descriptive name\n + for the flag.\n \n Rename cmdmode to transform_mode in cat-file.c\n \n -: ----------- > 2: ae2dfa512a7 cat-file: introduce batch_command enum to replace print_contents\n 2: 1b63164ad4d ! 3: 1ab5524ee87 cat-file: add --batch-command mode\n @@ Commit message\n contents. Since we have a pair of cat-file processes per repository,\n this means we can get rid of roughly half of long lived git cat-file\n processes. Given there are many repositories being accessed at any given\n - time, this can lead to huge savings since on a given server.\n + time, this can lead to huge savings.\n \n git cat-file --batch-command\n \n will enter an interactive command mode whereby the user can enter in\n commands and their arguments that get queued in memory:\n \n - [arg1] [arg2] NL\n - [arg1] [arg2] NL\n + [arg1] [arg2] LF\n + [arg1] [arg2] LF\n \n When --buffer mode is used, commands will be queued in memory until a\n flush command is issued that execute them:\n \n - flush NL\n + flush LF\n \n The reason for a flush command is that when a consumer process (A)\n talks to a git cat-file process (B) and interactively writes to and\n @@ Commit message\n With this mechanism of queueing up commands and letting (A) issue a\n flush command, process (A) can control when the buffer is flushed and\n can guarantee it will receive all of the output when in --buffer mode.\n + --batch-command also will not allow (B) to flush to stdout until a flush\n + is received.\n \n This patch adds the basic structure for adding command which can be\n extended in the future to add more commands. It also adds the following\n two commands (on top of the flush command):\n \n - contents NL\n - info NL\n + contents LF\n + info LF\n \n The contents command takes an argument and prints out the object\n contents.\n @@ Commit message\n \n These can be used in the following way with --buffer:\n \n - contents NL\n - object NL\n - object NL\n - contents NL\n + info LF\n + contents LF\n + contents LF\n + info LF\n flush\n - contents NL\n + info LF\n flush\n \n When used without --buffer:\n \n - contents NL\n - object NL\n - object NL\n - contents NL\n - contents NL\n + info LF\n + contents LF\n + contents LF\n + info LF\n + info LF\n \n Helped-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: John Cai \n @@ Documentation/git-cat-file.txt: OPTIONS\n +\t`--textconv` or `--filters`, in which case the input lines also need to\n +\tspecify the path, separated by whitespace. See the section\n +\t`BATCH OUTPUT` below for details.\n -+\n +++\n ++--\n +contents ::\n -+\tPrint object contents for object reference \n ++\tPrint object contents for object reference . This corresponds to\n ++\tthe output of --batch.\n +\n +info ::\n -+\tPrint object info for object reference \n ++\tPrint object info for object reference . This corresponds to the\n ++\toutput of --batch-check.\n +\n +flush::\n -+\tExecute all preceding commands that were issued since the beginning or\n -+\tsince the last flush command was issued. Only used with --buffer. When\n -+\t--buffer is not used, commands are flushed each time without issuing\n -+\t`flush`.\n ++\tUsed in --buffer mode to execute all preceding commands that were issued\n ++\tsince the beginning or since the last flush was issued. When --buffer\n ++\tis used, no output will come until flush is issued. When --buffer is not\n ++\tused, commands are flushed each time without issuing `flush`.\n ++--\n +++\n +\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\n \n ## builtin/cat-file.c ##\n -@@ builtin/cat-file.c: struct batch_options {\n - \tint unordered;\n - \tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n - \tconst char *format;\n -+\tint command;\n +@@\n + #include \"object-store.h\"\n + #include \"promisor-remote.h\"\n + \n +-enum batch_command {\n +-\tBATCH_COMMAND_CONTENTS,\n +-\tBATCH_COMMAND_INFO,\n ++enum batch_mode {\n ++\tBATCH_MODE_CONTENTS,\n ++\tBATCH_MODE_INFO,\n ++\tBATCH_MODE_PARSE_CMDS,\n };\n \n - static const char *force_path;\n + struct batch_options {\n + \tint enabled;\n + \tint follow_symlinks;\n +-\tenum batch_command command_mode;\n ++\tenum batch_mode batch_mode;\n + \tint buffer_output;\n + \tint all_objects;\n + \tint unordered;\n +@@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n + \tstrbuf_addch(scratch, '\\n');\n + \tbatch_write(opt, scratch->buf, scratch->len);\n + \n +-\tif (opt->command_mode == BATCH_COMMAND_CONTENTS) {\n ++\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n + \t\tprint_object_or_die(opt, data);\n + \t\tbatch_write(opt, \"\\n\", 1);\n + \t}\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\t struct strbuf *output,\n +\t\t\t struct expand_data *data)\n +{\n -+\topt->print_contents = 1;\n ++\topt->batch_mode = BATCH_MODE_CONTENTS;\n +\tbatch_one_object(line, output, opt, data);\n +}\n +\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\t struct strbuf *output,\n +\t\t\t struct expand_data *data)\n +{\n -+\topt->print_contents = 0;\n ++\topt->batch_mode = BATCH_MODE_INFO;\n +\tbatch_one_object(line, output, opt, data);\n +}\n +\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\tstruct strbuf *output,\n +\t\tstruct expand_data *data,\n +\t\tstruct queued_cmd *cmds,\n -+\t\tint queued)\n ++\t\tint nr)\n +{\n +\tint i;\n -+\tfor(i = 0; i < queued; i++){\n ++\tfor (i = 0; i < nr; i++)\n +\t\tcmds[i].fn(opt, cmds[i].line, output, data);\n -+\t}\n ++\n +\tfflush(stdout);\n +}\n +\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\tstruct strbuf input = STRBUF_INIT;\n +\tstruct queued_cmd *cmds = NULL;\n +\tsize_t alloc = 0, nr = 0;\n -+\tint queued = 0;\n +\n +\twhile (!strbuf_getline(&input, stdin)) {\n +\t\tint i;\n +\t\tconst struct parse_cmd *cmd = NULL;\n -+\t\tconst char *p, *cmd_end;\n ++\t\tconst char *p = NULL, *cmd_end;\n +\t\tstruct queued_cmd call = {0};\n +\n +\t\tif (!input.len)\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n +\t\t\tif (*cmd_end)\n +\t\t\t\tdie(_(\"flush takes no arguments\"));\n -+\t\t\tif (!queued)\n -+\t\t\t\tdie(_(\"nothing to flush\"));\n -+\t\t\tflush_batch_calls(opt, output, data, cmds, queued);\n -+\t\t\tqueued = 0;\n ++\t\t\tif (!nr)\n ++\t\t\t\terror(_(\"nothing to flush\"));\n ++\n ++\t\t\tflush_batch_calls(opt, output, data, cmds, nr);\n ++\t\t\tnr = 0;\n +\t\t\tcontinue;\n +\t\t}\n +\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\tcmd->fn(opt, p, output, data);\n +\t\t\tcontinue;\n +\t\t}\n -+\n -+\t\tqueued++;\n -+\t\tif (queued > nr) {\n -+\t\t\tALLOC_GROW(cmds, nr+1, alloc);\n -+\t\t\tnr++;\n -+\t\t}\n -+\n ++\t\t\n ++\t\tALLOC_GROW(cmds, nr + 1, alloc);\n +\t\tcall.fn = cmd->fn;\n +\t\tcall.line = xstrdup(p);\n -+\t\tcmds[queued-1] = call;\n ++\t\tcmds[nr++] = call;\n +\t}\n ++\n ++\tif (opt->buffer_output && nr)\n ++\t\tflush_batch_calls(opt, output, data, cmds, nr);\n ++\n +\tfree(cmds);\n +\tstrbuf_release(&input);\n +}\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n {\n \tstruct strbuf input = STRBUF_INIT;\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n - \tstruct expand_data data;\n - \tint save_warning;\n - \tint retval = 0;\n -+\tconst int command = opt->command;\n + \t * If we are printing out the object, then always fill in the type,\n + \t * since we will want to decide whether or not to stream.\n + \t */\n +-\tif (opt->command_mode == BATCH_COMMAND_CONTENTS)\n ++\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n + \t\tdata.info.typep = &data.type;\n \n - \tif (!opt->format)\n - \t\topt->format = \"%(objectname) %(objecttype) %(objectsize)\";\n + \tif (opt->all_objects) {\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n -+\tif (command) {\n ++\tif (opt->batch_mode == BATCH_MODE_PARSE_CMDS) {\n +\t\tbatch_objects_command(opt, &output, &data);\n +\t\tgoto cleanup;\n +\t}\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n @@ builtin/cat-file.c: static int batch_option_callback(const struct option *opt,\n + \t}\n \n \tbo->enabled = 1;\n - \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n -+\tbo->command = !strcmp(opt->long_name, \"batch-command\");\n +-\n + \tif (!strcmp(opt->long_name, \"batch\"))\n +-\t\tbo->command_mode = BATCH_COMMAND_CONTENTS;\n ++\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n + \tif (!strcmp(opt->long_name, \"batch-check\"))\n +-\t\tbo->command_mode = BATCH_COMMAND_INFO;\n ++\t\tbo->batch_mode = BATCH_MODE_INFO;\n ++\tif (!strcmp(opt->long_name, \"batch-command\"))\n ++\t\tbo->batch_mode = BATCH_MODE_PARSE_CMDS;\n + \n \tbo->format = arg;\n \n - \treturn 0;\n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n @@ t/t1006-cat-file.sh: $content\"\n \ttest_cmp expect actual\n '\n \n -+ test -z \"$content\" ||\n -+ test_expect_success \"--batch-command output of $type content is correct\" '\n -+\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n -+\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n -+\t| git cat-file --batch-command)\" $no_ts >actual &&\n -+\ttest_cmp expect actual\n -+ '\n -+\n -+ test_expect_success \"--batch-command output of $type info is correct\" '\n -+\techo \"$sha1 $type $size\" >expect &&\n -+\ttest_write_lines \"info $sha1\" | git cat-file --batch-command >actual &&\n -+\ttest_cmp expect actual\n -+ '\n ++ for opt in --buffer --no-buffer\n ++ do\n ++\ttest -z \"$content\" ||\n ++\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n ++\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n ++\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n ++\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n ++\t\ttest_cmp expect actual\n ++\t'\n ++\n ++\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n ++\t\techo \"$sha1 $type $size\" >expect &&\n ++\t\ttest_write_lines \"info $sha1\" \\\n ++\t\t| git cat-file --batch-command $opt >actual &&\n ++\t\ttest_cmp expect actual\n ++\t'\n ++ done\n +\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n @@ t/t1006-cat-file.sh: $content\"\n +\texec 9<>output &&\n +\ttest_when_finished 'rm output; exec 9<&-'\n +\t(\n ++\t\t# TODO - Ideally we'd pipe the output of cat-file\n ++\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n ++\t\t# would consume all the available\n ++\t\t# output. Unfortunately we cannot do this as we cannot\n ++\t\t# control when sed flushes its output. We could write\n ++\t\t# a test helper in C that appended a '\\' to the end of\n ++\t\t# each line and flushes its output after every line.\n +\t\tgit cat-file --buffer --batch-command err &\n +\t\techo $! &&\n +\t\twait $!\n @@ t/t1006-cat-file.sh: $content\"\n +\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n +\t# TODO - consume all available input, not just one\n +\t# line (see above).\n -+\t# check output is flushed on exit\n +\tread actual <&9 &&\n +\techo \"$actual\" >actual &&\n +\ttest_cmp expect actual &&\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch-all-objects --batch-c\n +\n +test_expect_success 'batch-command flush empty queue' '\n +\techo flush >cmd &&\n -+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n -+\tgrep -E \"^fatal:.*nothing to flush.*\" err\n ++\ttest_expect_code 0 git cat-file --batch-command --buffer err &&\n ++\tgrep -E \"^error:.*nothing to flush.*\" err\n +'\n \n test_done\n\n-- \ngitgitgadget\n"},{"id":"448009","messageId":"ae2dfa512a760446bf6d40c456cb4126e952db98.1644353884.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v3.git.git.1644353884.gitgitgadget@gmail.com","subject":"[PATCH v3 2/3] cat-file: introduce batch_command enum to replace print_contents","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-08T20:58:03Z","receivedAt":"2022-02-08T22:25:46Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nThe next patch introduces a new --batch-command flag. Including --batch\nand --batch-check, we will have a total of three batch modes. Currently,\nfrom the batch_options struct's perspective, print_options is the only\nmember used to distinguish between the different modes. This makes the\ncode harder to read.\n\nTo reduce potential confusion, replace print_contents with an enum to\nhelp readability and clarity.\n\nHelped-by: Junio C Hamano \nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..1c673385868 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,10 +17,15 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n+enum batch_command {\n+\tBATCH_COMMAND_CONTENTS,\n+\tBATCH_COMMAND_INFO,\n+};\n+\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tint print_contents;\n+\tenum batch_command command_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n+\tif (opt->command_mode == BATCH_COMMAND_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (opt->command_mode == BATCH_COMMAND_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -635,7 +640,12 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\n+\tif (!strcmp(opt->long_name, \"batch\"))\n+\t\tbo->command_mode = BATCH_COMMAND_CONTENTS;\n+\tif (!strcmp(opt->long_name, \"batch-check\"))\n+\t\tbo->command_mode = BATCH_COMMAND_INFO;\n+\n \tbo->format = arg;\n \n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"448013","messageId":"xmqqk0e4c4bi.fsf@gitster.g","threadId":"57365","inReplyTo":"ae2dfa512a760446bf6d40c456cb4126e952db98.1644353884.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/3] cat-file: introduce batch_command enum to replace print_contents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-08T23:43:29Z","receivedAt":"2022-02-08T23:43:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> +enum batch_command {\n> +\tBATCH_COMMAND_CONTENTS,\n> +\tBATCH_COMMAND_INFO,\n> +};\n> +\n> struct batch_options {\n> \tint enabled;\n> \tint follow_symlinks;\n> -\tint print_contents;\n> +\tenum batch_command command_mode;\n> \tint buffer_output;\n> \tint all_objects;\n> \tint unordered;\n> @@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n> \tstrbuf_addch(scratch, '\\n');\n> \tbatch_write(opt, scratch->buf, scratch->len);\n> \n> -\tif (opt->print_contents) {\n> +\tif (opt->command_mode == BATCH_COMMAND_CONTENTS) {\n> \t\tprint_object_or_die(opt, data);\n> \t\tbatch_write(opt, \"\\n\", 1);\n> \t}\n\nNice.\n\n> @@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n> \t * If we are printing out the object, then always fill in the type,\n> \t * since we will want to decide whether or not to stream.\n> \t */\n> -\tif (opt->print_contents)\n> +\tif (opt->command_mode == BATCH_COMMAND_CONTENTS)\n> \t\tdata.info.typep = &data.type;\n> \n> \tif (opt->all_objects) {\n> @@ -635,7 +640,12 @@ static int batch_option_callback(const struct option *opt,\n> \t}\n> \n> \tbo->enabled = 1;\n> -\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n> +\n> +\tif (!strcmp(opt->long_name, \"batch\"))\n> +\t\tbo->command_mode = BATCH_COMMAND_CONTENTS;\n> +\tif (!strcmp(opt->long_name, \"batch-check\"))\n> +\t\tbo->command_mode = BATCH_COMMAND_INFO;\n\nThis may want to become if / else if / else cascade, whose last\n\"else\" clause would say\n\n\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n\nperhaps, but it is so minor that there is no need to reroll only for\nthis.\n\nLooking good.\n"},{"id":"448015","messageId":"xmqqwni4ap0t.fsf@gitster.g","threadId":"57365","inReplyTo":"1ab5524ee874fc7af65efaca37fb7cd933ea8913.1644353884.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/3] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-08T23:59:14Z","receivedAt":"2022-02-08T23:59:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 1c673385868..ec266ff95e9 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -17,15 +17,16 @@\n> #include \"object-store.h\"\n> #include \"promisor-remote.h\"\n> \n> -enum batch_command {\n> -\tBATCH_COMMAND_CONTENTS,\n> -\tBATCH_COMMAND_INFO,\n> +enum batch_mode {\n> +\tBATCH_MODE_CONTENTS,\n> +\tBATCH_MODE_INFO,\n\nWould have been better to introduce batch_mode at [2/3] insteads of\nhaving to rename it like this, I guess.\n\n> +\tBATCH_MODE_PARSE_CMDS,\n\nWhat the new mode does looks more like queue-and-dispatch, as\nopposed to a mode that can only do \"info\" or another mode that can\nonly do \"contents\".\n\n> @@ -513,6 +514,117 @@ static int batch_unordered_packed(const struct object_id *oid,\n> \t\t\t\t data);\n> }\n> \n> +typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n> +\t\t\t struct strbuf *, struct expand_data *);\n> +\n> +struct queued_cmd {\n> +\tparse_cmd_fn_t fn;\n> +\tconst char *line;\n> +};\n> +\n> +static void parse_cmd_contents(struct batch_options *opt,\n> +\t\t\t const char *line,\n> +\t\t\t struct strbuf *output,\n> +\t\t\t struct expand_data *data)\n> +{\n> +\topt->batch_mode = BATCH_MODE_CONTENTS;\n> +\tbatch_one_object(line, output, opt, data);\n> +}\n> +\n> +static void parse_cmd_info(struct batch_options *opt,\n> +\t\t\t const char *line,\n> +\t\t\t struct strbuf *output,\n> +\t\t\t struct expand_data *data)\n> +{\n> +\topt->batch_mode = BATCH_MODE_INFO;\n> +\tbatch_one_object(line, output, opt, data);\n> +}\n\nOK.\n\n> +static void flush_batch_calls(struct batch_options *opt,\n> +\t\tstruct strbuf *output,\n> +\t\tstruct expand_data *data,\n> +\t\tstruct queued_cmd *cmds,\n> +\t\tint nr)\n> +{\n> +\tint i;\n\nHave a blank line here between the decl(s) and the first statement.\n\n> +\tfor (i = 0; i < nr; i++)\n> +\t\tcmds[i].fn(opt, cmds[i].line, output, data);\n> +\n> +\tfflush(stdout);\n> +}\n> +\n> +static const struct parse_cmd {\n> +\tconst char *prefix;\n> +\tparse_cmd_fn_t fn;\n> +\tunsigned takes_args;\n> +} commands[] = {\n> +\t{ \"contents\", parse_cmd_contents, 1},\n> +\t{ \"info\", parse_cmd_info, 1},\n> +};\n> +\n> +static void batch_objects_command(struct batch_options *opt,\n> +\t\t\t\t struct strbuf *output,\n> +\t\t\t\t struct expand_data *data)\n> +{\n> +\tstruct strbuf input = STRBUF_INIT;\n> +\tstruct queued_cmd *cmds = NULL;\n> +\tsize_t alloc = 0, nr = 0;\n> +\n> +\twhile (!strbuf_getline(&input, stdin)) {\n> +\t\tint i;\n> +\t\tconst struct parse_cmd *cmd = NULL;\n> +\t\tconst char *p = NULL, *cmd_end;\n> +\t\tstruct queued_cmd call = {0};\n> +\n> +\t\tif (!input.len)\n> +\t\t\tdie(_(\"empty command in input\"));\n> +\t\tif (isspace(*input.buf))\n> +\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n> +\n> +\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n> +\t\t\tif (!opt->buffer_output)\n> +\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n> +\t\t\tif (*cmd_end)\n> +\t\t\t\tdie(_(\"flush takes no arguments\"));\n> +\t\t\tif (!nr)\n> +\t\t\t\terror(_(\"nothing to flush\"));\n\nI am not sure if \"you already gave us flush and haven't given a new\ncommand, saying 'flush' in such a state is an error\" is a good\ninterface. What does it achieve to punish such a caller like this\n(as opposed to just iterate the loop zero times)?\n\n> +\t\t\tflush_batch_calls(opt, output, data, cmds, nr);\n\nThis iterated cmds[] array and called .fn() for each. For a command\nin the cmds[] array that takes an argument, each element in cmds[]\nhas a pointer that holds memory obtained from xstrdup_or_null().\n\nRewinding the array with \"nr = 0\" to reuse the slots we have\nallocated is good, but the memory pointed at by the .line member of\nthese elements must be freed when this happens.\n\nPerhaps flush_batch_calls() should do so while iterating, i.e.\n\n\tfor (i = 0; i < nr; i++) {\n\t\tcmds[i].fn(opt, cmds[i].line, output, data);\n\t\tfree(cmds[i].line);\n\t}\n\tfflush(stdout);\n\nor something like that?\n\nA tangent, but naming an array as singular, cmd[], would work more\nnatural when the code often works on individual elements of the\narray, i.e. work_on(cmd[4]) would name the \"4th cmd\", which would\nnot work well if the array were named cmds[].\n\n"},{"id":"448016","messageId":"20220209001139.326834-1-jonathantanmy@google.com","threadId":"57365","inReplyTo":"xmqq5yppdyc0.fsf@gitster.g","subject":"Re: [PATCH v2 2/2] cat-file: add --batch-command mode","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-09T00:11:39Z","receivedAt":"2022-02-09T00:11:53Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano writes:\n> I agree that it would be ideal to have just one way generic and\n> extensible enough. I do not know if there are much difference in\n> that area between the two approaches, though. The RFC I saw did\n> look more complex and rigidly specified with framing and such, but\n> that is only the syntax part---in the way in which interaction\n> between two processes happen, I didn't quite see fundamental\n> differences. I'd expect it wouldn't be too much trouble to add new\n> commands to code written using either approach (although I haven't\n> seen yours yet ;-).\n> \n> Thanks.\n\nIt is similar to this approach, except:\n - the approach I sent out uses pkt-line, which might be difficult to\n retrofit to cat-file if we need it\n - in the future, we want the Git side to be able to initiate requests\n - (possibly minor) it may be confusing if we add functionality to\n cat-file that is not about reading objects\n\nOther than that, yes, they are similar.\n"},{"id":"448073","messageId":"xmqqa6ez7m6t.fsf@gitster.g","threadId":"57365","inReplyTo":"pull.1212.v3.git.git.1644353884.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/3] Add cat-file --batch-command flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-09T21:40:58Z","receivedAt":"2022-02-09T21:41:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> John Cai (3):\n> cat-file: rename cmdmode to transform_mode\n> cat-file: introduce batch_command enum to replace print_contents\n> cat-file: add --batch-command mode\n>\n> Documentation/git-cat-file.txt | 24 ++++\n> builtin/cat-file.c | 154 ++++++++++++++++++++++--\n> t/t1006-cat-file.sh | 207 ++++++++++++++++++++++++++++++++-\n> 3 files changed, 373 insertions(+), 12 deletions(-)\n\nDoes t1006-cat-file.sh pass the --stress test? I have no time to\ntest it for you but I've seen \"make test\" got stuck and this is the\nonly cat-file related change in flight.\n\n"},{"id":"448074","messageId":"691561CD-A3CF-457F-ACD0-E45153EBB829@gmail.com","threadId":"57365","inReplyTo":"xmqqa6ez7m6t.fsf@gitster.g","subject":"Re: [PATCH v3 0/3] Add cat-file --batch-command flag","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-09T22:22:12Z","receivedAt":"2022-02-09T22:22:21Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Junio\n\nOn 9 Feb 2022, at 16:40, Junio C Hamano wrote:\n\n> \"John Cai via GitGitGadget\" writes:\n>\n>> John Cai (3):\n>> cat-file: rename cmdmode to transform_mode\n>> cat-file: introduce batch_command enum to replace print_contents\n>> cat-file: add --batch-command mode\n>>\n>> Documentation/git-cat-file.txt | 24 ++++\n>> builtin/cat-file.c | 154 ++++++++++++++++++++++--\n>> t/t1006-cat-file.sh | 207 ++++++++++++++++++++++++++++++++-\n>> 3 files changed, 373 insertions(+), 12 deletions(-)\n>\n> Does t1006-cat-file.sh pass the --stress test? I have no time to\n> test it for you but I've seen \"make test\" got stuck and this is the\n> only cat-file related change in flight.\n\nYes it looks like there are some failures. Thanks for pointing this out. It\nlooks like the flush test is getting stuck. I can actually reproduce it on my\nend when I do a make clean in t/ and then run the test. Will investigate.\n\nthanks!\n"},{"id":"448077","messageId":"A1234067-820F-4828-B3A1-17BE6944E18F@gmail.com","threadId":"57365","inReplyTo":"691561CD-A3CF-457F-ACD0-E45153EBB829@gmail.com","subject":"Re: [PATCH v3 0/3] Add cat-file --batch-command flag","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-09T23:10:24Z","receivedAt":"2022-02-09T23:10:35Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"On 9 Feb 2022, at 17:22, John Cai wrote:\n\n> Hi Junio\n>\n> On 9 Feb 2022, at 16:40, Junio C Hamano wrote:\n>\n>> \"John Cai via GitGitGadget\" writes:\n>>\n>>> John Cai (3):\n>>> cat-file: rename cmdmode to transform_mode\n>>> cat-file: introduce batch_command enum to replace print_contents\n>>> cat-file: add --batch-command mode\n>>>\n>>> Documentation/git-cat-file.txt | 24 ++++\n>>> builtin/cat-file.c | 154 ++++++++++++++++++++++--\n>>> t/t1006-cat-file.sh | 207 ++++++++++++++++++++++++++++++++-\n>>> 3 files changed, 373 insertions(+), 12 deletions(-)\n>>\n>> Does t1006-cat-file.sh pass the --stress test? I have no time to\n>> test it for you but I've seen \"make test\" got stuck and this is the\n>> only cat-file related change in flight.\n>\n> Yes it looks like there are some failures. Thanks for pointing this out. It\n> looks like the flush test is getting stuck. I can actually reproduce it on my\n> end when I do a make clean in t/ and then run the test. Will investigate.\n\nI believe this was the culprit, as the stress tests that failed passed once I\nremoved this:\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 9428a04482..a20c8dae85 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -237,7 +237,7 @@ run_buffer_test_flush () {\n size=$3\n\n mkfifo input &&\n- test_when_finished 'rm input; exec 8<&-' &&\n+ test_when_finished 'rm input' &&\n mkfifo output &&\n exec 9<>output &&\n test_when_finished 'rm output; exec 9<&-'\n\nI was closing a file descriptor 8 that was never opened. But, I don't fully\nunderstand why that would create problems.\n\n>\n> thanks!\n"},{"id":"448086","messageId":"pull.1212.v4.git.git.1644465706.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v3.git.git.1644353884.gitgitgadget@gmail.com","subject":"[PATCH v4 0/3] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-10T04:01:43Z","receivedAt":"2022-02-10T04:01:52Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has three parts:\n\n 1. preparation patch to rename a variable\n 2. adding an enum to keep track of batch modes\n 3. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v3 (thanks to Junio's feedback):\n\n * added cascading logic in batch_options_callback()\n * free memory for queued call input lines\n * do not throw error when flushing an empty queue\n * renamed cmds array to singular queued_cmd\n * fixed flaky test that failed --stress\n\nChanges since v2:\n\n * added enum to keep track of which batch mode we are in (thanks to Junio's\n feedback)\n * fixed array allocation logic (thanks to Junio's feedback)\n * added code to flush commands when --batch-commands receives an EOF and\n exits (thanks to Phillip's feedback)\n * fixed docs formatting (thanks to Jonathan's feedback)\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (3):\n cat-file: rename cmdmode to transform_mode\n cat-file: introduce batch_mode enum to replace print_contents\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 24 ++++\n builtin/cat-file.c | 159 +++++++++++++++++++++++--\n t/t1006-cat-file.sh | 211 ++++++++++++++++++++++++++++++++-\n 3 files changed, 382 insertions(+), 12 deletions(-)\n\n\nbase-commit: 38062e73e009f27ea192d50481fcb5e7b0e9d6eb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v4\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v3:\n\n 1: fa6294387ab = 1: fa6294387ab cat-file: rename cmdmode to transform_mode\n 2: ae2dfa512a7 ! 2: 81bc5ae1fc1 cat-file: introduce batch_command enum to replace print_contents\n @@ Metadata\n Author: John Cai \n \n ## Commit message ##\n - cat-file: introduce batch_command enum to replace print_contents\n + cat-file: introduce batch_mode enum to replace print_contents\n \n The next patch introduces a new --batch-command flag. Including --batch\n and --batch-check, we will have a total of three batch modes. Currently,\n @@ builtin/cat-file.c\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n -+enum batch_command {\n -+\tBATCH_COMMAND_CONTENTS,\n -+\tBATCH_COMMAND_INFO,\n ++enum batch_mode {\n ++\tBATCH_MODE_CONTENTS,\n ++\tBATCH_MODE_INFO,\n +};\n +\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n -\tint print_contents;\n -+\tenum batch_command command_mode;\n ++\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n -\tif (opt->print_contents) {\n -+\tif (opt->command_mode == BATCH_COMMAND_CONTENTS) {\n ++\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n \t * since we will want to decide whether or not to stream.\n \t */\n -\tif (opt->print_contents)\n -+\tif (opt->command_mode == BATCH_COMMAND_CONTENTS)\n ++\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n @@ builtin/cat-file.c: static int batch_option_callback(const struct option *opt,\n \tbo->enabled = 1;\n -\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n +\n -+\tif (!strcmp(opt->long_name, \"batch\"))\n -+\t\tbo->command_mode = BATCH_COMMAND_CONTENTS;\n -+\tif (!strcmp(opt->long_name, \"batch-check\"))\n -+\t\tbo->command_mode = BATCH_COMMAND_INFO;\n ++\tif (!strcmp(opt->long_name, \"batch\")) {\n ++\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n ++\t} else if (!strcmp(opt->long_name, \"batch-check\")) {\n ++\t\tbo->batch_mode = BATCH_MODE_INFO;\n ++\t} else {\n ++\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n ++\t}\n +\n \tbo->format = arg;\n \n 3: 1ab5524ee87 ! 3: 6c51324a662 cat-file: add --batch-command mode\n @@ Documentation/git-cat-file.txt: OPTIONS\n \n ## builtin/cat-file.c ##\n @@\n - #include \"object-store.h\"\n - #include \"promisor-remote.h\"\n - \n --enum batch_command {\n --\tBATCH_COMMAND_CONTENTS,\n --\tBATCH_COMMAND_INFO,\n -+enum batch_mode {\n -+\tBATCH_MODE_CONTENTS,\n -+\tBATCH_MODE_INFO,\n -+\tBATCH_MODE_PARSE_CMDS,\n + enum batch_mode {\n + \tBATCH_MODE_CONTENTS,\n + \tBATCH_MODE_INFO,\n ++\tBATCH_MODE_QUEUE_AND_DISPATCH,\n };\n \n struct batch_options {\n - \tint enabled;\n - \tint follow_symlinks;\n --\tenum batch_command command_mode;\n -+\tenum batch_mode batch_mode;\n - \tint buffer_output;\n - \tint all_objects;\n - \tint unordered;\n -@@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n - \tstrbuf_addch(scratch, '\\n');\n - \tbatch_write(opt, scratch->buf, scratch->len);\n - \n --\tif (opt->command_mode == BATCH_COMMAND_CONTENTS) {\n -+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n - \t\tprint_object_or_die(opt, data);\n - \t\tbatch_write(opt, \"\\n\", 1);\n - \t}\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\n +struct queued_cmd {\n +\tparse_cmd_fn_t fn;\n -+\tconst char *line;\n ++\tchar *line;\n +};\n +\n +static void parse_cmd_contents(struct batch_options *opt,\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\tbatch_one_object(line, output, opt, data);\n +}\n +\n -+static void flush_batch_calls(struct batch_options *opt,\n ++static void dispatch_calls(struct batch_options *opt,\n +\t\tstruct strbuf *output,\n +\t\tstruct expand_data *data,\n -+\t\tstruct queued_cmd *cmds,\n ++\t\tstruct queued_cmd *cmd,\n +\t\tint nr)\n +{\n +\tint i;\n -+\tfor (i = 0; i < nr; i++)\n -+\t\tcmds[i].fn(opt, cmds[i].line, output, data);\n ++\n ++\tfor (i = 0; i < nr; i++){\n ++\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n ++\t\tfree(cmd[i].line);\n ++\t}\n +\n +\tfflush(stdout);\n +}\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\t\t struct expand_data *data)\n +{\n +\tstruct strbuf input = STRBUF_INIT;\n -+\tstruct queued_cmd *cmds = NULL;\n ++\tstruct queued_cmd *queued_cmd = NULL;\n +\tsize_t alloc = 0, nr = 0;\n +\n +\twhile (!strbuf_getline(&input, stdin)) {\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n +\t\t\tif (*cmd_end)\n +\t\t\t\tdie(_(\"flush takes no arguments\"));\n -+\t\t\tif (!nr)\n -+\t\t\t\terror(_(\"nothing to flush\"));\n +\n -+\t\t\tflush_batch_calls(opt, output, data, cmds, nr);\n ++\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n +\t\t\tnr = 0;\n +\t\t\tcontinue;\n +\t\t}\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\tcmd->fn(opt, p, output, data);\n +\t\t\tcontinue;\n +\t\t}\n -+\t\t\n -+\t\tALLOC_GROW(cmds, nr + 1, alloc);\n ++\n ++\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n +\t\tcall.fn = cmd->fn;\n -+\t\tcall.line = xstrdup(p);\n -+\t\tcmds[nr++] = call;\n ++\t\tcall.line = xstrdup_or_null(p);\n ++\t\tqueued_cmd[nr++] = call;\n +\t}\n +\n +\tif (opt->buffer_output && nr)\n -+\t\tflush_batch_calls(opt, output, data, cmds, nr);\n ++\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n +\n -+\tfree(cmds);\n ++\tfree(queued_cmd);\n +\tstrbuf_release(&input);\n +}\n +\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n -@@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n - \t * If we are printing out the object, then always fill in the type,\n - \t * since we will want to decide whether or not to stream.\n - \t */\n --\tif (opt->command_mode == BATCH_COMMAND_CONTENTS)\n -+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n - \t\tdata.info.typep = &data.type;\n - \n - \tif (opt->all_objects) {\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n -+\tif (opt->batch_mode == BATCH_MODE_PARSE_CMDS) {\n ++\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n +\t\tbatch_objects_command(opt, &output, &data);\n +\t\tgoto cleanup;\n +\t}\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n @@ builtin/cat-file.c: static int batch_option_callback(const struct option *opt,\n + \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n + \t} else if (!strcmp(opt->long_name, \"batch-check\")) {\n + \t\tbo->batch_mode = BATCH_MODE_INFO;\n ++\t} else if (!strcmp(opt->long_name, \"batch-command\")) {\n ++\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n + \t} else {\n + \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \t}\n - \n - \tbo->enabled = 1;\n --\n - \tif (!strcmp(opt->long_name, \"batch\"))\n --\t\tbo->command_mode = BATCH_COMMAND_CONTENTS;\n -+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n - \tif (!strcmp(opt->long_name, \"batch-check\"))\n --\t\tbo->command_mode = BATCH_COMMAND_INFO;\n -+\t\tbo->batch_mode = BATCH_MODE_INFO;\n -+\tif (!strcmp(opt->long_name, \"batch-command\"))\n -+\t\tbo->batch_mode = BATCH_MODE_PARSE_CMDS;\n - \n - \tbo->format = arg;\n - \n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n @@ t/t1006-cat-file.sh: $content\"\n +\tsize=$3\n +\n +\tmkfifo input &&\n -+\ttest_when_finished 'rm input; exec 8<&-' &&\n ++\ttest_when_finished 'rm input' &&\n +\tmkfifo output &&\n +\texec 9<>output &&\n +\ttest_when_finished 'rm output; exec 9<&-'\n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n +info $commit_sha1\n +info $tag_sha1\n +info deadbeef\n -+info \n +flush\n +\"\n +\n ++batch_command_info_output=\"$hello_sha1 blob $hello_size\n ++$tree_sha1 tree $tree_size\n ++$commit_sha1 commit $commit_size\n ++$tag_sha1 tag $tag_size\n ++deadbeef missing\"\n ++\n +test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n -+\ttest \"$batch_check_output\" = \"$(echo_without_newline \\\n ++\ttest \"$batch_command_info_output\" = \"$(echo_without_newline \\\n +\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n +'\n +\n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n +contents $commit_sha1\n +contents $tag_sha1\n +contents deadbeef\n -+contents \n +flush\n +\"\n +\n ++batch_command_output=\"$hello_sha1 blob $hello_size\n ++$hello_content\n ++$commit_sha1 commit $commit_size\n ++$commit_content\n ++$tag_sha1 tag $tag_size\n ++$tag_content\n ++deadbeef missing\"\n ++\n +test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n -+\ttest \"$(maybe_remove_timestamp \"$batch_output\" 1)\" = \\\n ++\ttest \"$(maybe_remove_timestamp \"$batch_command_output\" 1)\" = \\\n +\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n +'\n +\n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n +info $tag_sha1\n +contents $tag_sha1\n +contents deadbeef\n -+info \n +flush\n +\"\n +\n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n +$tag_sha1 tag $tag_size\n +$tag_sha1 tag $tag_size\n +$tag_content\n -+deadbeef missing\n -+ missing\"\n ++deadbeef missing\"\n +\n +test_expect_success \"--batch-command with mixed calls gives correct format\" '\n +\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch-all-objects --batch-c\n +\techo \"flush arg\" >cmd &&\n +\ttest_expect_code 128 git cat-file --batch-command err &&\n +\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n -+'\n -+\n -+test_expect_success 'batch-command flush empty queue' '\n -+\techo flush >cmd &&\n -+\ttest_expect_code 0 git cat-file --batch-command --buffer err &&\n -+\tgrep -E \"^error:.*nothing to flush.*\" err\n +'\n \n test_done\n\n-- \ngitgitgadget\n"},{"id":"448087","messageId":"fa6294387ab223d56e6ab66448f1ebe1ec370cd7.1644465706.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v4.git.git.1644465706.gitgitgadget@gmail.com","subject":"[PATCH v4 1/3] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-10T04:01:44Z","receivedAt":"2022-02-10T04:02:00Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIn the next patch, we will add an enum on the batch_options struct that\nindicates which type of batch operation will be used: --batch,\n--batch-check and the soon to be --batch-command that will read\ncommands from stdin. --batch-command mode might get confused with\nthe cmdmode flag.\n\nThere is value in renaming cmdmode in any case. cmdmode refers to how\nthe result output of the blob will be transformed, either according to\n--filter or --textconv. So transform_mode is a more descriptive name\nfor the flag.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"448088","messageId":"81bc5ae1fc1415b9bd751d93ba1ad305e31af4d6.1644465706.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v4.git.git.1644465706.gitgitgadget@gmail.com","subject":"[PATCH v4 2/3] cat-file: introduce batch_mode enum to replace print_contents","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-10T04:01:45Z","receivedAt":"2022-02-10T04:02:02Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nThe next patch introduces a new --batch-command flag. Including --batch\nand --batch-check, we will have a total of three batch modes. Currently,\nfrom the batch_options struct's perspective, print_options is the only\nmember used to distinguish between the different modes. This makes the\ncode harder to read.\n\nTo reduce potential confusion, replace print_contents with an enum to\nhelp readability and clarity.\n\nHelped-by: Junio C Hamano \nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 21 +++++++++++++++++----\n 1 file changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..709510c6564 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,10 +17,15 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n+enum batch_mode {\n+\tBATCH_MODE_CONTENTS,\n+\tBATCH_MODE_INFO,\n+};\n+\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tint print_contents;\n+\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -635,7 +640,15 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\n+\tif (!strcmp(opt->long_name, \"batch\")) {\n+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n+\t} else if (!strcmp(opt->long_name, \"batch-check\")) {\n+\t\tbo->batch_mode = BATCH_MODE_INFO;\n+\t} else {\n+\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n+\t}\n+\n \tbo->format = arg;\n \n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"448089","messageId":"6c51324a6623b62c385ec707a773c21375596584.1644465706.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v4.git.git.1644465706.gitgitgadget@gmail.com","subject":"[PATCH v4 3/3] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-10T04:01:46Z","receivedAt":"2022-02-10T04:02:05Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] LF\n [arg1] [arg2] LF\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush LF\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n--batch-command also will not allow (B) to flush to stdout until a flush\nis received.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents LF\ninfo LF\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes a argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\nflush\ninfo LF\nflush\n\nWhen used without --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\ninfo LF\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 24 ++++\n builtin/cat-file.c | 124 +++++++++++++++++++\n t/t1006-cat-file.sh | 211 ++++++++++++++++++++++++++++++++-\n 3 files changed, 358 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..d77a61c47de 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,30 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+\tEnter a command mode that reads commands and arguments from stdin.\n+\tMay not be combined with any other options or arguments except\n+\t`--textconv` or `--filters`, in which case the input lines also need to\n+\tspecify the path, separated by whitespace. See the section\n+\t`BATCH OUTPUT` below for details.\n++\n+--\n+contents ::\n+\tPrint object contents for object reference . This corresponds to\n+\tthe output of --batch.\n+\n+info ::\n+\tPrint object info for object reference . This corresponds to the\n+\toutput of --batch-check.\n+\n+flush::\n+\tUsed in --buffer mode to execute all preceding commands that were issued\n+\tsince the beginning or since the last flush was issued. When --buffer\n+\tis used, no output will come until flush is issued. When --buffer is not\n+\tused, commands are flushed each time without issuing `flush`.\n+--\n++\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 709510c6564..713658cc222 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -20,6 +20,7 @@\n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n \tBATCH_MODE_INFO,\n+\tBATCH_MODE_QUEUE_AND_DISPATCH,\n };\n \n struct batch_options {\n@@ -513,6 +514,118 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tchar *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_CONTENTS;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_INFO;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void dispatch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmd,\n+\t\tint nr)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < nr; i++){\n+\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\t\tfree(cmd[i].line);\n+\t}\n+\n+\tfflush(stdout);\n+}\n+\n+static const struct parse_cmd {\n+\tconst char *prefix;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *queued_cmd = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p = NULL, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n+\t\t\tif (!opt->buffer_output)\n+\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n+\t\t\tif (*cmd_end)\n+\t\t\t\tdie(_(\"flush takes no arguments\"));\n+\n+\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\t\t\tnr = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args)\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n+\t\tcall.fn = cmd->fn;\n+\t\tcall.line = xstrdup_or_null(p);\n+\t\tqueued_cmd[nr++] = call;\n+\t}\n+\n+\tif (opt->buffer_output && nr)\n+\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\n+\tfree(queued_cmd);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -595,6 +708,10 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -613,6 +730,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -645,6 +763,8 @@ static int batch_option_callback(const struct option *opt,\n \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n \t} else if (!strcmp(opt->long_name, \"batch-check\")) {\n \t\tbo->batch_mode = BATCH_MODE_INFO;\n+\t} else if (!strcmp(opt->long_name, \"batch-command\")) {\n+\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n \t} else {\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \t}\n@@ -696,6 +816,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..a20c8dae85d 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -177,6 +177,24 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ for opt in --buffer --no-buffer\n+ do\n+\ttest -z \"$content\" ||\n+\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n+\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n+\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_write_lines \"info $sha1\" \\\n+\t\t| git cat-file --batch-command $opt >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+ done\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -213,6 +231,70 @@ $content\"\n '\n }\n \n+run_buffer_test_flush () {\n+\ttype=$1\n+\tsha1=$2\n+\tsize=$3\n+\n+\tmkfifo input &&\n+\ttest_when_finished 'rm input' &&\n+\tmkfifo output &&\n+\texec 9<>output &&\n+\ttest_when_finished 'rm output; exec 9<&-'\n+\t(\n+\t\t# TODO - Ideally we'd pipe the output of cat-file\n+\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n+\t\t# would consume all the available\n+\t\t# output. Unfortunately we cannot do this as we cannot\n+\t\t# control when sed flushes its output. We could write\n+\t\t# a test helper in C that appended a '\\' to the end of\n+\t\t# each line and flushes its output after every line.\n+\t\tgit cat-file --buffer --batch-command err &\n+\t\techo $! &&\n+\t\twait $!\n+\t) >&9 &\n+\tsh_pid=$! &&\n+\tread cat_file_pid <&9 &&\n+\ttest_when_finished \"kill $cat_file_pid\n+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n+\techo \"$sha1 $type $size\" >expect &&\n+\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n+\t# TODO - consume all available input, not just one\n+\t# line (see above).\n+\tread actual <&9 &&\n+\techo \"$actual\" >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_be_empty err\n+}\n+\n+run_buffer_test_no_flush () {\n+\ttype=$1\n+\tsha1=$2\n+\tsize=$3\n+\n+\ttouch output &&\n+\ttest_when_finished 'rm output' &&\n+\tmkfifo input &&\n+\ttest_when_finished 'rm input' &&\n+\tmkfifo pid &&\n+\texec 9<>pid &&\n+\ttest_when_finished 'rm pid; exec 9<&-'\n+\t(\n+\t\tgit cat-file --buffer --batch-command output &\n+\t\techo $! &&\n+\t\twait $!\n+\t\techo $?\n+\t) >&9 &\n+\tsh_pid=$! &&\n+\tread cat_file_pid <&9 &&\n+\ttest_when_finished \"kill $cat_file_pid\n+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n+\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n+\tkill $cat_file_pid &&\n+\tread status <&9 &&\n+\ttest_must_be_empty output\n+}\n+\n hello_content=\"Hello World\"\n hello_size=$(strlen \"$hello_content\")\n hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n@@ -224,6 +306,14 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n+ run_buffer_test_flush blob $hello_sha1 $hello_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n+ run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -238,6 +328,14 @@ tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n \n run_tests 'tree' $tree_sha1 $tree_size \"\" \"$tree_pretty_content\"\n \n+test_expect_success PIPE '--batch-command --buffer with flush for tree info' '\n+ run_buffer_test_flush tree $tree_sha1 $tree_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for tree info' '\n+ run_buffer_test_no_flush tree $tree_sha1 $tree_size false\n+'\n+\n commit_message=\"Initial commit\"\n commit_sha1=$(echo_without_newline \"$commit_message\" | git commit-tree $tree_sha1)\n commit_size=$(($(test_oid hexsz) + 137))\n@@ -249,6 +347,14 @@ $commit_message\"\n \n run_tests 'commit' $commit_sha1 $commit_size \"$commit_content\" \"$commit_content\" 1\n \n+test_expect_success PIPE '--batch-command --buffer with flush for commit info' '\n+ run_buffer_test_flush commit $commit_sha1 $commit_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for commit info' '\n+ run_buffer_test_no_flush commit $commit_sha1 $commit_size false\n+'\n+\n tag_header_without_timestamp=\"object $hello_sha1\n type blob\n tag hellotag\n@@ -263,11 +369,19 @@ tag_size=$(strlen \"$tag_content\")\n \n run_tests 'tag' $tag_sha1 $tag_size \"$tag_content\" \"$tag_content\" 1\n \n+test_expect_success PIPE '--batch-command --buffer with flush for tag info' '\n+ run_buffer_test_flush tag $tag_sha1 $tag_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for tag info' '\n+ run_buffer_test_no_flush tag $tag_sha1 $tag_size false\n+'\n+\n test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -373,6 +487,72 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+batch_command_info_input=\"info $hello_sha1\n+info $tree_sha1\n+info $commit_sha1\n+info $tag_sha1\n+info deadbeef\n+flush\n+\"\n+\n+batch_command_info_output=\"$hello_sha1 blob $hello_size\n+$tree_sha1 tree $tree_size\n+$commit_sha1 commit $commit_size\n+$tag_sha1 tag $tag_size\n+deadbeef missing\"\n+\n+test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n+\ttest \"$batch_command_info_output\" = \"$(echo_without_newline \\\n+\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n+'\n+\n+batch_command_contents_input=\"contents $hello_sha1\n+contents $commit_sha1\n+contents $tag_sha1\n+contents deadbeef\n+flush\n+\"\n+\n+batch_command_output=\"$hello_sha1 blob $hello_size\n+$hello_content\n+$commit_sha1 commit $commit_size\n+$commit_content\n+$tag_sha1 tag $tag_size\n+$tag_content\n+deadbeef missing\"\n+\n+test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n+\ttest \"$(maybe_remove_timestamp \"$batch_command_output\" 1)\" = \\\n+\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n+'\n+\n+batch_command_mixed_input=\"info $hello_sha1\n+contents $hello_sha1\n+info $commit_sha1\n+contents $commit_sha1\n+info $tag_sha1\n+contents $tag_sha1\n+contents deadbeef\n+flush\n+\"\n+\n+batch_command_mixed_output=\"$hello_sha1 blob $hello_size\n+$hello_sha1 blob $hello_size\n+$hello_content\n+$commit_sha1 commit $commit_size\n+$commit_sha1 commit $commit_size\n+$commit_content\n+$tag_sha1 tag $tag_size\n+$tag_sha1 tag $tag_size\n+$tag_content\n+deadbeef missing\"\n+\n+test_expect_success \"--batch-command with mixed calls gives correct format\" '\n+\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n+\t\"$(maybe_remove_timestamp \"$(echo_without_newline \\\n+\t\"$batch_command_mixed_input\" | git cat-file --batch-command --buffer)\" 1)\"\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -963,5 +1143,34 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep -E \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"448129","messageId":"CAP8UFD1axq6x624pLFMZNT5H5nUdwWFZ0Eb8MQy8eYd3+6qthA@mail.gmail.com","threadId":"57365","inReplyTo":"81bc5ae1fc1415b9bd751d93ba1ad305e31af4d6.1644465706.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 2/3] cat-file: introduce batch_mode enum to replace print_contents","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2022-02-10T10:10:50Z","receivedAt":"2022-02-10T10:11:04Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Feb 10, 2022 at 9:46 AM John Cai via GitGitGadget\n wrote:\n>\n> From: John Cai \n>\n> The next patch introduces a new --batch-command flag. Including --batch\n> and --batch-check, we will have a total of three batch modes. Currently,\n> from the batch_options struct's perspective, print_options is the only\n\nHere you talk about \"print_options\"...\n\n> member used to distinguish between the different modes. This makes the\n> code harder to read.\n>\n> To reduce potential confusion, replace print_contents with an enum to\n\n...but here it's \"print_contents\".\n\nAlso it would perhaps be a bit clearer if you introduced it saying\nsomething like \"the print_contents flag (or boolean?) is the only\nmember...\"\n\n> help readability and clarity.\n>\n> Helped-by: Junio C Hamano \n> Signed-off-by: John Cai \n\n> @@ -635,7 +640,15 @@ static int batch_option_callback(const struct option *opt,\n> }\n>\n> bo->enabled = 1;\n> - bo->print_contents = !strcmp(opt->long_name, \"batch\");\n> +\n> + if (!strcmp(opt->long_name, \"batch\")) {\n> + bo->batch_mode = BATCH_MODE_CONTENTS;\n> + } else if (!strcmp(opt->long_name, \"batch-check\")) {\n> + bo->batch_mode = BATCH_MODE_INFO;\n> + } else {\n> + BUG(\"%s given to batch-option-callback\", opt->long_name);\n> + }\n\nI think we prefer to remove braces when there is only one instruction.\nSo the above could be just:\n\n if (!strcmp(opt->long_name, \"batch\"))\n bo->batch_mode = BATCH_MODE_CONTENTS;\n else if (!strcmp(opt->long_name, \"batch-check\"))\n bo->batch_mode = BATCH_MODE_INFO;\n else\n BUG(\"%s given to batch-option-callback\", opt->long_name);\n"},{"id":"448130","messageId":"0b40d1bb-4db1-002f-d665-f51db6c2882c@gmail.com","threadId":"57365","inReplyTo":"6c51324a6623b62c385ec707a773c21375596584.1644465706.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 3/3] cat-file: add --batch-command mode","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-10T10:57:02Z","receivedAt":"2022-02-10T10:57:09Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John\n\nI've concentrated on the tests as others have commented on the \nimplementation\n\nOn 10/02/2022 04:01, John Cai via GitGitGadget wrote:\n> From: John Cai \n> [...]\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 145eee11df9..a20c8dae85d 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -177,6 +177,24 @@ $content\"\n> \ttest_cmp expect actual\n> '\n> \n> + for opt in --buffer --no-buffer\n> + do\n> +\ttest -z \"$content\" ||\n> +\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n> +\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n> +\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n> +\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n> +\t\ttest_cmp expect actual\n> +\t'\n> +\n> +\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n> +\t\techo \"$sha1 $type $size\" >expect &&\n> +\t\ttest_write_lines \"info $sha1\" \\\n> +\t\t| git cat-file --batch-command $opt >actual &&\n> +\t\ttest_cmp expect actual\n> +\t'\n> + done\n> +\n> test_expect_success \"custom --batch-check format\" '\n> \techo \"$type $sha1\" >expect &&\n> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n> @@ -213,6 +231,70 @@ $content\"\n> '\n> }\n> \n> +run_buffer_test_flush () {\n> +\ttype=$1\n> +\tsha1=$2\n> +\tsize=$3\n> +\n> +\tmkfifo input &&\n> +\ttest_when_finished 'rm input' &&\n> +\tmkfifo output &&\n> +\texec 9<>output &&\n> +\ttest_when_finished 'rm output; exec 9<&-'\n> +\t(\n> +\t\t# TODO - Ideally we'd pipe the output of cat-file\n> +\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n> +\t\t# would consume all the available\n> +\t\t# output. Unfortunately we cannot do this as we cannot\n> +\t\t# control when sed flushes its output. We could write\n> +\t\t# a test helper in C that appended a '\\' to the end of\n> +\t\t# each line and flushes its output after every line.\n> +\t\tgit cat-file --buffer --batch-command err &\n> +\t\techo $! &&\n> +\t\twait $!\n> +\t) >&9 &\n> +\tsh_pid=$! &&\n> +\tread cat_file_pid <&9 &&\n> +\ttest_when_finished \"kill $cat_file_pid\n> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n> +\techo \"$sha1 $type $size\" >expect &&\n> +\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n\nThis closes input and so cat-file exits and flushes its output - \ntherefore you are not testing whether flush actually flushes. When I \nwrote this test in[1] this line was inside a subshell that was \nredirected to the input fifo so that the read happened before cat-file \nexited. This test is also not testing the exit code of cat-file or that \nthe output is flushed on exit. Is there a reason you can't just use the \ntest as I wrote it? I'm happy to explain anything that isn't clear.\n\n> +\t# TODO - consume all available input, not just one\n> +\t# line (see above).\n> +\tread actual <&9 &&\n> +\techo \"$actual\" >actual &&\n> +\ttest_cmp expect actual &&\n> +\ttest_must_be_empty err\n> +}\n> +\n> +run_buffer_test_no_flush () {\n> +\ttype=$1\n> +\tsha1=$2\n> +\tsize=$3\n> +\n> +\ttouch output &&\n> +\ttest_when_finished 'rm output' &&\n> +\tmkfifo input &&\n> +\ttest_when_finished 'rm input' &&\n> +\tmkfifo pid &&\n> +\texec 9<>pid &&\n> +\ttest_when_finished 'rm pid; exec 9<&-'\n> +\t(\n> +\t\tgit cat-file --buffer --batch-command output &\n> +\t\techo $! &&\n> +\t\twait $!\n> +\t\techo $?\n> +\t) >&9 &\n> +\tsh_pid=$! &&\n> +\tread cat_file_pid <&9 &&\n> +\ttest_when_finished \"kill $cat_file_pid\n> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n> +\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n\nThis prints to stdout rather than piping into cat-file so it would not \nproduce any output even if it exited normally. In my original[1] this \nline is inside a subshell that is redirected to the input fifo.\n\n> +\tkill $cat_file_pid &&\n> +\tread status <&9 &&\n> +\ttest_must_be_empty output\n> +}\n> +\n> hello_content=\"Hello World\"\n> hello_size=$(strlen \"$hello_content\")\n> hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n> @@ -224,6 +306,14 @@ test_expect_success \"setup\" '\n> \n> run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n> + run_buffer_test_flush blob $hello_sha1 $hello_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n> + run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n> +'\n\nIf we need to run the flush tests for each object type then could they \ngo inside run_tests? Personally I think I'd be happy just to test the \nflush command on one object type.\n\n> test_expect_success '--batch-check without %(rest) considers whole line' '\n> \techo \"$hello_sha1 blob $hello_size\" >expect &&\n> \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n> @@ -238,6 +328,14 @@ tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n> \n> run_tests 'tree' $tree_sha1 $tree_size \"\" \"$tree_pretty_content\"\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for tree info' '\n> + run_buffer_test_flush tree $tree_sha1 $tree_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for tree info' '\n> + run_buffer_test_no_flush tree $tree_sha1 $tree_size false\n> +'\n> +\n> commit_message=\"Initial commit\"\n> commit_sha1=$(echo_without_newline \"$commit_message\" | git commit-tree $tree_sha1)\n> commit_size=$(($(test_oid hexsz) + 137))\n> @@ -249,6 +347,14 @@ $commit_message\"\n> \n> run_tests 'commit' $commit_sha1 $commit_size \"$commit_content\" \"$commit_content\" 1\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for commit info' '\n> + run_buffer_test_flush commit $commit_sha1 $commit_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for commit info' '\n> + run_buffer_test_no_flush commit $commit_sha1 $commit_size false\n> +'\n> +\n> tag_header_without_timestamp=\"object $hello_sha1\n> type blob\n> tag hellotag\n> @@ -263,11 +369,19 @@ tag_size=$(strlen \"$tag_content\")\n> \n> run_tests 'tag' $tag_sha1 $tag_size \"$tag_content\" \"$tag_content\" 1\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for tag info' '\n> + run_buffer_test_flush tag $tag_sha1 $tag_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for tag info' '\n> + run_buffer_test_no_flush tag $tag_sha1 $tag_size false\n> +'\n> +\n> test_expect_success \\\n> \"Reach a blob from a tag pointing to it\" \\\n> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n> \n> -for batch in batch batch-check\n> +for batch in batch batch-check batch-command\n> do\n> for opt in t s e p\n> do\n> @@ -373,6 +487,72 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n> '\n> \n> +batch_command_info_input=\"info $hello_sha1\n> +info $tree_sha1\n> +info $commit_sha1\n> +info $tag_sha1\n> +info deadbeef\n\nI know there are existing uses of the constant in the file but I'm not \nthrilled about adding more.\n\n> +flush\n\nThis flush in redundant isn't it\n\n> +\"\n> +\n> +batch_command_info_output=\"$hello_sha1 blob $hello_size\n> +$tree_sha1 tree $tree_size\n> +$commit_sha1 commit $commit_size\n> +$tag_sha1 tag $tag_size\n> +deadbeef missing\"\n> +\n> +test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n\ndouble quotes are generally reserved for test titles that use parameter \nsubstitution which this one does not.\n\n> +\ttest \"$batch_command_info_output\" = \"$(echo_without_newline \\\n> +\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n> +'\n\nThis test and the one below are quite hard to follow. These days we try \nto avoid using test to compare strings as when it fails it does not \nprovide any clues as to what when wrong. Instead we use here documents \nand test_cmp so that when a test fails you can see what went wrong. Also \nthe setup happens inside the test\n\ntest_expect_success '--batch-command with multiple info calls gives \ncorrect format' '\n\tbatch_command_info_input=\"info $hello_sha1\\\n\tinfo $tree_sha1\\\n\tinfo $commit_sha1\\\n\tinfo $tag_sha1\\\n\tinfo deadbeef\\\n\tflush\"\n\t\n\tcat >expect <<-EOF &&\n\t$hello_sha1 blob $hello_size\n\t$tree_sha1 tree $tree_size\n\t$commit_sha1 commit $commit_size\n\t$tag_sha1 tag $tag_size\n\tdeadbeef missing\n\tEOF\n\n\techo_without_newline \"$batch_command_info_input\" | git cat-file \n--batch-command --buffer >actual &&\n\ttest_cmp expect actual\n'\n\n> +batch_command_contents_input=\"contents $hello_sha1\n> +contents $commit_sha1\n> +contents $tag_sha1\n> +contents deadbeef\n> +flush\n> +\"\n> +\n> +batch_command_output=\"$hello_sha1 blob $hello_size\n> +$hello_content\n> +$commit_sha1 commit $commit_size\n> +$commit_content\n> +$tag_sha1 tag $tag_size\n> +$tag_content\n> +deadbeef missing\"\n> +\n> +test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n> +\ttest \"$(maybe_remove_timestamp \"$batch_command_output\" 1)\" = \\\n> +\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n> +'\n> +\n> +batch_command_mixed_input=\"info $hello_sha1\n> +contents $hello_sha1\n> +info $commit_sha1\n> +contents $commit_sha1\n> +info $tag_sha1\n> +contents $tag_sha1\n> +contents deadbeef\n> +flush\n> +\"\n> +\n> +batch_command_mixed_output=\"$hello_sha1 blob $hello_size\n> +$hello_sha1 blob $hello_size\n> +$hello_content\n> +$commit_sha1 commit $commit_size\n> +$commit_sha1 commit $commit_size\n> +$commit_content\n> +$tag_sha1 tag $tag_size\n> +$tag_sha1 tag $tag_size\n> +$tag_content\n> +deadbeef missing\"\n> +\n> +test_expect_success \"--batch-command with mixed calls gives correct format\" '\n> +\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n> +\t\"$(maybe_remove_timestamp \"$(echo_without_newline \\\n> +\t\"$batch_command_mixed_input\" | git cat-file --batch-command --buffer)\" 1)\"\n> +'\n> +\n> test_expect_success 'setup blobs which are likely to delta' '\n> \ttest-tool genrandom foo 10240 >foo &&\n> \t{ cat foo && echo plus; } >foo-plus &&\n> @@ -963,5 +1143,34 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n> \techo \"$orig commit $orig_size\" >expect &&\n> \ttest_cmp expect actual\n> '\n> +test_expect_success 'batch-command empty command' '\n> +\techo \"\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep -E \"^fatal:.*empty command in input.*\" err\n> +'\n\nThis test and the ones below look good but they don't need to pass -E to \ngrep are they are not using an extended regex.\n\nBest Wishes\n\nPhillip\n\n[1] \nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com\n\n> +test_expect_success 'batch-command whitespace before command' '\n> +\techo \" info deadbeef\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep -E \"^fatal:.*whitespace before command.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command unknown command' '\n> +\techo unknown_command >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep -E \"^fatal:.*unknown command.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush with arguments' '\n> +\techo \"flush arg\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n> +\tgrep -E \"^fatal:.*flush takes no arguments.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush without --buffer' '\n> +\techo \"flush arg\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n> +'\n> \n> test_done\n\n"},{"id":"448172","messageId":"xmqq5ypmwt1m.fsf@gitster.g","threadId":"57365","inReplyTo":"0b40d1bb-4db1-002f-d665-f51db6c2882c@gmail.com","subject":"Re: [PATCH v4 3/3] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-10T17:05:57Z","receivedAt":"2022-02-10T17:06:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n>> +\ttype=$1\n>> +\tsha1=$2\n>> +\tsize=$3\n>> +\n>> +\tmkfifo input &&\n>> +\ttest_when_finished 'rm input' &&\n>> +\tmkfifo output &&\n>> +\texec 9<>output &&\n>> +\ttest_when_finished 'rm output; exec 9<&-'\n>> +\t(\n>> +\t\t# TODO - Ideally we'd pipe the output of cat-file\n>> +\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n>> +\t\t# would consume all the available\n>> +\t\t# output. Unfortunately we cannot do this as we cannot\n>> +\t\t# control when sed flushes its output. We could write\n>> +\t\t# a test helper in C that appended a '\\' to the end of\n>> +\t\t# each line and flushes its output after every line.\n>> +\t\tgit cat-file --buffer --batch-command err &\n>> +\t\techo $! &&\n>> +\t\twait $!\n>> +\t) >&9 &\n>> +\tsh_pid=$! &&\n>> +\tread cat_file_pid <&9 &&\n>> +\ttest_when_finished \"kill $cat_file_pid\n>> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n>> +\techo \"$sha1 $type $size\" >expect &&\n>> +\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n>\n> This closes input and so cat-file exits and flushes its output -\n> therefore you are not testing whether flush actually flushes. When I \n> wrote this test in[1] this line was inside a subshell that was\n> redirected to the input fifo so that the read happened before cat-file \n> exited.\n\nYeah, very good point.\n\n> This test is also not testing the exit code of cat-file or\n> that the output is flushed on exit. Is there a reason you can't just\n> use the test as I wrote it? I'm happy to explain anything that isn't\n> clear.\n\nI admit I do not offhand recall what your tests did but help with\nthis (and more) level of detail with an offer to collaborate is\nsomething I am very happy to see. Thanks for working well together.\n\nOne thing that I wasn't quite sure was how well failure cases are\ntested. If we ask, in a batch mode, \"info\" for two objects and then\n\"flush\", does the asker get enough clue when to read and when to\nstop reading with all four combinations of states, i.e. asking for\ntwo missing objects, one good object and one bad object, one bad\nobject and one good object, two good objects, for example?\n\nTesting such combinations reliably is tricky---if the asker needs to\nreact to different response differently, a test that expects good\nand then bad may not just fail but can get into deadlock, for\nexample if the reaction to good response has to read a lot but the\nreaction to bad response is to just consume the \"bad object\" notice,\nwhen a bug in the program being tested makes it issue the response\nfor a bad case when the asker is expecting a response for a good\nobject, because the asker will keep waiting for more response to\nread which may not come.\n"},{"id":"448180","messageId":"D9F8BB0D-FCB9-4E35-BD02-5F898134E4BB@gmail.com","threadId":"57365","inReplyTo":"0b40d1bb-4db1-002f-d665-f51db6c2882c@gmail.com","subject":"Re: [PATCH v4 3/3] cat-file: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-10T18:55:38Z","receivedAt":"2022-02-10T18:55:43Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Phillip,\n\nThanks again for helping with this! a few comments/questions below:\n\nOn 10 Feb 2022, at 5:57, Phillip Wood wrote:\n\n> Hi John\n>\n> I've concentrated on the tests as others have commented on the implementation\n>\n> On 10/02/2022 04:01, John Cai via GitGitGadget wrote:\n>> From: John Cai \n>> [...]\n>> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n>> index 145eee11df9..a20c8dae85d 100755\n>> --- a/t/t1006-cat-file.sh\n>> +++ b/t/t1006-cat-file.sh\n>> @@ -177,6 +177,24 @@ $content\"\n>> \ttest_cmp expect actual\n>> '\n>> + for opt in --buffer --no-buffer\n>> + do\n>> +\ttest -z \"$content\" ||\n>> +\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n>> +\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n>> +\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n>> +\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t'\n>> +\n>> +\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n>> +\t\techo \"$sha1 $type $size\" >expect &&\n>> +\t\ttest_write_lines \"info $sha1\" \\\n>> +\t\t| git cat-file --batch-command $opt >actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t'\n>> + done\n>> +\n>> test_expect_success \"custom --batch-check format\" '\n>> \techo \"$type $sha1\" >expect &&\n>> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n>> @@ -213,6 +231,70 @@ $content\"\n>> '\n>> }\n>> +run_buffer_test_flush () {\n>> +\ttype=$1\n>> +\tsha1=$2\n>> +\tsize=$3\n>> +\n>> +\tmkfifo input &&\n>> +\ttest_when_finished 'rm input' &&\n>> +\tmkfifo output &&\n>> +\texec 9<>output &&\n>> +\ttest_when_finished 'rm output; exec 9<&-'\n>> +\t(\n>> +\t\t# TODO - Ideally we'd pipe the output of cat-file\n>> +\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n>> +\t\t# would consume all the available\n>> +\t\t# output. Unfortunately we cannot do this as we cannot\n>> +\t\t# control when sed flushes its output. We could write\n>> +\t\t# a test helper in C that appended a '\\' to the end of\n>> +\t\t# each line and flushes its output after every line.\n>> +\t\tgit cat-file --buffer --batch-command err &\n>> +\t\techo $! &&\n>> +\t\twait $!\n>> +\t) >&9 &\n>> +\tsh_pid=$! &&\n>> +\tread cat_file_pid <&9 &&\n>> +\ttest_when_finished \"kill $cat_file_pid\n>> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n>> +\techo \"$sha1 $type $size\" >expect &&\n>> +\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n>\n> This closes input and so cat-file exits and flushes its output - therefore you are not testing whether flush actually flushes. When I wrote this test in[1] this line was inside a subshell that was redirected to the input fifo so that the read happened before cat-file exited. This test is also not testing the exit code of cat-file or that the output is flushed on exit. Is there a reason you can't just use the test as I wrote it? I'm happy to explain anything that isn't clear.\n\nI've restored the tests in the form you suggested. I had removed some lines to simplify the test but as it turns out I removed some of the important aspects of the test.\n\nHere are my modifications to the tests you helped me with. Let me know if these changes make sense, or if I'm missing something.\n\n> @@ -3,6 +3,7 @@ run_buffer_test_flush () {\n> sha1=$2\n> size=$3\n>\n> + rm -f input output &&\n\non my end some tests were hanging because these files were not getting removed\nby test_when_finished.\n\n> mkfifo input &&\n> test_when_finished 'rm input'\n> mkfifo output &&\n> @@ -26,7 +27,7 @@ run_buffer_test_flush () {\n> test_when_finished \"kill $cat_file_pid\n> kill $sh_pid; wait $sh_pid; :\" &&\n> (\n> - test_write_lines \"info $sha1\" fflush \"info $sha1\" &&\n> + test_write_lines \"info $sha1\" flush \"info $sha1\" &&\n> # TODO - consume all available input, not just one\n> # line (see above).\n> read actual <&9 &&\n> @@ -48,13 +49,14 @@ run_buffer_test_no_flush () {\n> sha1=$2\n> size=$3\n>\n> + touch output &&\n\nIt looks like test_must_be_empty expects a file, and if output is never written\nto it doesn't open the file.\n\n> mkfifo input &&\n> test_when_finished 'rm input'\n> mkfifo pid &&\n> exec 9<>pid &&\n> test_when_finished 'rm pid; exec 9<&-'\n> (\n> - git cat-file --buffer --batch-command output &\n> + git cat-file --buffer --batch-command >output &\n> echo $! &&\n> wait $!\n> echo $?\n> @@ -67,7 +69,7 @@ run_buffer_test_no_flush () {\n> test_write_lines \"info $sha1\" \"info $sha1\" &&\n> kill $cat_file_pid &&\n> read status <&9 &&\n> - test \"$status\" -ne 0 &&\n> - test_must_be_empty output\n> - ) >input\n> + test \"$status\" -ne 0\n> + ) >input &&\n> + test_must_be_empty output\n\nI wanted to ask about this, because the test hung here. I surmised that it was\nbecause we are checking the output before writing to the input.\n\n> }\n\n>\n>> +\t# TODO - consume all available input, not just one\n>> +\t# line (see above).\n>> +\tread actual <&9 &&\n>> +\techo \"$actual\" >actual &&\n>> +\ttest_cmp expect actual &&\n>> +\ttest_must_be_empty err\n>> +}\n>> +\n>> +run_buffer_test_no_flush () {\n>> +\ttype=$1\n>> +\tsha1=$2\n>> +\tsize=$3\n>> +\n>> +\ttouch output &&\n>> +\ttest_when_finished 'rm output' &&\n>> +\tmkfifo input &&\n>> +\ttest_when_finished 'rm input' &&\n>> +\tmkfifo pid &&\n>> +\texec 9<>pid &&\n>> +\ttest_when_finished 'rm pid; exec 9<&-'\n>> +\t(\n>> +\t\tgit cat-file --buffer --batch-command output &\n>> +\t\techo $! &&\n>> +\t\twait $!\n>> +\t\techo $?\n>> +\t) >&9 &\n>> +\tsh_pid=$! &&\n>> +\tread cat_file_pid <&9 &&\n>> +\ttest_when_finished \"kill $cat_file_pid\n>> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n>> +\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n>\n> This prints to stdout rather than piping into cat-file so it would not produce any output even if it exited normally. In my original[1] this line is inside a subshell that is redirected to the input fifo.\n>\n>> +\tkill $cat_file_pid &&\n>> +\tread status <&9 &&\n>> +\ttest_must_be_empty output\n>> +}\n>> +\n>> hello_content=\"Hello World\"\n>> hello_size=$(strlen \"$hello_content\")\n>> hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n>> @@ -224,6 +306,14 @@ test_expect_success \"setup\" '\n>> run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n>> +test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n>> + run_buffer_test_flush blob $hello_sha1 $hello_size\n>> +'\n>> +\n>> +test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n>> + run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n>> +'\n>\n> If we need to run the flush tests for each object type then could they go inside run_tests? Personally I think I'd be happy just to test the flush command on one object type.\n\nyeah, that makes sense\n\n>\n>> test_expect_success '--batch-check without %(rest) considers whole line' '\n>> \techo \"$hello_sha1 blob $hello_size\" >expect &&\n>> \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n>> @@ -238,6 +328,14 @@ tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n>> run_tests 'tree' $tree_sha1 $tree_size \"\" \"$tree_pretty_content\"\n>> +test_expect_success PIPE '--batch-command --buffer with flush for tree info' '\n>> + run_buffer_test_flush tree $tree_sha1 $tree_size\n>> +'\n>> +\n>> +test_expect_success PIPE '--batch-command --buffer without flush for tree info' '\n>> + run_buffer_test_no_flush tree $tree_sha1 $tree_size false\n>> +'\n>> +\n>> commit_message=\"Initial commit\"\n>> commit_sha1=$(echo_without_newline \"$commit_message\" | git commit-tree $tree_sha1)\n>> commit_size=$(($(test_oid hexsz) + 137))\n>> @@ -249,6 +347,14 @@ $commit_message\"\n>> run_tests 'commit' $commit_sha1 $commit_size \"$commit_content\" \"$commit_content\" 1\n>> +test_expect_success PIPE '--batch-command --buffer with flush for commit info' '\n>> + run_buffer_test_flush commit $commit_sha1 $commit_size\n>> +'\n>> +\n>> +test_expect_success PIPE '--batch-command --buffer without flush for commit info' '\n>> + run_buffer_test_no_flush commit $commit_sha1 $commit_size false\n>> +'\n>> +\n>> tag_header_without_timestamp=\"object $hello_sha1\n>> type blob\n>> tag hellotag\n>> @@ -263,11 +369,19 @@ tag_size=$(strlen \"$tag_content\")\n>> run_tests 'tag' $tag_sha1 $tag_size \"$tag_content\" \"$tag_content\" 1\n>> +test_expect_success PIPE '--batch-command --buffer with flush for tag info' '\n>> + run_buffer_test_flush tag $tag_sha1 $tag_size\n>> +'\n>> +\n>> +test_expect_success PIPE '--batch-command --buffer without flush for tag info' '\n>> + run_buffer_test_no_flush tag $tag_sha1 $tag_size false\n>> +'\n>> +\n>> test_expect_success \\\n>> \"Reach a blob from a tag pointing to it\" \\\n>> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n>> -for batch in batch batch-check\n>> +for batch in batch batch-check batch-command\n>> do\n>> for opt in t s e p\n>> do\n>> @@ -373,6 +487,72 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n>> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n>> '\n>> +batch_command_info_input=\"info $hello_sha1\n>> +info $tree_sha1\n>> +info $commit_sha1\n>> +info $tag_sha1\n>> +info deadbeef\n>\n> I know there are existing uses of the constant in the file but I'm not thrilled about adding more.\n>\n>> +flush\n>\n> This flush in redundant isn't it\n\ntrue, we don't actually need it\n\n>\n>> +\"\n>> +\n>> +batch_command_info_output=\"$hello_sha1 blob $hello_size\n>> +$tree_sha1 tree $tree_size\n>> +$commit_sha1 commit $commit_size\n>> +$tag_sha1 tag $tag_size\n>> +deadbeef missing\"\n>> +\n>> +test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n>\n> double quotes are generally reserved for test titles that use parameter substitution which this one does not.\n>\n>> +\ttest \"$batch_command_info_output\" = \"$(echo_without_newline \\\n>> +\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n>> +'\n>\n> This test and the one below are quite hard to follow. These days we try to avoid using test to compare strings as when it fails it does not provide any clues as to what when wrong. Instead we use here documents and test_cmp so that when a test fails you can see what went wrong. Also the setup happens inside the test\n>\n> test_expect_success '--batch-command with multiple info calls gives correct format' '\n> batch_command_info_input=\"info $hello_sha1\\\n> info $tree_sha1\\\n> info $commit_sha1\\\n> info $tag_sha1\\\n> info deadbeef\\\n> flush\"\n>\n> cat >expect <<-EOF &&\n> $hello_sha1 blob $hello_size\n> $tree_sha1 tree $tree_size\n> $commit_sha1 commit $commit_size\n> $tag_sha1 tag $tag_size\n> deadbeef missing\n> EOF\n>\n> echo_without_newline \"$batch_command_info_input\" | git cat-file --batch-command --buffer >actual &&\n> test_cmp expect actual\n> '\n\nsounds good, will adjust\n>\n>> +batch_command_contents_input=\"contents $hello_sha1\n>> +contents $commit_sha1\n>> +contents $tag_sha1\n>> +contents deadbeef\n>> +flush\n>> +\"\n>> +\n>> +batch_command_output=\"$hello_sha1 blob $hello_size\n>> +$hello_content\n>> +$commit_sha1 commit $commit_size\n>> +$commit_content\n>> +$tag_sha1 tag $tag_size\n>> +$tag_content\n>> +deadbeef missing\"\n>> +\n>> +test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n>> +\ttest \"$(maybe_remove_timestamp \"$batch_command_output\" 1)\" = \\\n>> +\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n>> +'\n>> +\n>> +batch_command_mixed_input=\"info $hello_sha1\n>> +contents $hello_sha1\n>> +info $commit_sha1\n>> +contents $commit_sha1\n>> +info $tag_sha1\n>> +contents $tag_sha1\n>> +contents deadbeef\n>> +flush\n>> +\"\n>> +\n>> +batch_command_mixed_output=\"$hello_sha1 blob $hello_size\n>> +$hello_sha1 blob $hello_size\n>> +$hello_content\n>> +$commit_sha1 commit $commit_size\n>> +$commit_sha1 commit $commit_size\n>> +$commit_content\n>> +$tag_sha1 tag $tag_size\n>> +$tag_sha1 tag $tag_size\n>> +$tag_content\n>> +deadbeef missing\"\n>> +\n>> +test_expect_success \"--batch-command with mixed calls gives correct format\" '\n>> +\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n>> +\t\"$(maybe_remove_timestamp \"$(echo_without_newline \\\n>> +\t\"$batch_command_mixed_input\" | git cat-file --batch-command --buffer)\" 1)\"\n>> +'\n>> +\n>> test_expect_success 'setup blobs which are likely to delta' '\n>> \ttest-tool genrandom foo 10240 >foo &&\n>> \t{ cat foo && echo plus; } >foo-plus &&\n>> @@ -963,5 +1143,34 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n>> \techo \"$orig commit $orig_size\" >expect &&\n>> \ttest_cmp expect actual\n>> '\n>> +test_expect_success 'batch-command empty command' '\n>> +\techo \"\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep -E \"^fatal:.*empty command in input.*\" err\n>> +'\n>\n> This test and the ones below look good but they don't need to pass -E to grep are they are not using an extended regex.\n>\n> Best Wishes\n>\n> Phillip\n>\n> [1] https://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com\n>\n>> +test_expect_success 'batch-command whitespace before command' '\n>> +\techo \" info deadbeef\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep -E \"^fatal:.*whitespace before command.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command unknown command' '\n>> +\techo unknown_command >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep -E \"^fatal:.*unknown command.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command flush with arguments' '\n>> +\techo \"flush arg\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n>> +\tgrep -E \"^fatal:.*flush takes no arguments.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command flush without --buffer' '\n>> +\techo \"flush arg\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n>> +'\n>> test_done\n"},{"id":"448190","messageId":"xmqq8ruiforz.fsf@gitster.g","threadId":"57365","inReplyTo":"pull.1212.v4.git.git.1644465706.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/3] Add cat-file --batch-command flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-10T20:30:08Z","receivedAt":"2022-02-10T20:30:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> The feature proposal of adding a command interface to cat-file was first\n> discussed in [A]. In [B], Taylor expressed the need for a fuller proposal\n> before moving forward with a new flag. An RFC was created [C] and the idea\n> was discussed more thoroughly, and overall it seemed like it was headed in\n> the right direction.\n>\n> This patch series consolidates the feedback from these different threads.\n>\n> This patch series has three parts:\n>\n> 1. preparation patch to rename a variable\n> 2. adding an enum to keep track of batch modes\n> 3. logic to handle --batch-command flag, adding contents, info, flush\n> commands\n>\n> Changes since v3 (thanks to Junio's feedback):\n>\n> * added cascading logic in batch_options_callback()\n> * free memory for queued call input lines\n> * do not throw error when flushing an empty queue\n> * renamed cmds array to singular queued_cmd\n> * fixed flaky test that failed --stress\n\nUnfortunately this round still seems to fail fairly reliably without\n--stress. FWIW, I usually run tests with \"cd t && make prove\" but\nit seems to get stuck after passing \n\n ok 69 - --batch without size (blob)\n ok 70 - --batch-command --buffer with flush for blob info\n\neven when the test is run without prove.\n\n"},{"id":"448194","messageId":"CAPig+cQ6mOipiCKSOUo-d0UgoGq+YwShiw0CgcqGvRqxV5K77g@mail.gmail.com","threadId":"57365","inReplyTo":"6c51324a6623b62c385ec707a773c21375596584.1644465706.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 3/3] cat-file: add --batch-command mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-10T22:46:39Z","receivedAt":"2022-02-10T22:46:57Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Feb 9, 2022 at 11:01 PM John Cai via GitGitGadget\n wrote:\n> Add a new flag --batch-command that accepts commands and arguments\n> from stdin, similar to git-update-ref --stdin.\n\nSome comments not offered by other reviewers...\n\n> This patch adds the basic structure for adding command which can be\n> extended in the future to add more commands. It also adds the following\n> two commands (on top of the flush command):\n>\n> contents LF\n> info LF\n>\n> The contents command takes an argument and prints out the object\n> contents.\n>\n> The info command takes a argument and prints out the object\n> metadata.\n>\n> These can be used in the following way with --buffer:\n>\n> info LF\n> contents LF\n> contents LF\n> info LF\n> flush\n> info LF\n> flush\n\ns/// for consistency with the usage information earlier\nin the commit message, and since Git is migrating to SHA-256, and to\navoid reviewer confusion as occurred earlier[1].\n\nAlso: s/flush$/flush LF/\n\n> When used without --buffer:\n>\n> info LF\n> contents LF\n> contents LF\n> info LF\n> info LF\n\nDitto.\n\n[1]: https://lore.kernel.org/git/CAPig+cTeqhOYTu9WBiY=LnZtt35hAp3Qa5RduC2yLut6p01_1w@mail.gmail.com/\n\n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> @@ -96,6 +96,30 @@ OPTIONS\n> +--batch-command::\n> + Enter a command mode that reads commands and arguments from stdin.\n> + May not be combined with any other options or arguments except\n> + `--textconv` or `--filters`, in which case the input lines also need to\n> + specify the path, separated by whitespace. See the section\n> + `BATCH OUTPUT` below for details.\n\nThe SYNOPSIS probably needs an update too.\n\nPerhaps say something like \"Recognized commands include:\" here before\nenumerating the commands themselves?\n\n> +--\n> +contents ::\n> + Print object contents for object reference . This corresponds to\n> + the output of --batch.\n\ns//``/\ns/--batch/`--batch`/\n\n> +info ::\n> + Print object info for object reference . This corresponds to the\n> + output of --batch-check.\n\ns//``/\ns/--batch/`--batch-check`/\n\n> +flush::\n> + Used in --buffer mode to execute all preceding commands that were issued\n> + since the beginning or since the last flush was issued. When --buffer\n> + is used, no output will come until flush is issued. When --buffer is not\n> + used, commands are flushed each time without issuing `flush`.\n> +--\n\ns/--buffer/`--buffer`/g\ns/flush/`flush`/g\n\nThis says that it's legal to use `--buffer` along with\n`--batch-command`, but the description of `--batch-command` itself\njust above says that it can be combined only with `--textconv` or\n`--filters`. (I see you copied the problematic text from the other\nbatch options, so they also are guilty of not mentioning `--buffer`.\nThis series doesn't necessarily need to fix those existing\ndocumentation problems, but perhaps don't repeat the problem with\nnewly-added text?)\n\nThe description of the `--buffer` option probably also needs to be\nupdated to mention the new `--batch-command` option, and there may be\nother places in this document which should mention it, as well.\n\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> +static const struct parse_cmd {\n> + const char *prefix;\n> + parse_cmd_fn_t fn;\n> + unsigned takes_args;\n> +} commands[] = {\n> + { \"contents\", parse_cmd_contents, 1},\n> + { \"info\", parse_cmd_info, 1},\n> +};\n> +\n> +static void batch_objects_command(struct batch_options *opt,\n> + struct strbuf *output,\n> + struct expand_data *data)\n> +{\n> + while (!strbuf_getline(&input, stdin)) {\n> + if (!input.len)\n> + die(_(\"empty command in input\"));\n> + if (isspace(*input.buf))\n> + die(_(\"whitespace before command: '%s'\"), input.buf);\n> +\n> + if (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n> + if (!opt->buffer_output)\n> + die(_(\"flush is only for --buffer mode\"));\n> + if (*cmd_end)\n> + die(_(\"flush takes no arguments\"));\n> +\n> + dispatch_calls(opt, output, data, queued_cmd, nr);\n> + nr = 0;\n> + continue;\n> + }\n> +\n> + for (i = 0; i < ARRAY_SIZE(commands); i++) {\n> + if (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n> + continue;\n\nThis prefix-matching is going to incorrectly match non-commands such\nas \"contentsify \" and \"information \" and then treat\nthem as \"contents fy \" and \"info mation \",\nrespectively, with undesirable results. You need to verify that there\nis a space or NUL at `*cmd_end` before treating `input.buf` as an\nactual command.\n\n> + cmd = &commands[i];\n> + if (cmd->takes_args)\n\nWhat happens if `cmd->takes_arg` is true but no arguments follow the\ncommand? Should that be diagnosed as an error?\n\n> + p = cmd_end + 1;\n\nThis unconditional +1 is going to make `p` point beyond the NUL\ncharacter if the input is just a bare command, such as \"contents\" or\n\"info\" without any space or any argument...\n\n> + break;\n> + }\n> +\n> + if (!cmd)\n> + die(_(\"unknown command: '%s'\"), input.buf);\n> +\n> + if (!opt->buffer_output) {\n> + cmd->fn(opt, p, output, data);\n> + continue;\n> + }\n> +\n> + ALLOC_GROW(queued_cmd, nr + 1, alloc);\n> + call.fn = cmd->fn;\n> + call.line = xstrdup_or_null(p);\n\n... which means that xstrdup_or_null() will be copying whatever random\ngarbage is in memory following the bare command.\n\n> + queued_cmd[nr++] = call;\n> + }\n> +\n> + if (opt->buffer_output && nr)\n> + dispatch_calls(opt, output, data, queued_cmd, nr);\n> +\n> + free(queued_cmd);\n> + strbuf_release(&input);\n> +}\n"},{"id":"448232","messageId":"9B08BF05-1C35-4C4F-9C76-CC34A8E7D0F0@gmail.com","threadId":"57365","inReplyTo":"xmqq5ypmwt1m.fsf@gitster.g","subject":"Re: [PATCH v4 3/3] cat-file: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-11T17:45:51Z","receivedAt":"2022-02-11T17:46:03Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Junio\n\nOn 10 Feb 2022, at 12:05, Junio C Hamano wrote:\n\n> Phillip Wood writes:\n>\n>>> +\ttype=$1\n>>> +\tsha1=$2\n>>> +\tsize=$3\n>>> +\n>>> +\tmkfifo input &&\n>>> +\ttest_when_finished 'rm input' &&\n>>> +\tmkfifo output &&\n>>> +\texec 9<>output &&\n>>> +\ttest_when_finished 'rm output; exec 9<&-'\n>>> +\t(\n>>> +\t\t# TODO - Ideally we'd pipe the output of cat-file\n>>> +\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n>>> +\t\t# would consume all the available\n>>> +\t\t# output. Unfortunately we cannot do this as we cannot\n>>> +\t\t# control when sed flushes its output. We could write\n>>> +\t\t# a test helper in C that appended a '\\' to the end of\n>>> +\t\t# each line and flushes its output after every line.\n>>> +\t\tgit cat-file --buffer --batch-command err &\n>>> +\t\techo $! &&\n>>> +\t\twait $!\n>>> +\t) >&9 &\n>>> +\tsh_pid=$! &&\n>>> +\tread cat_file_pid <&9 &&\n>>> +\ttest_when_finished \"kill $cat_file_pid\n>>> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n>>> +\techo \"$sha1 $type $size\" >expect &&\n>>> +\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n>>\n>> This closes input and so cat-file exits and flushes its output -\n>> therefore you are not testing whether flush actually flushes. When I\n>> wrote this test in[1] this line was inside a subshell that was\n>> redirected to the input fifo so that the read happened before cat-file\n>> exited.\n>\n> Yeah, very good point.\n>\n>> This test is also not testing the exit code of cat-file or\n>> that the output is flushed on exit. Is there a reason you can't just\n>> use the test as I wrote it? I'm happy to explain anything that isn't\n>> clear.\n>\n> I admit I do not offhand recall what your tests did but help with\n> this (and more) level of detail with an offer to collaborate is\n> something I am very happy to see. Thanks for working well together.\n>\n> One thing that I wasn't quite sure was how well failure cases are\n> tested. If we ask, in a batch mode, \"info\" for two objects and then\n> \"flush\", does the asker get enough clue when to read and when to\n> stop reading with all four combinations of states, i.e. asking for\n> two missing objects, one good object and one bad object, one bad\n> object and one good object, two good objects, for example?\n\nThis is a good point. We currently don't have tests that exercise these\ncombinations.\n\n>\n> Testing such combinations reliably is tricky---if the asker needs to\n> react to different response differently, a test that expects good\n> and then bad may not just fail but can get into deadlock, for\n> example if the reaction to good response has to read a lot but the\n> reaction to bad response is to just consume the \"bad object\" notice,\n> when a bug in the program being tested makes it issue the response\n> for a bad case when the asker is expecting a response for a good\n> object, because the asker will keep waiting for more response to\n> read which may not come.\n\nLet me see if I understand you. What I'm hearing is that it's hard to test a git\nprocesses (A) that read/write from/to pipes without knowing exactly how (A) will\nbehave. By necessity, the test logic will have embedded some logic in it that\nassumes certain behavior from (A), which might or might not be the case.\n\nThis can lead to a hanging test if, say, it is waiting around for (A) to output\ndata when due to a bug in the code, it never does. Did I get that right?\n\nI still see value in having a test that hangs when it doesn't receive expected\noutput from the git process. If we had something that detected timeout on tests\nthen this could catch such a case. But since we don't, then that means having\ntests like run_buffer_test_flush() and run_buffer_test_no_flush() will run the\nrisk of being a deadlocked test if there is a regression of the code in the future.\nWhile still providing value in showing that something is wrong, these deadlocked\ntests can be inconvenient to debug.\n\n\n\n"},{"id":"448243","messageId":"fa6294387ab223d56e6ab66448f1ebe1ec370cd7.1644609683.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v5.git.git.1644609683.gitgitgadget@gmail.com","subject":"[PATCH v5 1/3] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:01:21Z","receivedAt":"2022-02-11T20:01:33Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIn the next patch, we will add an enum on the batch_options struct that\nindicates which type of batch operation will be used: --batch,\n--batch-check and the soon to be --batch-command that will read\ncommands from stdin. --batch-command mode might get confused with\nthe cmdmode flag.\n\nThere is value in renaming cmdmode in any case. cmdmode refers to how\nthe result output of the blob will be transformed, either according to\n--filter or --textconv. So transform_mode is a more descriptive name\nfor the flag.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"448244","messageId":"pull.1212.v5.git.git.1644609683.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v4.git.git.1644465706.gitgitgadget@gmail.com","subject":"[PATCH v5 0/3] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:01:20Z","receivedAt":"2022-02-11T20:01:37Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has three parts:\n\n 1. preparation patch to rename a variable\n 2. adding an enum to keep track of batch modes\n 3. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v4\n\n * added Phillip's suggested test for testing flush. This should have\n addressed the flaky test that was hanging. I tested it on my side and\n wasn't able to reproduce the deadlock.\n * plugged some holes in the logic that parsed the command and arguments,\n thanks to Eric's feedback\n * fixed verbiage in commit messages per Christian's feedback\n * clarified places in documentation that should mention --batch-command per\n Eric's feedback\n\nChanges since v3 (thanks to Junio's feedback):\n\n * added cascading logic in batch_options_callback()\n * free memory for queued call input lines\n * do not throw error when flushing an empty queue\n * renamed cmds array to singular queued_cmd\n * fixed flaky test that failed --stress\n\nChanges since v2:\n\n * added enum to keep track of which batch mode we are in (thanks to Junio's\n feedback)\n * fixed array allocation logic (thanks to Junio's feedback)\n * added code to flush commands when --batch-commands receives an EOF and\n exits (thanks to Phillip's feedback)\n * fixed docs formatting (thanks to Jonathan's feedback)\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (3):\n cat-file: rename cmdmode to transform_mode\n cat-file: introduce batch_mode enum to replace print_contents\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 41 +++++++-\n builtin/cat-file.c | 167 +++++++++++++++++++++++++++++--\n t/t1006-cat-file.sh | 178 ++++++++++++++++++++++++++++++++-\n 3 files changed, 370 insertions(+), 16 deletions(-)\n\n\nbase-commit: 38062e73e009f27ea192d50481fcb5e7b0e9d6eb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v5\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v4:\n\n 1: fa6294387ab = 1: fa6294387ab cat-file: rename cmdmode to transform_mode\n 2: 81bc5ae1fc1 ! 2: 5e0d1161df4 cat-file: introduce batch_mode enum to replace print_contents\n @@ Commit message\n cat-file: introduce batch_mode enum to replace print_contents\n \n The next patch introduces a new --batch-command flag. Including --batch\n - and --batch-check, we will have a total of three batch modes. Currently,\n - from the batch_options struct's perspective, print_options is the only\n - member used to distinguish between the different modes. This makes the\n - code harder to read.\n + and --batch-check, we will have a total of three batch modes. print_contents\n + is the only boolean on the batch_options sturct used to distinguish\n + between the different modes. This makes the code harder to read.\n \n To reduce potential confusion, replace print_contents with an enum to\n help readability and clarity.\n @@ builtin/cat-file.c: static int batch_option_callback(const struct option *opt,\n \tbo->enabled = 1;\n -\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n +\n -+\tif (!strcmp(opt->long_name, \"batch\")) {\n ++\tif (!strcmp(opt->long_name, \"batch\"))\n +\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n -+\t} else if (!strcmp(opt->long_name, \"batch-check\")) {\n ++\telse if (!strcmp(opt->long_name, \"batch-check\"))\n +\t\tbo->batch_mode = BATCH_MODE_INFO;\n -+\t} else {\n ++\telse\n +\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n -+\t}\n +\n \tbo->format = arg;\n \n 3: 6c51324a662 ! 3: ad66d1f3e2b cat-file: add --batch-command mode\n @@ Commit message\n \n These can be used in the following way with --buffer:\n \n - info LF\n - contents LF\n - contents LF\n - info LF\n - flush\n - info LF\n - flush\n + info LF\n + contents LF\n + contents LF\n + info LF\n + flush LF\n + info LF\n + flush LF\n \n When used without --buffer:\n \n - info LF\n - contents LF\n - contents LF\n - info LF\n - info LF\n + info LF\n + contents LF\n + contents LF\n + info LF\n + info LF\n \n Helped-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: John Cai \n @@ Documentation/git-cat-file.txt: OPTIONS\n \tsection `BATCH OUTPUT` below for details.\n \n +--batch-command::\n -+\tEnter a command mode that reads commands and arguments from stdin.\n -+\tMay not be combined with any other options or arguments except\n -+\t`--textconv` or `--filters`, in which case the input lines also need to\n -+\tspecify the path, separated by whitespace. See the section\n -+\t`BATCH OUTPUT` below for details.\n ++\tEnter a command mode that reads commands and arguments from stdin. May\n ++\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n ++\tcase of `--textconv` or `--filters`, the input lines also need to specify\n ++\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n ++\tfor details.\n +++\n ++`--batch-command` recognizes the following commands:\n ++\n +--\n -+contents ::\n -+\tPrint object contents for object reference . This corresponds to\n -+\tthe output of --batch.\n ++contents ``::\n ++\tPrint object contents for object reference ``. This corresponds to\n ++\tthe output of `--batch`.\n +\n -+info ::\n -+\tPrint object info for object reference . This corresponds to the\n -+\toutput of --batch-check.\n ++info ``::\n ++\tPrint object info for object reference ``. This corresponds to the\n ++\toutput of `--batch-check`.\n +\n +flush::\n -+\tUsed in --buffer mode to execute all preceding commands that were issued\n -+\tsince the beginning or since the last flush was issued. When --buffer\n -+\tis used, no output will come until flush is issued. When --buffer is not\n -+\tused, commands are flushed each time without issuing `flush`.\n ++\tUsed with `--buffer` to execute all preceding commands that were issued\n ++\tsince the beginning or since the last flush was issued. When `--buffer`\n ++\tis used, no output will come until a `flush` is issued. When `--buffer`\n ++\tis not used, commands are flushed each time without issuing `flush`.\n +--\n ++\n +\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\n +@@ Documentation/git-cat-file.txt: OPTIONS\n + \tthat a process can interactively read and write from\n + \t`cat-file`. With this option, the output uses normal stdio\n + \tbuffering; this is much more efficient when invoking\n +-\t`--batch-check` on a large number of objects.\n ++\t`--batch-check` or `--batch-command` on a large number of objects.\n + \n + --unordered::\n + \tWhen `--batch-all-objects` is in use, visit objects in an\n +@@ Documentation/git-cat-file.txt: from stdin, one per line, and print information about them. By default,\n + the whole line is considered as an object, as if it were fed to\n + linkgit:git-rev-parse[1].\n + \n ++When `--batch-command` is given, `cat-file` will read commands from stdin,\n ++one per line, and print information based on the command given. With\n ++`--batch-command`, the `info` command followed by an object will print\n ++information about the object the same way `--batch-check` would, and the\n ++`contents` command followed by an object prints contents in the same way\n ++`--batch` would.\n ++\n + You can specify the information shown for each object by using a custom\n + ``. The `` is copied literally to stdout for each\n + object, with placeholders of the form `%(atom)` expanded, followed by a\n +@@ Documentation/git-cat-file.txt: newline. The available atoms are:\n + If no format is specified, the default format is `%(objectname)\n + %(objecttype) %(objectsize)`.\n + \n +-If `--batch` is specified, the object information is followed by the\n +-object contents (consisting of `%(objectsize)` bytes), followed by a\n +-newline.\n ++If `--batch` is specified, or if `--batch-command` is used with the `contents`\n ++command, the object information is followed by the object contents (consisting\n ++of `%(objectsize)` bytes), followed by a newline.\n + \n + For example, `--batch` without a custom format would produce:\n + \n \n ## builtin/cat-file.c ##\n @@\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\t\tcontinue;\n +\n +\t\t\tcmd = &commands[i];\n -+\t\t\tif (cmd->takes_args)\n ++\t\t\tif (cmd->takes_args) {\n ++\t\t\t\tif (*cmd_end != ' ')\n ++\t\t\t\t\tdie(_(\"%s requires arguments\"),\n ++\t\t\t\t\t commands[i].prefix);\n ++\n +\t\t\t\tp = cmd_end + 1;\n ++\t\t\t} else if (*cmd_end) {\n ++\t\t\t\tdie(_(\"%s takes no arguments\"),\n ++\t\t\t\t commands[i].prefix);\n ++\t\t\t}\n ++\n +\t\t\tbreak;\n +\t\t}\n +\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n \twarn_on_object_refname_ambiguity = save_warning;\n @@ builtin/cat-file.c: static int batch_option_callback(const struct option *opt,\n \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n - \t} else if (!strcmp(opt->long_name, \"batch-check\")) {\n + \telse if (!strcmp(opt->long_name, \"batch-check\"))\n \t\tbo->batch_mode = BATCH_MODE_INFO;\n -+\t} else if (!strcmp(opt->long_name, \"batch-command\")) {\n ++\telse if (!strcmp(opt->long_name, \"batch-command\"))\n +\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n - \t} else {\n + \telse\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n - \t}\n + \n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n @@ t/t1006-cat-file.sh: $content\"\n +\tsha1=$2\n +\tsize=$3\n +\n ++\trm -f input output &&\n +\tmkfifo input &&\n -+\ttest_when_finished 'rm input' &&\n ++\ttest_when_finished 'rm input'\n +\tmkfifo output &&\n +\texec 9<>output &&\n +\ttest_when_finished 'rm output; exec 9<&-'\n @@ t/t1006-cat-file.sh: $content\"\n +\t\tgit cat-file --buffer --batch-command err &\n +\t\techo $! &&\n +\t\twait $!\n ++\t\techo $?\n +\t) >&9 &\n +\tsh_pid=$! &&\n +\tread cat_file_pid <&9 &&\n +\ttest_when_finished \"kill $cat_file_pid\n +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n -+\techo \"$sha1 $type $size\" >expect &&\n -+\ttest_write_lines \"info $sha1\" flush \"info $sha1\" >input\n -+\t# TODO - consume all available input, not just one\n -+\t# line (see above).\n ++\t(\n ++\t\ttest_write_lines \"info $sha1\" flush \"info $sha1\" &&\n ++\t\t# TODO - consume all available input, not just one\n ++\t\t# line (see above).\n ++\t\tread actual <&9 &&\n ++\t\techo \"$actual\" >actual &&\n ++\t\techo \"$sha1 $type $size\" >expect &&\n ++\t\ttest_cmp expect actual\n ++\t) >input &&\n ++\t# check output is flushed on exit\n +\tread actual <&9 &&\n +\techo \"$actual\" >actual &&\n +\ttest_cmp expect actual &&\n -+\ttest_must_be_empty err\n ++\ttest_must_be_empty err &&\n ++\tread status <&9 &&\n ++\ttest \"$status\" -eq 0\n +}\n +\n +run_buffer_test_no_flush () {\n @@ t/t1006-cat-file.sh: $content\"\n +\tsize=$3\n +\n +\ttouch output &&\n -+\ttest_when_finished 'rm output' &&\n ++\ttest_when_finished 'rm output'\n +\tmkfifo input &&\n -+\ttest_when_finished 'rm input' &&\n ++\ttest_when_finished 'rm input'\n +\tmkfifo pid &&\n +\texec 9<>pid &&\n +\ttest_when_finished 'rm pid; exec 9<&-'\n +\t(\n -+\t\tgit cat-file --buffer --batch-command output &\n ++\t\tgit cat-file --buffer --batch-command >output &\n +\t\techo $! &&\n +\t\twait $!\n +\t\techo $?\n @@ t/t1006-cat-file.sh: $content\"\n +\tread cat_file_pid <&9 &&\n +\ttest_when_finished \"kill $cat_file_pid\n +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n -+\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n -+\tkill $cat_file_pid &&\n -+\tread status <&9 &&\n -+\ttest_must_be_empty output\n ++\t(\n ++\t\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n ++\t\tkill $cat_file_pid &&\n ++\t\tread status <&9 &&\n ++\t\ttest \"$status\" -ne 0 &&\n ++\t\ttest_must_be_empty output\n ++\t) >input\n +}\n ++\n +\n hello_content=\"Hello World\"\n hello_size=$(strlen \"$hello_content\")\n @@ t/t1006-cat-file.sh: test_expect_success \"setup\" '\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n -@@ t/t1006-cat-file.sh: tree_pretty_content=\"100644 blob $hello_sha1\thello\"\n - \n - run_tests 'tree' $tree_sha1 $tree_size \"\" \"$tree_pretty_content\"\n - \n -+test_expect_success PIPE '--batch-command --buffer with flush for tree info' '\n -+ run_buffer_test_flush tree $tree_sha1 $tree_size\n -+'\n -+\n -+test_expect_success PIPE '--batch-command --buffer without flush for tree info' '\n -+ run_buffer_test_no_flush tree $tree_sha1 $tree_size false\n -+'\n -+\n - commit_message=\"Initial commit\"\n - commit_sha1=$(echo_without_newline \"$commit_message\" | git commit-tree $tree_sha1)\n - commit_size=$(($(test_oid hexsz) + 137))\n -@@ t/t1006-cat-file.sh: $commit_message\"\n - \n - run_tests 'commit' $commit_sha1 $commit_size \"$commit_content\" \"$commit_content\" 1\n - \n -+test_expect_success PIPE '--batch-command --buffer with flush for commit info' '\n -+ run_buffer_test_flush commit $commit_sha1 $commit_size\n -+'\n -+\n -+test_expect_success PIPE '--batch-command --buffer without flush for commit info' '\n -+ run_buffer_test_no_flush commit $commit_sha1 $commit_size false\n -+'\n -+\n - tag_header_without_timestamp=\"object $hello_sha1\n - type blob\n - tag hellotag\n -@@ t/t1006-cat-file.sh: tag_size=$(strlen \"$tag_content\")\n - \n - run_tests 'tag' $tag_sha1 $tag_size \"$tag_content\" \"$tag_content\" 1\n - \n -+test_expect_success PIPE '--batch-command --buffer with flush for tag info' '\n -+ run_buffer_test_flush tag $tag_sha1 $tag_size\n -+'\n -+\n -+test_expect_success PIPE '--batch-command --buffer without flush for tag info' '\n -+ run_buffer_test_no_flush tag $tag_sha1 $tag_size false\n -+'\n -+\n - test_expect_success \\\n +@@ t/t1006-cat-file.sh: test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n -+batch_command_info_input=\"info $hello_sha1\n -+info $tree_sha1\n -+info $commit_sha1\n -+info $tag_sha1\n -+info deadbeef\n -+flush\n -+\"\n -+\n -+batch_command_info_output=\"$hello_sha1 blob $hello_size\n -+$tree_sha1 tree $tree_size\n -+$commit_sha1 commit $commit_size\n -+$tag_sha1 tag $tag_size\n -+deadbeef missing\"\n -+\n -+test_expect_success \"--batch-command with multiple info calls gives correct format\" '\n -+\ttest \"$batch_command_info_output\" = \"$(echo_without_newline \\\n -+\t\"$batch_command_info_input\" | git cat-file --batch-command --buffer)\"\n -+'\n -+\n -+batch_command_contents_input=\"contents $hello_sha1\n -+contents $commit_sha1\n -+contents $tag_sha1\n -+contents deadbeef\n -+flush\n -+\"\n -+\n -+batch_command_output=\"$hello_sha1 blob $hello_size\n -+$hello_content\n -+$commit_sha1 commit $commit_size\n -+$commit_content\n -+$tag_sha1 tag $tag_size\n -+$tag_content\n -+deadbeef missing\"\n -+\n -+test_expect_success \"--batch-command with multiple contents calls gives correct format\" '\n -+\ttest \"$(maybe_remove_timestamp \"$batch_command_output\" 1)\" = \\\n -+\t\"$(maybe_remove_timestamp \"$(echo_without_newline \"$batch_command_contents_input\" | git cat-file --batch-command)\" 1)\"\n ++test_expect_success '--batch-command with multiple info calls gives correct format' '\n ++\tcat >expect <<-EOF &&\n ++\t$hello_sha1 blob $hello_size\n ++\t$tree_sha1 tree $tree_size\n ++\t$commit_sha1 commit $commit_size\n ++\t$tag_sha1 tag $tag_size\n ++\tdeadbeef missing\n ++\tEOF\n ++\n ++\ttest_write_lines \"info $hello_sha1\"\\\n ++\t\"info $tree_sha1\"\\\n ++\t\"info $commit_sha1\"\\\n ++\t\"info $tag_sha1\"\\\n ++\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n ++\ttest_cmp expect actual\n +'\n +\n -+batch_command_mixed_input=\"info $hello_sha1\n -+contents $hello_sha1\n -+info $commit_sha1\n -+contents $commit_sha1\n -+info $tag_sha1\n -+contents $tag_sha1\n -+contents deadbeef\n -+flush\n -+\"\n -+\n -+batch_command_mixed_output=\"$hello_sha1 blob $hello_size\n -+$hello_sha1 blob $hello_size\n -+$hello_content\n -+$commit_sha1 commit $commit_size\n -+$commit_sha1 commit $commit_size\n -+$commit_content\n -+$tag_sha1 tag $tag_size\n -+$tag_sha1 tag $tag_size\n -+$tag_content\n -+deadbeef missing\"\n -+\n -+test_expect_success \"--batch-command with mixed calls gives correct format\" '\n -+\ttest \"$(maybe_remove_timestamp \"$batch_command_mixed_output\" 1)\" = \\\n -+\t\"$(maybe_remove_timestamp \"$(echo_without_newline \\\n -+\t\"$batch_command_mixed_input\" | git cat-file --batch-command --buffer)\" 1)\"\n ++test_expect_success '--batch-command with multiple command calls gives correct format' '\n ++\tcat >expect <<-EOF &&\n ++\t$hello_sha1 blob $hello_size\n ++\t$hello_content\n ++\t$commit_sha1 commit $commit_size\n ++\t$commit_content\n ++\t$tag_sha1 tag $tag_size\n ++\t$tag_content\n ++\tdeadbeef missing\n ++\tEOF\n ++\n ++\tmaybe_remove_timestamp \"$(cat expect)\" 1 >expect &&\n ++\tmaybe_remove_timestamp \"$(test_write_lines \"contents $hello_sha1\"\\\n ++\t\"contents $commit_sha1\"\\\n ++\t\"contents $tag_sha1\"\\\n ++\t\"contents deadbeef\"\\\n ++\t\"flush\" | git cat-file --batch-command --buffer)\" 1 >actual &&\n ++\ttest_cmp expect actual\n +'\n +\n test_expect_success 'setup blobs which are likely to delta' '\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch-all-objects --batch-c\n +test_expect_success 'batch-command empty command' '\n +\techo \"\" >cmd &&\n +\ttest_expect_code 128 git cat-file --batch-command err &&\n -+\tgrep -E \"^fatal:.*empty command in input.*\" err\n ++\tgrep \"^fatal:.*empty command in input.*\" err\n +'\n +\n +test_expect_success 'batch-command whitespace before command' '\n +\techo \" info deadbeef\" >cmd &&\n +\ttest_expect_code 128 git cat-file --batch-command err &&\n -+\tgrep -E \"^fatal:.*whitespace before command.*\" err\n ++\tgrep \"^fatal:.*whitespace before command.*\" err\n +'\n +\n +test_expect_success 'batch-command unknown command' '\n +\techo unknown_command >cmd &&\n +\ttest_expect_code 128 git cat-file --batch-command err &&\n -+\tgrep -E \"^fatal:.*unknown command.*\" err\n ++\tgrep \"^fatal:.*unknown command.*\" err\n ++'\n ++\n ++test_expect_success 'batch-command missing arguments' '\n ++\techo \"info\" >cmd &&\n ++\ttest_expect_code 128 git cat-file --batch-command err &&\n ++\tgrep \"^fatal:.*info requires arguments.*\" err\n +'\n +\n +test_expect_success 'batch-command flush with arguments' '\n +\techo \"flush arg\" >cmd &&\n +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n -+\tgrep -E \"^fatal:.*flush takes no arguments.*\" err\n ++\tgrep \"^fatal:.*flush takes no arguments.*\" err\n +'\n +\n +test_expect_success 'batch-command flush without --buffer' '\n +\techo \"flush arg\" >cmd &&\n +\ttest_expect_code 128 git cat-file --batch-command err &&\n -+\tgrep -E \"^fatal:.*flush is only for --buffer mode.*\" err\n ++\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n +'\n \n test_done\n\n-- \ngitgitgadget\n"},{"id":"448245","messageId":"5e0d1161df46b0cb9417db1cd2a82dd8325db9eb.1644609683.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v5.git.git.1644609683.gitgitgadget@gmail.com","subject":"[PATCH v5 2/3] cat-file: introduce batch_mode enum to replace print_contents","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:01:22Z","receivedAt":"2022-02-11T20:01:38Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nThe next patch introduces a new --batch-command flag. Including --batch\nand --batch-check, we will have a total of three batch modes. print_contents\nis the only boolean on the batch_options sturct used to distinguish\nbetween the different modes. This makes the code harder to read.\n\nTo reduce potential confusion, replace print_contents with an enum to\nhelp readability and clarity.\n\nHelped-by: Junio C Hamano \nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..5e38af82af1 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,10 +17,15 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n+enum batch_mode {\n+\tBATCH_MODE_CONTENTS,\n+\tBATCH_MODE_INFO,\n+};\n+\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tint print_contents;\n+\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -635,7 +640,14 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\n+\tif (!strcmp(opt->long_name, \"batch\"))\n+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n+\telse if (!strcmp(opt->long_name, \"batch-check\"))\n+\t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse\n+\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n+\n \tbo->format = arg;\n \n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"448246","messageId":"ad66d1f3e2bc84498b3168416ae1ccdd41f8a7ce.1644609683.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v5.git.git.1644609683.gitgitgadget@gmail.com","subject":"[PATCH v5 3/3] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:01:23Z","receivedAt":"2022-02-11T20:01:48Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] LF\n [arg1] [arg2] LF\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush LF\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n--batch-command also will not allow (B) to flush to stdout until a flush\nis received.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents LF\ninfo LF\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes a argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\nflush LF\ninfo LF\nflush LF\n\nWhen used without --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\ninfo LF\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 41 +++++++-\n builtin/cat-file.c | 133 ++++++++++++++++++++++++\n t/t1006-cat-file.sh | 178 ++++++++++++++++++++++++++++++++-\n 3 files changed, 347 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..e8da704477d 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,32 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+\tEnter a command mode that reads commands and arguments from stdin. May\n+\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n+\tcase of `--textconv` or `--filters`, the input lines also need to specify\n+\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n+\tfor details.\n++\n+`--batch-command` recognizes the following commands:\n++\n+--\n+contents ``::\n+\tPrint object contents for object reference ``. This corresponds to\n+\tthe output of `--batch`.\n+\n+info ``::\n+\tPrint object info for object reference ``. This corresponds to the\n+\toutput of `--batch-check`.\n+\n+flush::\n+\tUsed with `--buffer` to execute all preceding commands that were issued\n+\tsince the beginning or since the last flush was issued. When `--buffer`\n+\tis used, no output will come until a `flush` is issued. When `--buffer`\n+\tis not used, commands are flushed each time without issuing `flush`.\n+--\n++\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\n@@ -110,7 +136,7 @@ OPTIONS\n \tthat a process can interactively read and write from\n \t`cat-file`. With this option, the output uses normal stdio\n \tbuffering; this is much more efficient when invoking\n-\t`--batch-check` on a large number of objects.\n+\t`--batch-check` or `--batch-command` on a large number of objects.\n \n --unordered::\n \tWhen `--batch-all-objects` is in use, visit objects in an\n@@ -202,6 +228,13 @@ from stdin, one per line, and print information about them. By default,\n the whole line is considered as an object, as if it were fed to\n linkgit:git-rev-parse[1].\n \n+When `--batch-command` is given, `cat-file` will read commands from stdin,\n+one per line, and print information based on the command given. With\n+`--batch-command`, the `info` command followed by an object will print\n+information about the object the same way `--batch-check` would, and the\n+`contents` command followed by an object prints contents in the same way\n+`--batch` would.\n+\n You can specify the information shown for each object by using a custom\n ``. The `` is copied literally to stdout for each\n object, with placeholders of the form `%(atom)` expanded, followed by a\n@@ -237,9 +270,9 @@ newline. The available atoms are:\n If no format is specified, the default format is `%(objectname)\n %(objecttype) %(objectsize)`.\n \n-If `--batch` is specified, the object information is followed by the\n-object contents (consisting of `%(objectsize)` bytes), followed by a\n-newline.\n+If `--batch` is specified, or if `--batch-command` is used with the `contents`\n+command, the object information is followed by the object contents (consisting\n+of `%(objectsize)` bytes), followed by a newline.\n \n For example, `--batch` without a custom format would produce:\n \ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5e38af82af1..6d54a0eb38d 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -20,6 +20,7 @@\n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n \tBATCH_MODE_INFO,\n+\tBATCH_MODE_QUEUE_AND_DISPATCH,\n };\n \n struct batch_options {\n@@ -513,6 +514,127 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tchar *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_CONTENTS;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_INFO;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void dispatch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmd,\n+\t\tint nr)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < nr; i++){\n+\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\t\tfree(cmd[i].line);\n+\t}\n+\n+\tfflush(stdout);\n+}\n+\n+static const struct parse_cmd {\n+\tconst char *prefix;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *queued_cmd = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p = NULL, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n+\t\t\tif (!opt->buffer_output)\n+\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n+\t\t\tif (*cmd_end)\n+\t\t\t\tdie(_(\"flush takes no arguments\"));\n+\n+\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\t\t\tnr = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args) {\n+\t\t\t\tif (*cmd_end != ' ')\n+\t\t\t\t\tdie(_(\"%s requires arguments\"),\n+\t\t\t\t\t commands[i].prefix);\n+\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\t} else if (*cmd_end) {\n+\t\t\t\tdie(_(\"%s takes no arguments\"),\n+\t\t\t\t commands[i].prefix);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n+\t\tcall.fn = cmd->fn;\n+\t\tcall.line = xstrdup_or_null(p);\n+\t\tqueued_cmd[nr++] = call;\n+\t}\n+\n+\tif (opt->buffer_output && nr)\n+\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\n+\tfree(queued_cmd);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -595,6 +717,10 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -613,6 +739,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -645,6 +772,8 @@ static int batch_option_callback(const struct option *opt,\n \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n \telse if (!strcmp(opt->long_name, \"batch-check\"))\n \t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse if (!strcmp(opt->long_name, \"batch-command\"))\n+\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n \telse\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \n@@ -695,6 +824,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..a501dbcc39b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -177,6 +177,24 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ for opt in --buffer --no-buffer\n+ do\n+\ttest -z \"$content\" ||\n+\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n+\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n+\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_write_lines \"info $sha1\" \\\n+\t\t| git cat-file --batch-command $opt >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+ done\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -213,6 +231,84 @@ $content\"\n '\n }\n \n+run_buffer_test_flush () {\n+\ttype=$1\n+\tsha1=$2\n+\tsize=$3\n+\n+\trm -f input output &&\n+\tmkfifo input &&\n+\ttest_when_finished 'rm input'\n+\tmkfifo output &&\n+\texec 9<>output &&\n+\ttest_when_finished 'rm output; exec 9<&-'\n+\t(\n+\t\t# TODO - Ideally we'd pipe the output of cat-file\n+\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n+\t\t# would consume all the available\n+\t\t# output. Unfortunately we cannot do this as we cannot\n+\t\t# control when sed flushes its output. We could write\n+\t\t# a test helper in C that appended a '\\' to the end of\n+\t\t# each line and flushes its output after every line.\n+\t\tgit cat-file --buffer --batch-command err &\n+\t\techo $! &&\n+\t\twait $!\n+\t\techo $?\n+\t) >&9 &\n+\tsh_pid=$! &&\n+\tread cat_file_pid <&9 &&\n+\ttest_when_finished \"kill $cat_file_pid\n+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n+\t(\n+\t\ttest_write_lines \"info $sha1\" flush \"info $sha1\" &&\n+\t\t# TODO - consume all available input, not just one\n+\t\t# line (see above).\n+\t\tread actual <&9 &&\n+\t\techo \"$actual\" >actual &&\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_cmp expect actual\n+\t) >input &&\n+\t# check output is flushed on exit\n+\tread actual <&9 &&\n+\techo \"$actual\" >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_must_be_empty err &&\n+\tread status <&9 &&\n+\ttest \"$status\" -eq 0\n+}\n+\n+run_buffer_test_no_flush () {\n+\ttype=$1\n+\tsha1=$2\n+\tsize=$3\n+\n+\ttouch output &&\n+\ttest_when_finished 'rm output'\n+\tmkfifo input &&\n+\ttest_when_finished 'rm input'\n+\tmkfifo pid &&\n+\texec 9<>pid &&\n+\ttest_when_finished 'rm pid; exec 9<&-'\n+\t(\n+\t\tgit cat-file --buffer --batch-command >output &\n+\t\techo $! &&\n+\t\twait $!\n+\t\techo $?\n+\t) >&9 &\n+\tsh_pid=$! &&\n+\tread cat_file_pid <&9 &&\n+\ttest_when_finished \"kill $cat_file_pid\n+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n+\t(\n+\t\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n+\t\tkill $cat_file_pid &&\n+\t\tread status <&9 &&\n+\t\ttest \"$status\" -ne 0 &&\n+\t\ttest_must_be_empty output\n+\t) >input\n+}\n+\n+\n hello_content=\"Hello World\"\n hello_size=$(strlen \"$hello_content\")\n hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n@@ -224,6 +320,14 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n+ run_buffer_test_flush blob $hello_sha1 $hello_size\n+'\n+\n+test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n+ run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -267,7 +371,7 @@ test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -373,6 +477,43 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+test_expect_success '--batch-command with multiple info calls gives correct format' '\n+\tcat >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$tree_sha1 tree $tree_size\n+\t$commit_sha1 commit $commit_size\n+\t$tag_sha1 tag $tag_size\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"info $hello_sha1\"\\\n+\t\"info $tree_sha1\"\\\n+\t\"info $commit_sha1\"\\\n+\t\"info $tag_sha1\"\\\n+\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command with multiple command calls gives correct format' '\n+\tcat >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$hello_content\n+\t$commit_sha1 commit $commit_size\n+\t$commit_content\n+\t$tag_sha1 tag $tag_size\n+\t$tag_content\n+\tdeadbeef missing\n+\tEOF\n+\n+\tmaybe_remove_timestamp \"$(cat expect)\" 1 >expect &&\n+\tmaybe_remove_timestamp \"$(test_write_lines \"contents $hello_sha1\"\\\n+\t\"contents $commit_sha1\"\\\n+\t\"contents $tag_sha1\"\\\n+\t\"contents deadbeef\"\\\n+\t\"flush\" | git cat-file --batch-command --buffer)\" 1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -963,5 +1104,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command missing arguments' '\n+\techo \"info\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*info requires arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"448247","messageId":"xmqqpmnt9ngx.fsf@gitster.g","threadId":"57365","inReplyTo":"9B08BF05-1C35-4C4F-9C76-CC34A8E7D0F0@gmail.com","subject":"Re: [PATCH v4 3/3] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-11T20:07:10Z","receivedAt":"2022-02-11T20:07:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Cai writes:\n\n> Let me see if I understand you. What I'm hearing is that it's hard to test a git\n> processes (A) that read/write from/to pipes without knowing exactly how (A) will\n> behave. By necessity, the test logic will have embedded some logic in it that\n> assumes certain behavior from (A), which might or might not be the case.\n>\n> This can lead to a hanging test if, say, it is waiting around for (A) to output\n> data when due to a bug in the code, it never does. Did I get that right?\n\nExactly. And we've seen such tests that are designed to hang, when\nthey detect bugs, which made us very unhappy and we fixed them not\nto hang but reliably fail. Otherwise, such tests weren't very\nuseful in unattended CI environment, which we do not want to wait\nfor 3 hours to timeout and leave later steps in the same script\nuntested.\n\nThanks.\n"},{"id":"448286","messageId":"06C925DC-DAAC-4BA6-8CFA-E4ACF2039BB6@gmail.com","threadId":"57365","inReplyTo":"xmqqpmnt9ngx.fsf@gitster.g","subject":"Re: [PATCH v4 3/3] cat-file: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-11T21:30:58Z","receivedAt":"2022-02-11T21:31:13Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Junio\n\nOn 11 Feb 2022, at 15:07, Junio C Hamano wrote:\n\n> John Cai writes:\n>\n>> Let me see if I understand you. What I'm hearing is that it's hard to test a git\n>> processes (A) that read/write from/to pipes without knowing exactly how (A) will\n>> behave. By necessity, the test logic will have embedded some logic in it that\n>> assumes certain behavior from (A), which might or might not be the case.\n>>\n>> This can lead to a hanging test if, say, it is waiting around for (A) to output\n>> data when due to a bug in the code, it never does. Did I get that right?\n>\n> Exactly. And we've seen such tests that are designed to hang, when\n> they detect bugs, which made us very unhappy and we fixed them not\n> to hang but reliably fail. Otherwise, such tests weren't very\n> useful in unattended CI environment, which we do not want to wait\n> for 3 hours to timeout and leave later steps in the same script\n> untested.\n\nThat makes sense. Do you have an example of one of these tests? I'd like to see\nhow it was converted from a test that hung to a test that failed reliably. As\nI'm thinking about converting run_buffer_test_flush() and run_buffer_test_no_flush()\ninto tests that fail rather than hang, I'm having a hard time avoiding the\npattern of A writes to B and waits for B to respond.\n\n>\n> Thanks.\n"},{"id":"448355","messageId":"3932558a-ce38-d94b-b974-65510688d8c0@gmail.com","threadId":"57365","inReplyTo":"ad66d1f3e2bc84498b3168416ae1ccdd41f8a7ce.1644609683.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 3/3] cat-file: add --batch-command mode","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-14T13:59:50Z","receivedAt":"2022-02-14T13:59:57Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John\n\nI've concentrated on the tests again, I think the flush tests still need \nsome work but the others are looking good\n\n >[...]> Documentation/git-cat-file.txt | 41 +++++++-\n> builtin/cat-file.c | 133 ++++++++++++++++++++++++\n> t/t1006-cat-file.sh | 178 ++++++++++++++++++++++++++++++++-\n> 3 files changed, 347 insertions(+), 5 deletions(-)\n> \n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> index bef76f4dd06..e8da704477d 100644\n> --- a/Documentation/git-cat-file.txt\n> +++ b/Documentation/git-cat-file.txt\n> @@ -96,6 +96,32 @@ OPTIONS\n> \tneed to specify the path, separated by whitespace. See the\n> \tsection `BATCH OUTPUT` below for details.\n> \n> +--batch-command::\n\n+--batch-command=::\n\nas we also take an optional format string\n\n> +\tEnter a command mode that reads commands and arguments from stdin. May\n> +\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n> +\tcase of `--textconv` or `--filters`, the input lines also need to specify\n> +\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n> +\tfor details.\n> ++\n> +`--batch-command` recognizes the following commands:\n> ++\n> +--\n> +contents ``::\n> +\tPrint object contents for object reference ``. This corresponds to\n> +\tthe output of `--batch`.\n> +\n> +info ``::\n> +\tPrint object info for object reference ``. This corresponds to the\n> +\toutput of `--batch-check`.\n> +\n> +flush::\n> +\tUsed with `--buffer` to execute all preceding commands that were issued\n> +\tsince the beginning or since the last flush was issued. When `--buffer`\n> +\tis used, no output will come until a `flush` is issued. When `--buffer`\n> +\tis not used, commands are flushed each time without issuing `flush`.\n> +--\n> ++\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 145eee11df9..a501dbcc39b 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -177,6 +177,24 @@ $content\"\n> \ttest_cmp expect actual\n> '\n> \n> + for opt in --buffer --no-buffer\n> + do\n> +\ttest -z \"$content\" ||\n> +\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n> +\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n> +\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n> +\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n> +\t\ttest_cmp expect actual\n> +\t'\n> +\n> +\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n> +\t\techo \"$sha1 $type $size\" >expect &&\n> +\t\ttest_write_lines \"info $sha1\" \\\n> +\t\t| git cat-file --batch-command $opt >actual &&\n> +\t\ttest_cmp expect actual\n> +\t'\n> + done\n> +\n> test_expect_success \"custom --batch-check format\" '\n> \techo \"$type $sha1\" >expect &&\n> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n> @@ -213,6 +231,84 @@ $content\"\n> '\n> }\n> \n> +run_buffer_test_flush () {\n> +\ttype=$1\n> +\tsha1=$2\n> +\tsize=$3\n> +\n> +\trm -f input output &&\n\nI think that this should not be needed with the addition of \n\"test_when_finished 'rm input output'\" in run_buffer_test_no_flush()\n\n> +\tmkfifo input &&\n> +\ttest_when_finished 'rm input'\n> +\tmkfifo output &&\n> +\texec 9<>output &&\n\nTo address the worries about this test hanging rather than failing if \nsomething goes wrong I wonder if we could do something like\n\n\t(\n\t\tsleep 10\n\t\techo \"error: timeout\" >&2\n\t\techo TIMEOUT >&9\n\t) &\n\twatchdog_pid=$! &&\n\ttest_when_finished 'kill $watchdog_pid; wait $watchdog_pid'\n\nThat should unblock any reads from fd 9 if the test hangs\n\n> +\ttest_when_finished 'rm output; exec 9<&-'\n> +\t(\n> +\t\t# TODO - Ideally we'd pipe the output of cat-file\n> +\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n> +\t\t# would consume all the available\n> +\t\t# output. Unfortunately we cannot do this as we cannot\n> +\t\t# control when sed flushes its output. We could write\n> +\t\t# a test helper in C that appended a '\\' to the end of\n> +\t\t# each line and flushes its output after every line.\n> +\t\tgit cat-file --buffer --batch-command err &\n> +\t\techo $! &&\n> +\t\twait $!\n> +\t\techo $?\n> +\t) >&9 &\n> +\tsh_pid=$! &&\n> +\tread cat_file_pid <&9 &&\n> +\ttest_when_finished \"kill $cat_file_pid\n> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n> +\t(\n> +\t\ttest_write_lines \"info $sha1\" flush \"info $sha1\" &&\n> +\t\t# TODO - consume all available input, not just one\n> +\t\t# line (see above).\n> +\t\tread actual <&9 &&\n> +\t\techo \"$actual\" >actual &&\n> +\t\techo \"$sha1 $type $size\" >expect &&\n> +\t\ttest_cmp expect actual\n> +\t) >input &&\n> +\t# check output is flushed on exit\n> +\tread actual <&9 &&\n> +\techo \"$actual\" >actual &&\n> +\ttest_cmp expect actual &&\n> +\ttest_must_be_empty err &&\n> +\tread status <&9 &&\n> +\ttest \"$status\" -eq 0\n> +}\n> +\n> +run_buffer_test_no_flush () {\n\nThis test reliably hangs for me when running with --stress\n\n> +\ttype=$1\n> +\tsha1=$2\n> +\tsize=$3\n> +\n> +\ttouch output &&\n\nIf output is missing at the end it means cat-file never ran which is an \nerror which we do not want to hide. This is because the subshell creates \noutput after opening input and before it executes cat-file below. As \ninput is a fifo the open will block until it is opened for writing by \nanother process and nothing wrote to it in V4 so I think that is why you \nsaw an error there.\n\n> +\ttest_when_finished 'rm output'\n> +\tmkfifo input &&\n> +\ttest_when_finished 'rm input'\n> +\tmkfifo pid &&\n> +\texec 9<>pid &&\n> +\ttest_when_finished 'rm pid; exec 9<&-'\n> +\t(\n> +\t\tgit cat-file --buffer --batch-command >output &\n> +\t\techo $! &&\n> +\t\twait $!\n> +\t\techo $?\n> +\t) >&9 &\n> +\tsh_pid=$! &&\n> +\tread cat_file_pid <&9 &&\n> +\ttest_when_finished \"kill $cat_file_pid\n> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n> +\t(\n> +\t\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n> +\t\tkill $cat_file_pid &&\n> +\t\tread status <&9 &&\n\nThis is where the test hangs. There seems to be a race (which I don't \nunderstand) where we're able to read the pid of cat-file but it is not \nkilled by the kill above (the subshell above is blocked on \"wait $!\"). \nAdding \"sleep 1\" before the kill above makes everything work but I'm not \nvery comfortable with it. I think we might be better taking a different \napproach and introducing an environment variable such as \nGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT which stops cat-file flushing its \noutput on exit and having a test along the lines of\n\ntest_write_lines \"info $sha1\" | GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 git \ncat-file --batch-command --buffer >output &&\ntest_must_be_empty_output\n\n\n\n> +\t\ttest \"$status\" -ne 0 &&\n> +\t\ttest_must_be_empty output\n> +\t) >input\n> +}\n> +\n> +\n> hello_content=\"Hello World\"\n> hello_size=$(strlen \"$hello_content\")\n> hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n> @@ -224,6 +320,14 @@ test_expect_success \"setup\" '\n> \n> run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n> \n> +test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n> + run_buffer_test_flush blob $hello_sha1 $hello_size\n> +'\n> +\n> +test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n> + run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n> +'\n> +\n> test_expect_success '--batch-check without %(rest) considers whole line' '\n> \techo \"$hello_sha1 blob $hello_size\" >expect &&\n> \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n> @@ -267,7 +371,7 @@ test_expect_success \\\n> \"Reach a blob from a tag pointing to it\" \\\n> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n> \n> -for batch in batch batch-check\n> +for batch in batch batch-check batch-command\n> do\n> for opt in t s e p\n> do\n> @@ -373,6 +477,43 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n> '\n> \n> +test_expect_success '--batch-command with multiple info calls gives correct format' '\n> +\tcat >expect <<-EOF &&\n> +\t$hello_sha1 blob $hello_size\n> +\t$tree_sha1 tree $tree_size\n> +\t$commit_sha1 commit $commit_size\n> +\t$tag_sha1 tag $tag_size\n> +\tdeadbeef missing\n> +\tEOF\n> +\n> +\ttest_write_lines \"info $hello_sha1\"\\\n> +\t\"info $tree_sha1\"\\\n> +\t\"info $commit_sha1\"\\\n> +\t\"info $tag_sha1\"\\\n> +\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success '--batch-command with multiple command calls gives correct format' '\n> +\tcat >expect <<-EOF &&\n> +\t$hello_sha1 blob $hello_size\n> +\t$hello_content\n> +\t$commit_sha1 commit $commit_size\n> +\t$commit_content\n> +\t$tag_sha1 tag $tag_size\n> +\t$tag_content\n> +\tdeadbeef missing\n> +\tEOF\n> +\n> +\tmaybe_remove_timestamp \"$(cat expect)\" 1 >expect &&\n> +\tmaybe_remove_timestamp \"$(test_write_lines \"contents $hello_sha1\"\\\n> +\t\"contents $commit_sha1\"\\\n> +\t\"contents $tag_sha1\"\\\n> +\t\"contents deadbeef\"\\\n> +\t\"flush\" | git cat-file --batch-command --buffer)\" 1 >actual &&\n> +\ttest_cmp expect actual\n\nIt is a shame that maybe_remove_timestamp does no read from stdin, this \ntest would look much nicer if it did. Apart from that these and the ones \nbelow are looking good\n\nBest Wishes\n\nPhillip\n\n> +'\n> +\n> test_expect_success 'setup blobs which are likely to delta' '\n> \ttest-tool genrandom foo 10240 >foo &&\n> \t{ cat foo && echo plus; } >foo-plus &&\n> @@ -963,5 +1104,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n> \techo \"$orig commit $orig_size\" >expect &&\n> \ttest_cmp expect actual\n> '\n> +test_expect_success 'batch-command empty command' '\n> +\techo \"\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*empty command in input.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command whitespace before command' '\n> +\techo \" info deadbeef\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*whitespace before command.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command unknown command' '\n> +\techo unknown_command >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*unknown command.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command missing arguments' '\n> +\techo \"info\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*info requires arguments.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush with arguments' '\n> +\techo \"flush arg\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n> +\tgrep \"^fatal:.*flush takes no arguments.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush without --buffer' '\n> +\techo \"flush arg\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n> +'\n> \n> test_done\n\n"},{"id":"448363","messageId":"3685CB9D-F0C6-4F18-A9D3-7F862C0043A2@gmail.com","threadId":"57365","inReplyTo":"3932558a-ce38-d94b-b974-65510688d8c0@gmail.com","subject":"Re: [PATCH v5 3/3] cat-file: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-14T16:19:37Z","receivedAt":"2022-02-14T16:19:46Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Phillip\n\nOn 14 Feb 2022, at 8:59, Phillip Wood wrote:\n\n> Hi John\n>\n> I've concentrated on the tests again, I think the flush tests still need some work but the others are looking good\n>\n>> [...]> Documentation/git-cat-file.txt | 41 +++++++-\n>> builtin/cat-file.c | 133 ++++++++++++++++++++++++\n>> t/t1006-cat-file.sh | 178 ++++++++++++++++++++++++++++++++-\n>> 3 files changed, 347 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n>> index bef76f4dd06..e8da704477d 100644\n>> --- a/Documentation/git-cat-file.txt\n>> +++ b/Documentation/git-cat-file.txt\n>> @@ -96,6 +96,32 @@ OPTIONS\n>> \tneed to specify the path, separated by whitespace. See the\n>> \tsection `BATCH OUTPUT` below for details.\n>> +--batch-command::\n>\n> +--batch-command=::\n>\n> as we also take an optional format string\n\ngood catch!\n\n>\n>> +\tEnter a command mode that reads commands and arguments from stdin. May\n>> +\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n>> +\tcase of `--textconv` or `--filters`, the input lines also need to specify\n>> +\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n>> +\tfor details.\n>> ++\n>> +`--batch-command` recognizes the following commands:\n>> ++\n>> +--\n>> +contents ``::\n>> +\tPrint object contents for object reference ``. This corresponds to\n>> +\tthe output of `--batch`.\n>> +\n>> +info ``::\n>> +\tPrint object info for object reference ``. This corresponds to the\n>> +\toutput of `--batch-check`.\n>> +\n>> +flush::\n>> +\tUsed with `--buffer` to execute all preceding commands that were issued\n>> +\tsince the beginning or since the last flush was issued. When `--buffer`\n>> +\tis used, no output will come until a `flush` is issued. When `--buffer`\n>> +\tis not used, commands are flushed each time without issuing `flush`.\n>> +--\n>> ++\n>> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n>> index 145eee11df9..a501dbcc39b 100755\n>> --- a/t/t1006-cat-file.sh\n>> +++ b/t/t1006-cat-file.sh\n>> @@ -177,6 +177,24 @@ $content\"\n>> \ttest_cmp expect actual\n>> '\n>> + for opt in --buffer --no-buffer\n>> + do\n>> +\ttest -z \"$content\" ||\n>> +\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n>> +\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n>> +\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n>> +\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t'\n>> +\n>> +\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n>> +\t\techo \"$sha1 $type $size\" >expect &&\n>> +\t\ttest_write_lines \"info $sha1\" \\\n>> +\t\t| git cat-file --batch-command $opt >actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t'\n>> + done\n>> +\n>> test_expect_success \"custom --batch-check format\" '\n>> \techo \"$type $sha1\" >expect &&\n>> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n>> @@ -213,6 +231,84 @@ $content\"\n>> '\n>> }\n>> +run_buffer_test_flush () {\n>> +\ttype=$1\n>> +\tsha1=$2\n>> +\tsize=$3\n>> +\n>> +\trm -f input output &&\n>\n> I think that this should not be needed with the addition of \"test_when_finished 'rm input output'\" in run_buffer_test_no_flush()\n>\n>> +\tmkfifo input &&\n>> +\ttest_when_finished 'rm input'\n>> +\tmkfifo output &&\n>> +\texec 9<>output &&\n>\n> To address the worries about this test hanging rather than failing if something goes wrong I wonder if we could do something like\n>\n> (\n> \tsleep 10\n> \techo \"error: timeout\" >&2\n> \techo TIMEOUT >&9\n> ) &\n> watchdog_pid=$! &&\n> test_when_finished 'kill $watchdog_pid; wait $watchdog_pid'\n>\n> That should unblock any reads from fd 9 if the test hangs\n>\n>> +\ttest_when_finished 'rm output; exec 9<&-'\n>> +\t(\n>> +\t\t# TODO - Ideally we'd pipe the output of cat-file\n>> +\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n>> +\t\t# would consume all the available\n>> +\t\t# output. Unfortunately we cannot do this as we cannot\n>> +\t\t# control when sed flushes its output. We could write\n>> +\t\t# a test helper in C that appended a '\\' to the end of\n>> +\t\t# each line and flushes its output after every line.\n>> +\t\tgit cat-file --buffer --batch-command err &\n>> +\t\techo $! &&\n>> +\t\twait $!\n>> +\t\techo $?\n>> +\t) >&9 &\n>> +\tsh_pid=$! &&\n>> +\tread cat_file_pid <&9 &&\n>> +\ttest_when_finished \"kill $cat_file_pid\n>> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n>> +\t(\n>> +\t\ttest_write_lines \"info $sha1\" flush \"info $sha1\" &&\n>> +\t\t# TODO - consume all available input, not just one\n>> +\t\t# line (see above).\n>> +\t\tread actual <&9 &&\n>> +\t\techo \"$actual\" >actual &&\n>> +\t\techo \"$sha1 $type $size\" >expect &&\n>> +\t\ttest_cmp expect actual\n>> +\t) >input &&\n>> +\t# check output is flushed on exit\n>> +\tread actual <&9 &&\n>> +\techo \"$actual\" >actual &&\n>> +\ttest_cmp expect actual &&\n>> +\ttest_must_be_empty err &&\n>> +\tread status <&9 &&\n>> +\ttest \"$status\" -eq 0\n>> +}\n>> +\n>> +run_buffer_test_no_flush () {\n>\n> This test reliably hangs for me when running with --stress\n>\n>> +\ttype=$1\n>> +\tsha1=$2\n>> +\tsize=$3\n>> +\n>> +\ttouch output &&\n>\n> If output is missing at the end it means cat-file never ran which is an error which we do not want to hide. This is because the subshell creates output after opening input and before it executes cat-file below. As input is a fifo the open will block until it is opened for writing by another process and nothing wrote to it in V4 so I think that is why you saw an error there.\n>\n>> +\ttest_when_finished 'rm output'\n>> +\tmkfifo input &&\n>> +\ttest_when_finished 'rm input'\n>> +\tmkfifo pid &&\n>> +\texec 9<>pid &&\n>> +\ttest_when_finished 'rm pid; exec 9<&-'\n>> +\t(\n>> +\t\tgit cat-file --buffer --batch-command >output &\n>> +\t\techo $! &&\n>> +\t\twait $!\n>> +\t\techo $?\n>> +\t) >&9 &\n>> +\tsh_pid=$! &&\n>> +\tread cat_file_pid <&9 &&\n>> +\ttest_when_finished \"kill $cat_file_pid\n>> +\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n>> +\t(\n>> +\t\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n>> +\t\tkill $cat_file_pid &&\n>> +\t\tread status <&9 &&\n>\n> This is where the test hangs. There seems to be a race (which I don't understand) where we're able to read the pid of cat-file but it is not killed by the kill above (the subshell above is blocked on \"wait $!\"). Adding \"sleep 1\" before the kill above makes everything work but I'm not very comfortable with it. I think we might be better taking a different approach and introducing an environment variable such as GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT which stops cat-file flushing its output on exit and having a test along the lines of\n>\n> test_write_lines \"info $sha1\" | GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 git cat-file --batch-command --buffer >output &&\n> test_must_be_empty_output\n\nThanks for this suggestion! this would allow us to test flushing in a much more\nstraightforward way without having to open up fifo pipes. This addresses Junio's\nconcern in [1] about this test hanging in the future when there's a regression.\n\nAbove you suggested having a timeout with sleep, which I was considering as\nwell. However, I feel like using this env var is overall simpler and safer, so\nmaybe we can use this for both testing the case when we get a flush and when we\ndo not get a flush\n\n1. https://lore.kernel.org/git/xmqqpmnt9ngx.fsf@gitster.g/\n\n>\n>\n>\n>> +\t\ttest \"$status\" -ne 0 &&\n>> +\t\ttest_must_be_empty output\n>> +\t) >input\n>> +}\n>> +\n>> +\n>> hello_content=\"Hello World\"\n>> hello_size=$(strlen \"$hello_content\")\n>> hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n>> @@ -224,6 +320,14 @@ test_expect_success \"setup\" '\n>> run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n>> +test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n>> + run_buffer_test_flush blob $hello_sha1 $hello_size\n>> +'\n>> +\n>> +test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n>> + run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n>> +'\n>> +\n>> test_expect_success '--batch-check without %(rest) considers whole line' '\n>> \techo \"$hello_sha1 blob $hello_size\" >expect &&\n>> \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n>> @@ -267,7 +371,7 @@ test_expect_success \\\n>> \"Reach a blob from a tag pointing to it\" \\\n>> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n>> -for batch in batch batch-check\n>> +for batch in batch batch-check batch-command\n>> do\n>> for opt in t s e p\n>> do\n>> @@ -373,6 +477,43 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n>> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n>> '\n>> +test_expect_success '--batch-command with multiple info calls gives correct format' '\n>> +\tcat >expect <<-EOF &&\n>> +\t$hello_sha1 blob $hello_size\n>> +\t$tree_sha1 tree $tree_size\n>> +\t$commit_sha1 commit $commit_size\n>> +\t$tag_sha1 tag $tag_size\n>> +\tdeadbeef missing\n>> +\tEOF\n>> +\n>> +\ttest_write_lines \"info $hello_sha1\"\\\n>> +\t\"info $tree_sha1\"\\\n>> +\t\"info $commit_sha1\"\\\n>> +\t\"info $tag_sha1\"\\\n>> +\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n>> +\ttest_cmp expect actual\n>> +'\n>> +\n>> +test_expect_success '--batch-command with multiple command calls gives correct format' '\n>> +\tcat >expect <<-EOF &&\n>> +\t$hello_sha1 blob $hello_size\n>> +\t$hello_content\n>> +\t$commit_sha1 commit $commit_size\n>> +\t$commit_content\n>> +\t$tag_sha1 tag $tag_size\n>> +\t$tag_content\n>> +\tdeadbeef missing\n>> +\tEOF\n>> +\n>> +\tmaybe_remove_timestamp \"$(cat expect)\" 1 >expect &&\n>> +\tmaybe_remove_timestamp \"$(test_write_lines \"contents $hello_sha1\"\\\n>> +\t\"contents $commit_sha1\"\\\n>> +\t\"contents $tag_sha1\"\\\n>> +\t\"contents deadbeef\"\\\n>> +\t\"flush\" | git cat-file --batch-command --buffer)\" 1 >actual &&\n>> +\ttest_cmp expect actual\n>\n> It is a shame that maybe_remove_timestamp does no read from stdin, this test would look much nicer if it did. Apart from that these and the ones below are looking good\n\nGood point. I'll see if I can adjust this in the next version.\n\n>\n> Best Wishes\n>\n> Phillip\n>\n>> +'\n>> +\n>> test_expect_success 'setup blobs which are likely to delta' '\n>> \ttest-tool genrandom foo 10240 >foo &&\n>> \t{ cat foo && echo plus; } >foo-plus &&\n>> @@ -963,5 +1104,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n>> \techo \"$orig commit $orig_size\" >expect &&\n>> \ttest_cmp expect actual\n>> '\n>> +test_expect_success 'batch-command empty command' '\n>> +\techo \"\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*empty command in input.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command whitespace before command' '\n>> +\techo \" info deadbeef\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*whitespace before command.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command unknown command' '\n>> +\techo unknown_command >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*unknown command.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command missing arguments' '\n>> +\techo \"info\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*info requires arguments.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command flush with arguments' '\n>> +\techo \"flush arg\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n>> +\tgrep \"^fatal:.*flush takes no arguments.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command flush without --buffer' '\n>> +\techo \"flush arg\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n>> +'\n>> test_done\n"},{"id":"448378","messageId":"pull.1212.v6.git.git.1644862988.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v5.git.git.1644609683.gitgitgadget@gmail.com","subject":"[PATCH v6 0/4] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-14T18:23:04Z","receivedAt":"2022-02-14T18:23:14Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has four parts:\n\n 1. preparation patch to rename a variable\n 2. adding an enum to keep track of batch modes\n 3. add a remove_timestamp() helper that takes stdin and removes timestamps\n 4. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v5\n\n * replaced flush tests that used fifo pipes to using a GIT_TEST_ env\n variable to control whether or not --batch-command flushes on exit.\n * added remove_timestamp helper in tests.\n * added documentation to show format can be used with --batch-command\n\nChanges since v4\n\n * added Phillip's suggested test for testing flush. This should have\n addressed the flaky test that was hanging. I tested it on my side and\n wasn't able to reproduce the deadlock.\n * plugged some holes in the logic that parsed the command and arguments,\n thanks to Eric's feedback\n * fixed verbiage in commit messages per Christian's feedback\n * clarified places in documentation that should mention --batch-command per\n Eric's feedback\n\nChanges since v3 (thanks to Junio's feedback):\n\n * added cascading logic in batch_options_callback()\n * free memory for queued call input lines\n * do not throw error when flushing an empty queue\n * renamed cmds array to singular queued_cmd\n * fixed flaky test that failed --stress\n\nChanges since v2:\n\n * added enum to keep track of which batch mode we are in (thanks to Junio's\n feedback)\n * fixed array allocation logic (thanks to Junio's feedback)\n * added code to flush commands when --batch-commands receives an EOF and\n exits (thanks to Phillip's feedback)\n * fixed docs formatting (thanks to Jonathan's feedback)\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (4):\n cat-file: rename cmdmode to transform_mode\n cat-file: introduce batch_mode enum to replace print_contents\n cat-file: add remove_timestamp helper\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 42 +++++++-\n builtin/cat-file.c | 169 ++++++++++++++++++++++++++++++---\n t/README | 3 +\n t/t1006-cat-file.sh | 122 ++++++++++++++++++++++--\n 4 files changed, 315 insertions(+), 21 deletions(-)\n\n\nbase-commit: 38062e73e009f27ea192d50481fcb5e7b0e9d6eb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v6\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v5:\n\n 1: fa6294387ab = 1: fa6294387ab cat-file: rename cmdmode to transform_mode\n 2: 5e0d1161df4 ! 2: 1a038097bfc cat-file: introduce batch_mode enum to replace print_contents\n @@ Metadata\n ## Commit message ##\n cat-file: introduce batch_mode enum to replace print_contents\n \n - The next patch introduces a new --batch-command flag. Including --batch\n + A future patch introduces a new --batch-command flag. Including --batch\n and --batch-check, we will have a total of three batch modes. print_contents\n is the only boolean on the batch_options sturct used to distinguish\n between the different modes. This makes the code harder to read.\n -: ----------- > 3: 486ee847816 cat-file: add remove_timestamp helper\n 3: ad66d1f3e2b ! 4: a6dd5d72fce cat-file: add --batch-command mode\n @@ Documentation/git-cat-file.txt: OPTIONS\n \tsection `BATCH OUTPUT` below for details.\n \n +--batch-command::\n ++--batch-command=::\n +\tEnter a command mode that reads commands and arguments from stdin. May\n +\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n +\tcase of `--textconv` or `--filters`, the input lines also need to specify\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\tqueued_cmd[nr++] = call;\n +\t}\n +\n -+\tif (opt->buffer_output && nr)\n ++\tif (opt->buffer_output &&\n ++\t nr &&\n ++\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n +\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n +\n +\tfree(queued_cmd);\n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *pr\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\n \n + ## t/README ##\n +@@ t/README: a test and then fails then the whole test run will abort. This can help to make\n + sure the expected tests are executed and not silently skipped when their\n + dependency breaks or is simply not present in a new environment.\n + \n ++GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file\n ++--batch-command from flushing to output on exit.\n ++\n + Naming Tests\n + ------------\n + \n +\n ## t/t1006-cat-file.sh ##\n @@ t/t1006-cat-file.sh: $content\"\n \ttest_cmp expect actual\n @@ t/t1006-cat-file.sh: $content\"\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n -@@ t/t1006-cat-file.sh: $content\"\n - '\n - }\n - \n -+run_buffer_test_flush () {\n -+\ttype=$1\n -+\tsha1=$2\n -+\tsize=$3\n -+\n -+\trm -f input output &&\n -+\tmkfifo input &&\n -+\ttest_when_finished 'rm input'\n -+\tmkfifo output &&\n -+\texec 9<>output &&\n -+\ttest_when_finished 'rm output; exec 9<&-'\n -+\t(\n -+\t\t# TODO - Ideally we'd pipe the output of cat-file\n -+\t\t# through \"sed s'/$/\\\\/'\" to make sure that that read\n -+\t\t# would consume all the available\n -+\t\t# output. Unfortunately we cannot do this as we cannot\n -+\t\t# control when sed flushes its output. We could write\n -+\t\t# a test helper in C that appended a '\\' to the end of\n -+\t\t# each line and flushes its output after every line.\n -+\t\tgit cat-file --buffer --batch-command err &\n -+\t\techo $! &&\n -+\t\twait $!\n -+\t\techo $?\n -+\t) >&9 &\n -+\tsh_pid=$! &&\n -+\tread cat_file_pid <&9 &&\n -+\ttest_when_finished \"kill $cat_file_pid\n -+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n -+\t(\n -+\t\ttest_write_lines \"info $sha1\" flush \"info $sha1\" &&\n -+\t\t# TODO - consume all available input, not just one\n -+\t\t# line (see above).\n -+\t\tread actual <&9 &&\n -+\t\techo \"$actual\" >actual &&\n -+\t\techo \"$sha1 $type $size\" >expect &&\n -+\t\ttest_cmp expect actual\n -+\t) >input &&\n -+\t# check output is flushed on exit\n -+\tread actual <&9 &&\n -+\techo \"$actual\" >actual &&\n -+\ttest_cmp expect actual &&\n -+\ttest_must_be_empty err &&\n -+\tread status <&9 &&\n -+\ttest \"$status\" -eq 0\n -+}\n -+\n -+run_buffer_test_no_flush () {\n -+\ttype=$1\n -+\tsha1=$2\n -+\tsize=$3\n -+\n -+\ttouch output &&\n -+\ttest_when_finished 'rm output'\n -+\tmkfifo input &&\n -+\ttest_when_finished 'rm input'\n -+\tmkfifo pid &&\n -+\texec 9<>pid &&\n -+\ttest_when_finished 'rm pid; exec 9<&-'\n -+\t(\n -+\t\tgit cat-file --buffer --batch-command >output &\n -+\t\techo $! &&\n -+\t\twait $!\n -+\t\techo $?\n -+\t) >&9 &\n -+\tsh_pid=$! &&\n -+\tread cat_file_pid <&9 &&\n -+\ttest_when_finished \"kill $cat_file_pid\n -+\t\t\t kill $sh_pid; wait $sh_pid; :\" &&\n -+\t(\n -+\t\ttest_write_lines \"info $sha1\" \"info $sha1\" &&\n -+\t\tkill $cat_file_pid &&\n -+\t\tread status <&9 &&\n -+\t\ttest \"$status\" -ne 0 &&\n -+\t\ttest_must_be_empty output\n -+\t) >input\n -+}\n -+\n -+\n - hello_content=\"Hello World\"\n - hello_size=$(strlen \"$hello_content\")\n - hello_sha1=$(echo_without_newline \"$hello_content\" | git hash-object --stdin)\n @@ t/t1006-cat-file.sh: test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n -+test_expect_success PIPE '--batch-command --buffer with flush for blob info' '\n -+ run_buffer_test_flush blob $hello_sha1 $hello_size\n ++test_expect_success '--batch-command --buffer with flush for blob info' '\n ++\techo \"$hello_sha1 blob $hello_size\" >expect &&\n ++\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n ++\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n ++\tgit cat-file --batch-command --buffer >actual &&\n ++\ttest_cmp expect actual\n +'\n +\n -+test_expect_success PIPE '--batch-command --buffer without flush for blob info' '\n -+ run_buffer_test_no_flush blob $hello_sha1 $hello_size false\n ++test_expect_success '--batch-command --buffer without flush for blob info' '\n ++\ttouch output &&\n ++\ttest_write_lines \"info $hello_sha1\" | \\\n ++\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n ++\tgit cat-file --batch-command --buffer >>output &&\n ++\ttest_must_be_empty output\n +'\n +\n test_expect_success '--batch-check without %(rest) considers whole line' '\n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n +'\n +\n +test_expect_success '--batch-command with multiple command calls gives correct format' '\n -+\tcat >expect <<-EOF &&\n ++\tremove_timestamp >expect <<-EOF &&\n +\t$hello_sha1 blob $hello_size\n +\t$hello_content\n +\t$commit_sha1 commit $commit_size\n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n +\tdeadbeef missing\n +\tEOF\n +\n -+\tmaybe_remove_timestamp \"$(cat expect)\" 1 >expect &&\n -+\tmaybe_remove_timestamp \"$(test_write_lines \"contents $hello_sha1\"\\\n ++\ttest_write_lines \"contents $hello_sha1\"\\\n +\t\"contents $commit_sha1\"\\\n +\t\"contents $tag_sha1\"\\\n +\t\"contents deadbeef\"\\\n -+\t\"flush\" | git cat-file --batch-command --buffer)\" 1 >actual &&\n ++\t\"flush\" | git cat-file --batch-command --buffer | remove_timestamp >actual &&\n +\ttest_cmp expect actual\n +'\n +\n\n-- \ngitgitgadget\n"},{"id":"448379","messageId":"fa6294387ab223d56e6ab66448f1ebe1ec370cd7.1644862988.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v6.git.git.1644862988.gitgitgadget@gmail.com","subject":"[PATCH v6 1/4] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-14T18:23:05Z","receivedAt":"2022-02-14T18:23:17Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIn the next patch, we will add an enum on the batch_options struct that\nindicates which type of batch operation will be used: --batch,\n--batch-check and the soon to be --batch-command that will read\ncommands from stdin. --batch-command mode might get confused with\nthe cmdmode flag.\n\nThere is value in renaming cmdmode in any case. cmdmode refers to how\nthe result output of the blob will be transformed, either according to\n--filter or --textconv. So transform_mode is a more descriptive name\nfor the flag.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"448380","messageId":"1a038097bfc1e0e7dc7737e4fb664e8f79a2da50.1644862988.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v6.git.git.1644862988.gitgitgadget@gmail.com","subject":"[PATCH v6 2/4] cat-file: introduce batch_mode enum to replace print_contents","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-14T18:23:06Z","receivedAt":"2022-02-14T18:23:19Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nA future patch introduces a new --batch-command flag. Including --batch\nand --batch-check, we will have a total of three batch modes. print_contents\nis the only boolean on the batch_options sturct used to distinguish\nbetween the different modes. This makes the code harder to read.\n\nTo reduce potential confusion, replace print_contents with an enum to\nhelp readability and clarity.\n\nHelped-by: Junio C Hamano \nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..5e38af82af1 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,10 +17,15 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n+enum batch_mode {\n+\tBATCH_MODE_CONTENTS,\n+\tBATCH_MODE_INFO,\n+};\n+\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tint print_contents;\n+\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -635,7 +640,14 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\n+\tif (!strcmp(opt->long_name, \"batch\"))\n+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n+\telse if (!strcmp(opt->long_name, \"batch-check\"))\n+\t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse\n+\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n+\n \tbo->format = arg;\n \n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"448381","messageId":"486ee84781679d587f137b9c24fb5377e4b138d8.1644862989.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v6.git.git.1644862988.gitgitgadget@gmail.com","subject":"[PATCH v6 3/4] cat-file: add remove_timestamp helper","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-14T18:23:07Z","receivedAt":"2022-02-14T18:23:25Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nmaybe_remove_timestamp() takes arguments, but it would be useful to have\na function that reads from stdin and strips the timestamp. This would\nallow tests to pipe data into a function to remove timestamps, and\nwouldn't have to always assign a variable. This is especially helpful\nwhen the data is multiple lines.\n\nKeep maybe_remove_timestamp() the same, but add a remove_timestamp\nhelper that reads from stdin.\n\nThe tests in the next patch will make use of this.\n\nSigned-off-by: John Cai \n---\n t/t1006-cat-file.sh | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..2d52851dadc 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -105,13 +105,18 @@ strlen () {\n }\n \n maybe_remove_timestamp () {\n- if test -z \"$2\"; then\n- echo_without_newline \"$1\"\n- else\n-\techo_without_newline \"$(printf '%s\\n' \"$1\" | sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//')\"\n- fi\n+\tif test -z \"$2\"; then\n+\t\techo_without_newline \"$1\"\n+\telse\n+\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n+\tfi\n }\n \n+remove_timestamp () {\n+\tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n+}\n+\n+\n run_tests () {\n type=$1\n sha1=$2\n-- \ngitgitgadget\n\n"},{"id":"448382","messageId":"a6dd5d72fce8cec430c0ec5da2dd9f0969a7fd91.1644862989.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v6.git.git.1644862988.gitgitgadget@gmail.com","subject":"[PATCH v6 4/4] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-14T18:23:08Z","receivedAt":"2022-02-14T18:23:32Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] LF\n [arg1] [arg2] LF\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush LF\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n--batch-command also will not allow (B) to flush to stdout until a flush\nis received.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents LF\ninfo LF\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes a argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\nflush LF\ninfo LF\nflush LF\n\nWhen used without --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\ninfo LF\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 42 +++++++++-\n builtin/cat-file.c | 135 +++++++++++++++++++++++++++++++++\n t/README | 3 +\n t/t1006-cat-file.sh | 107 +++++++++++++++++++++++++-\n 4 files changed, 282 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..1553f65c656 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,33 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+--batch-command=::\n+\tEnter a command mode that reads commands and arguments from stdin. May\n+\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n+\tcase of `--textconv` or `--filters`, the input lines also need to specify\n+\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n+\tfor details.\n++\n+`--batch-command` recognizes the following commands:\n++\n+--\n+contents ``::\n+\tPrint object contents for object reference ``. This corresponds to\n+\tthe output of `--batch`.\n+\n+info ``::\n+\tPrint object info for object reference ``. This corresponds to the\n+\toutput of `--batch-check`.\n+\n+flush::\n+\tUsed with `--buffer` to execute all preceding commands that were issued\n+\tsince the beginning or since the last flush was issued. When `--buffer`\n+\tis used, no output will come until a `flush` is issued. When `--buffer`\n+\tis not used, commands are flushed each time without issuing `flush`.\n+--\n++\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\n@@ -110,7 +137,7 @@ OPTIONS\n \tthat a process can interactively read and write from\n \t`cat-file`. With this option, the output uses normal stdio\n \tbuffering; this is much more efficient when invoking\n-\t`--batch-check` on a large number of objects.\n+\t`--batch-check` or `--batch-command` on a large number of objects.\n \n --unordered::\n \tWhen `--batch-all-objects` is in use, visit objects in an\n@@ -202,6 +229,13 @@ from stdin, one per line, and print information about them. By default,\n the whole line is considered as an object, as if it were fed to\n linkgit:git-rev-parse[1].\n \n+When `--batch-command` is given, `cat-file` will read commands from stdin,\n+one per line, and print information based on the command given. With\n+`--batch-command`, the `info` command followed by an object will print\n+information about the object the same way `--batch-check` would, and the\n+`contents` command followed by an object prints contents in the same way\n+`--batch` would.\n+\n You can specify the information shown for each object by using a custom\n ``. The `` is copied literally to stdout for each\n object, with placeholders of the form `%(atom)` expanded, followed by a\n@@ -237,9 +271,9 @@ newline. The available atoms are:\n If no format is specified, the default format is `%(objectname)\n %(objecttype) %(objectsize)`.\n \n-If `--batch` is specified, the object information is followed by the\n-object contents (consisting of `%(objectsize)` bytes), followed by a\n-newline.\n+If `--batch` is specified, or if `--batch-command` is used with the `contents`\n+command, the object information is followed by the object contents (consisting\n+of `%(objectsize)` bytes), followed by a newline.\n \n For example, `--batch` without a custom format would produce:\n \ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5e38af82af1..2d433bb6180 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -20,6 +20,7 @@\n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n \tBATCH_MODE_INFO,\n+\tBATCH_MODE_QUEUE_AND_DISPATCH,\n };\n \n struct batch_options {\n@@ -513,6 +514,129 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tchar *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_CONTENTS;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_INFO;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void dispatch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmd,\n+\t\tint nr)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < nr; i++){\n+\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\t\tfree(cmd[i].line);\n+\t}\n+\n+\tfflush(stdout);\n+}\n+\n+static const struct parse_cmd {\n+\tconst char *prefix;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *queued_cmd = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p = NULL, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n+\t\t\tif (!opt->buffer_output)\n+\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n+\t\t\tif (*cmd_end)\n+\t\t\t\tdie(_(\"flush takes no arguments\"));\n+\n+\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\t\t\tnr = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args) {\n+\t\t\t\tif (*cmd_end != ' ')\n+\t\t\t\t\tdie(_(\"%s requires arguments\"),\n+\t\t\t\t\t commands[i].prefix);\n+\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\t} else if (*cmd_end) {\n+\t\t\t\tdie(_(\"%s takes no arguments\"),\n+\t\t\t\t commands[i].prefix);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n+\t\tcall.fn = cmd->fn;\n+\t\tcall.line = xstrdup_or_null(p);\n+\t\tqueued_cmd[nr++] = call;\n+\t}\n+\n+\tif (opt->buffer_output &&\n+\t nr &&\n+\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n+\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\n+\tfree(queued_cmd);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -595,6 +719,10 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -613,6 +741,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -645,6 +774,8 @@ static int batch_option_callback(const struct option *opt,\n \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n \telse if (!strcmp(opt->long_name, \"batch-check\"))\n \t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse if (!strcmp(opt->long_name, \"batch-command\"))\n+\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n \telse\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \n@@ -695,6 +826,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/README b/t/README\nindex f48e0542cdc..bcd813b0c59 100644\n--- a/t/README\n+++ b/t/README\n@@ -472,6 +472,9 @@ a test and then fails then the whole test run will abort. This can help to make\n sure the expected tests are executed and not silently skipped when their\n dependency breaks or is simply not present in a new environment.\n \n+GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file\n+--batch-command from flushing to output on exit.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 2d52851dadc..3575cc63cff 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -182,6 +182,24 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ for opt in --buffer --no-buffer\n+ do\n+\ttest -z \"$content\" ||\n+\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n+\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n+\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_write_lines \"info $sha1\" \\\n+\t\t| git cat-file --batch-command $opt >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+ done\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -229,6 +247,22 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success '--batch-command --buffer with flush for blob info' '\n+\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command --buffer without flush for blob info' '\n+\ttouch output &&\n+\ttest_write_lines \"info $hello_sha1\" | \\\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >>output &&\n+\ttest_must_be_empty output\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -272,7 +306,7 @@ test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -378,6 +412,42 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+test_expect_success '--batch-command with multiple info calls gives correct format' '\n+\tcat >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$tree_sha1 tree $tree_size\n+\t$commit_sha1 commit $commit_size\n+\t$tag_sha1 tag $tag_size\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"info $hello_sha1\"\\\n+\t\"info $tree_sha1\"\\\n+\t\"info $commit_sha1\"\\\n+\t\"info $tag_sha1\"\\\n+\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command with multiple command calls gives correct format' '\n+\tremove_timestamp >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$hello_content\n+\t$commit_sha1 commit $commit_size\n+\t$commit_content\n+\t$tag_sha1 tag $tag_size\n+\t$tag_content\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"contents $hello_sha1\"\\\n+\t\"contents $commit_sha1\"\\\n+\t\"contents $tag_sha1\"\\\n+\t\"contents deadbeef\"\\\n+\t\"flush\" | git cat-file --batch-command --buffer | remove_timestamp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -968,5 +1038,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command missing arguments' '\n+\techo \"info\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*info requires arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"448471","messageId":"CAPig+cTwLhn1GZ_=6s0FXL0z=Q=p1w9ZGK0hAV8wfK9RsQYjnA@mail.gmail.com","threadId":"57365","inReplyTo":"a6dd5d72fce8cec430c0ec5da2dd9f0969a7fd91.1644862989.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 4/4] cat-file: add --batch-command mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-15T19:39:27Z","receivedAt":"2022-02-15T19:39:42Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Feb 14, 2022 at 1:23 PM John Cai via GitGitGadget\n wrote:\n> Add a new flag --batch-command that accepts commands and arguments\n> from stdin, similar to git-update-ref --stdin.\n\nSome relatively minor comments below. Not sure any of them are serious\nenough to warrant a reroll...\n\n> The contents command takes an argument and prints out the object\n> contents.\n>\n> The info command takes a argument and prints out the object\n> metadata.\n\ns/a /an /\n\n> Signed-off-by: John Cai \n> ---\n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> @@ -96,6 +96,33 @@ OPTIONS\n> +contents ``::\n> + Print object contents for object reference ``. This corresponds to\n> + the output of `--batch`.\n> +\n> +info ``::\n> + Print object info for object reference ``. This corresponds to the\n> + output of `--batch-check`.\n\nSorry if I wasn't clear in my earlier review, but when I suggested\ns//``/, I was referring only to the body of each item,\nnot to the item itself (for which we do not -- I think -- ever use\n`<...>`). So:\n\n content ::\n Print object contents ... ``. ...\n\nAs mentioned in my earlier review, I think the SYNOPSIS also needs an\nupdate to mention --batch-command.\n\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> @@ -513,6 +514,129 @@ static int batch_unordered_packed(const struct object_id *oid,\n> +static void dispatch_calls(struct batch_options *opt,\n> + struct strbuf *output,\n> + struct expand_data *data,\n> + struct queued_cmd *cmd,\n> + int nr)\n> +{\n> + int i;\n> +\n> + for (i = 0; i < nr; i++){\n\nStyle: for (...) {\n\n> + cmd[i].fn(opt, cmd[i].line, output, data);\n> + free(cmd[i].line);\n> + }\n> +\n> + fflush(stdout);\n> +}\n\nIf I recall correctly, Junio suggested calling free() within this\nloop, but this feels like an incorrect separation of concerns since it\ndoesn't also reset the caller's `nr` to 0. If (for some reason),\ndispatch_calls() is invoked twice in a row without resetting `nr` to 0\nin between the calls, then the dispatched commands will be called with\na pointer to freed memory.\n\nOne somewhat ugly way to fix this potential problem would be for this\nfunction to clear `nr`:\n\n static void dispatch_calls(..., int *nr)\n {\n for (...) {\n cmd[i].fn(...);\n free(cmd[i].line);\n }\n *nr = 0;\n flush(stdout);\n }\n\nBut, as this is a private helper, the code as presented in the patch\nmay be \"good enough\" even though it's a bit fragile.\n\n> +static void batch_objects_command(struct batch_options *opt,\n> + struct strbuf *output,\n> + struct expand_data *data)\n> +{\n> + while (!strbuf_getline(&input, stdin)) {\n> + if (!input.len)\n> + die(_(\"empty command in input\"));\n> + if (isspace(*input.buf))\n> + die(_(\"whitespace before command: '%s'\"), input.buf);\n> +\n> + if (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n> + if (!opt->buffer_output)\n> + die(_(\"flush is only for --buffer mode\"));\n> + if (*cmd_end)\n> + die(_(\"flush takes no arguments\"));\n\nI didn't articulate it in my previous review since the thought was\nonly half-formed, but given \"flushify\", this will incorrectly complain\nthat \"flush takes no arguments\" instead of complaining \"unknown\ncommand: flushify\" as is done below (or it will incorrectly complain\n\"flush is only for --buffer mode\" if --buffer wasn't specified).\n\nIf I'm reading the code correctly, it seems as if these problems could\nbe avoided by treating `flush` as just another parse_cmd::commands[]\nitem so that it gets all the same parsing/checking as the other\ncommands rather than parsing it separately here.\n\n> + dispatch_calls(opt, output, data, queued_cmd, nr);\n> + nr = 0;\n> + continue;\n> + }\n> +\n> + for (i = 0; i < ARRAY_SIZE(commands); i++) {\n> + if (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n> + continue;\n> +\n> + cmd = &commands[i];\n> + if (cmd->takes_args) {\n> + if (*cmd_end != ' ')\n> + die(_(\"%s requires arguments\"),\n> + commands[i].prefix);\n> +\n> + p = cmd_end + 1;\n> + } else if (*cmd_end) {\n> + die(_(\"%s takes no arguments\"),\n> + commands[i].prefix);\n> + }\n\nGood. Appears to be correctly handling the full matrix of\ncommand-requires-arguments and the actual input having or not having\narguments.\n\n> + break;\n> + }\n> +\n> + if (!cmd)\n> + die(_(\"unknown command: '%s'\"), input.buf);\n\nIf you treat `flush` as just another parse_cmd::commands[], then right\nhere is where you would handle it (I think):\n\n if (strcmp(cmd->prefix, \"flush\")) {\n dispatch_calls(opt, output, data, queued_cmd, nr);\n nr = 0;\n continue;\n }\n\n> + if (!opt->buffer_output) {\n> + cmd->fn(opt, p, output, data);\n> + continue;\n> + }\n> +\n> + ALLOC_GROW(queued_cmd, nr + 1, alloc);\n> + call.fn = cmd->fn;\n> + call.line = xstrdup_or_null(p);\n> + queued_cmd[nr++] = call;\n> + }\n> +\n> + if (opt->buffer_output &&\n> + nr &&\n> + !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n> + dispatch_calls(opt, output, data, queued_cmd, nr);\n> +\n> + free(queued_cmd);\n> + strbuf_release(&input);\n> +}\n"},{"id":"448496","messageId":"1FF9283F-C823-4D26-A39B-6506B900CEB2@gmail.com","threadId":"57365","inReplyTo":"CAPig+cTwLhn1GZ_=6s0FXL0z=Q=p1w9ZGK0hAV8wfK9RsQYjnA@mail.gmail.com","subject":"Re: [PATCH v6 4/4] cat-file: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-15T22:58:41Z","receivedAt":"2022-02-15T22:58:53Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Eric,\n\nThanks for taking another look!\n\nOn 15 Feb 2022, at 14:39, Eric Sunshine wrote:\n\n> On Mon, Feb 14, 2022 at 1:23 PM John Cai via GitGitGadget\n> wrote:\n>> Add a new flag --batch-command that accepts commands and arguments\n>> from stdin, similar to git-update-ref --stdin.\n>\n> Some relatively minor comments below. Not sure any of them are serious\n> enough to warrant a reroll...\n>\n>> The contents command takes an argument and prints out the object\n>> contents.\n>>\n>> The info command takes a argument and prints out the object\n>> metadata.\n>\n> s/a /an /\n>\n>> Signed-off-by: John Cai \n>> ---\n>> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n>> @@ -96,6 +96,33 @@ OPTIONS\n>> +contents ``::\n>> + Print object contents for object reference ``. This corresponds to\n>> + the output of `--batch`.\n>> +\n>> +info ``::\n>> + Print object info for object reference ``. This corresponds to the\n>> + output of `--batch-check`.\n>\n> Sorry if I wasn't clear in my earlier review, but when I suggested\n> s//``/, I was referring only to the body of each item,\n> not to the item itself (for which we do not -- I think -- ever use\n> `<...>`). So:\n>\n> content ::\n> Print object contents ... ``. ...\n>\n> As mentioned in my earlier review, I think the SYNOPSIS also needs an\n> update to mention --batch-command.\n\n:face-palm: yes I forgot about that in the last version.\n\n>\n>> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n>> @@ -513,6 +514,129 @@ static int batch_unordered_packed(const struct object_id *oid,\n>> +static void dispatch_calls(struct batch_options *opt,\n>> + struct strbuf *output,\n>> + struct expand_data *data,\n>> + struct queued_cmd *cmd,\n>> + int nr)\n>> +{\n>> + int i;\n>> +\n>> + for (i = 0; i < nr; i++){\n>\n> Style: for (...) {\n>\n>> + cmd[i].fn(opt, cmd[i].line, output, data);\n>> + free(cmd[i].line);\n>> + }\n>> +\n>> + fflush(stdout);\n>> +}\n>\n> If I recall correctly, Junio suggested calling free() within this\n> loop, but this feels like an incorrect separation of concerns since it\n> doesn't also reset the caller's `nr` to 0. If (for some reason),\n> dispatch_calls() is invoked twice in a row without resetting `nr` to 0\n> in between the calls, then the dispatched commands will be called with\n> a pointer to freed memory.\n>\n> One somewhat ugly way to fix this potential problem would be for this\n> function to clear `nr`:\n>\n> static void dispatch_calls(..., int *nr)\n> {\n> for (...) {\n> cmd[i].fn(...);\n> free(cmd[i].line);\n> }\n> *nr = 0;\n> flush(stdout);\n> }\n>\n> But, as this is a private helper, the code as presented in the patch\n> may be \"good enough\" even though it's a bit fragile.\n\nWhat you suggested makes sense from a separation of concerns point of view. I'm\nstill learning what looks ugly in C :), but I think this is easier to read\noverall than what I had before.\n\n>\n>> +static void batch_objects_command(struct batch_options *opt,\n>> + struct strbuf *output,\n>> + struct expand_data *data)\n>> +{\n>> + while (!strbuf_getline(&input, stdin)) {\n>> + if (!input.len)\n>> + die(_(\"empty command in input\"));\n>> + if (isspace(*input.buf))\n>> + die(_(\"whitespace before command: '%s'\"), input.buf);\n>> +\n>> + if (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n>> + if (!opt->buffer_output)\n>> + die(_(\"flush is only for --buffer mode\"));\n>> + if (*cmd_end)\n>> + die(_(\"flush takes no arguments\"));\n>\n> I didn't articulate it in my previous review since the thought was\n> only half-formed, but given \"flushify\", this will incorrectly complain\n> that \"flush takes no arguments\" instead of complaining \"unknown\n> command: flushify\" as is done below (or it will incorrectly complain\n> \"flush is only for --buffer mode\" if --buffer wasn't specified).\n>\n> If I'm reading the code correctly, it seems as if these problems could\n> be avoided by treating `flush` as just another parse_cmd::commands[]\n> item so that it gets all the same parsing/checking as the other\n> commands rather than parsing it separately here.\n\nThis is a good idea. I like the reduced complexity.\n\n>\n>> + dispatch_calls(opt, output, data, queued_cmd, nr);\n>> + nr = 0;\n>> + continue;\n>> + }\n>> +\n>> + for (i = 0; i < ARRAY_SIZE(commands); i++) {\n>> + if (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n>> + continue;\n>> +\n>> + cmd = &commands[i];\n>> + if (cmd->takes_args) {\n>> + if (*cmd_end != ' ')\n>> + die(_(\"%s requires arguments\"),\n>> + commands[i].prefix);\n>> +\n>> + p = cmd_end + 1;\n>> + } else if (*cmd_end) {\n>> + die(_(\"%s takes no arguments\"),\n>> + commands[i].prefix);\n>> + }\n>\n> Good. Appears to be correctly handling the full matrix of\n> command-requires-arguments and the actual input having or not having\n> arguments.\n>\n>> + break;\n>> + }\n>> +\n>> + if (!cmd)\n>> + die(_(\"unknown command: '%s'\"), input.buf);\n>\n> If you treat `flush` as just another parse_cmd::commands[], then right\n> here is where you would handle it (I think):\n>\n> if (strcmp(cmd->prefix, \"flush\")) {\n> dispatch_calls(opt, output, data, queued_cmd, nr);\n> nr = 0;\n> continue;\n> }\n>\n>> + if (!opt->buffer_output) {\n>> + cmd->fn(opt, p, output, data);\n>> + continue;\n>> + }\n>> +\n>> + ALLOC_GROW(queued_cmd, nr + 1, alloc);\n>> + call.fn = cmd->fn;\n>> + call.line = xstrdup_or_null(p);\n>> + queued_cmd[nr++] = call;\n>> + }\n>> +\n>> + if (opt->buffer_output &&\n>> + nr &&\n>> + !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n>> + dispatch_calls(opt, output, data, queued_cmd, nr);\n>> +\n>> + free(queued_cmd);\n>> + strbuf_release(&input);\n>> +}\n"},{"id":"448498","messageId":"CAPig+cR2fymDzEupNW3XvQDBmsd4rtQDQq-2ha6ODzbqoeFQ9A@mail.gmail.com","threadId":"57365","inReplyTo":"1FF9283F-C823-4D26-A39B-6506B900CEB2@gmail.com","subject":"Re: [PATCH v6 4/4] cat-file: add --batch-command mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-15T23:20:33Z","receivedAt":"2022-02-15T23:20:47Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Feb 15, 2022 at 5:58 PM John Cai wrote:\n> On 15 Feb 2022, at 14:39, Eric Sunshine wrote:\n> > On Mon, Feb 14, 2022 at 1:23 PM John Cai via GitGitGadget\n> > wrote:\n> >> +static void dispatch_calls(struct batch_options *opt,\n> >> + int nr)\n> >> +{\n> >> + for (i = 0; i < nr; i++){\n> >> + cmd[i].fn(opt, cmd[i].line, output, data);\n> >> + free(cmd[i].line);\n> >> + }\n> >> + fflush(stdout);\n> >> +}\n> >\n> > If I recall correctly, Junio suggested calling free() within this\n> > loop, but this feels like an incorrect separation of concerns since it\n> > doesn't also reset the caller's `nr` to 0. If (for some reason),\n> > dispatch_calls() is invoked twice in a row without resetting `nr` to 0\n> > in between the calls, then the dispatched commands will be called with\n> > a pointer to freed memory.\n> >\n> > One somewhat ugly way to fix this potential problem would be for this\n> > function to clear `nr`:\n> >\n> > static void dispatch_calls(..., int *nr)\n> > {\n> > for (...) {\n> > cmd[i].fn(...);\n> > free(cmd[i].line);\n> > }\n> > *nr = 0;\n> > flush(stdout);\n> > }\n> >\n> > But, as this is a private helper, the code as presented in the patch\n> > may be \"good enough\" even though it's a bit fragile.\n>\n> What you suggested makes sense from a separation of concerns point of view. I'm\n> still learning what looks ugly in C :), but I think this is easier to read\n> overall than what I had before.\n\nEven with my suggestion, it's still rather ugly for a \"dispatcher\"\nfunction to be freeing resources allocated by some other entity and\nfor it to be clearing the other entity's `nr` variable, but at least\nit's less fragile. It would be less ugly, perhaps, if this function\nwas named dispatch_and_free().\n\nA better way to make it less ugly would probably be to introduce a\nstructure which holds the array of batch_options* and the `nr` and\n`alloc` variables, and then have a dedicated function for\nclearing/freeing that structure. However, while such an approach is\nfine for reusable containers but is probably way overkill for this\none-off case.\n\n> > If I'm reading the code correctly, it seems as if these problems could\n> > be avoided by treating `flush` as just another parse_cmd::commands[]\n> > item so that it gets all the same parsing/checking as the other\n> > commands rather than parsing it separately here.\n>\n> This is a good idea. I like the reduced complexity.\n>\n> > If you treat `flush` as just another parse_cmd::commands[], then right\n> > here is where you would handle it (I think):\n> >\n> > if (strcmp(cmd->prefix, \"flush\")) {\n> > dispatch_calls(opt, output, data, queued_cmd, nr);\n> > nr = 0;\n> > continue;\n> > }\n\nOne other point which would make this suggested code even clearer is\nto rename the parse_cmd::prefix member to `command` or `name` or\n`token` or something other than \"prefix\" since it really _is_ the\ncommand name, not a prefix of the command. (You're only treating it as\na prefix semantically at parse time.) Thus:\n\n if (strcmp(cmd->name, \"flush\")) {\n dispatch_calls(opt, output, data, queued_cmd, nr);\n nr = 0;\n continue;\n }\n\nis easier to understand at a glance than `strcmp(cmd->prefix, \"flush\"))`.\n"},{"id":"448514","messageId":"fa6294387ab223d56e6ab66448f1ebe1ec370cd7.1644972810.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v7.git.git.1644972810.gitgitgadget@gmail.com","subject":"[PATCH v7 1/4] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T00:53:27Z","receivedAt":"2022-02-16T00:53:37Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIn the next patch, we will add an enum on the batch_options struct that\nindicates which type of batch operation will be used: --batch,\n--batch-check and the soon to be --batch-command that will read\ncommands from stdin. --batch-command mode might get confused with\nthe cmdmode flag.\n\nThere is value in renaming cmdmode in any case. cmdmode refers to how\nthe result output of the blob will be transformed, either according to\n--filter or --textconv. So transform_mode is a more descriptive name\nfor the flag.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"448515","messageId":"pull.1212.v7.git.git.1644972810.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v6.git.git.1644862988.gitgitgadget@gmail.com","subject":"[PATCH v7 0/4] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T00:53:26Z","receivedAt":"2022-02-16T00:53:39Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has four parts:\n\n 1. preparation patch to rename a variable\n 2. adding an enum to keep track of batch modes\n 3. add a remove_timestamp() helper that takes stdin and removes timestamps\n 4. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v6 (thanks to Eric's feedback)\n\n * allow command parsing logic to handle the case of flush as well\n * fixed documentation by adding --batch-command to the synopsis and\n adjusting tick marks\n * set nr=0 within helper function\n\nChanges since v5\n\n * replaced flush tests that used fifo pipes to using a GIT_TEST_ env\n variable to control whether or not --batch-command flushes on exit.\n * added remove_timestamp helper in tests.\n * added documentation to show format can be used with --batch-command\n\nChanges since v4\n\n * added Phillip's suggested test for testing flush. This should have\n addressed the flaky test that was hanging. I tested it on my side and\n wasn't able to reproduce the deadlock.\n * plugged some holes in the logic that parsed the command and arguments,\n thanks to Eric's feedback\n * fixed verbiage in commit messages per Christian's feedback\n * clarified places in documentation that should mention --batch-command per\n Eric's feedback\n\nChanges since v3 (thanks to Junio's feedback):\n\n * added cascading logic in batch_options_callback()\n * free memory for queued call input lines\n * do not throw error when flushing an empty queue\n * renamed cmds array to singular queued_cmd\n * fixed flaky test that failed --stress\n\nChanges since v2:\n\n * added enum to keep track of which batch mode we are in (thanks to Junio's\n feedback)\n * fixed array allocation logic (thanks to Junio's feedback)\n * added code to flush commands when --batch-commands receives an EOF and\n exits (thanks to Phillip's feedback)\n * fixed docs formatting (thanks to Jonathan's feedback)\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (4):\n cat-file: rename cmdmode to transform_mode\n cat-file: introduce batch_mode enum to replace print_contents\n cat-file: add remove_timestamp helper\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 42 +++++++-\n builtin/cat-file.c | 170 ++++++++++++++++++++++++++++++---\n t/README | 3 +\n t/t1006-cat-file.sh | 122 +++++++++++++++++++++--\n 4 files changed, 315 insertions(+), 22 deletions(-)\n\n\nbase-commit: 38062e73e009f27ea192d50481fcb5e7b0e9d6eb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v7\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v7\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v6:\n\n 1: fa6294387ab = 1: fa6294387ab cat-file: rename cmdmode to transform_mode\n 2: 1a038097bfc = 2: 1a038097bfc cat-file: introduce batch_mode enum to replace print_contents\n 3: 486ee847816 = 3: 486ee847816 cat-file: add remove_timestamp helper\n 4: a6dd5d72fce ! 4: aebaf7e3fe1 cat-file: add --batch-command mode\n @@ Commit message\n The contents command takes an argument and prints out the object\n contents.\n \n - The info command takes a argument and prints out the object\n + The info command takes an argument and prints out the object\n metadata.\n \n These can be used in the following way with --buffer:\n @@ Documentation/git-cat-file.txt: OPTIONS\n +`--batch-command` recognizes the following commands:\n ++\n +--\n -+contents ``::\n ++contents ::\n +\tPrint object contents for object reference ``. This corresponds to\n +\tthe output of `--batch`.\n +\n -+info ``::\n ++info ::\n +\tPrint object info for object reference ``. This corresponds to the\n +\toutput of `--batch-check`.\n +\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\tstruct strbuf *output,\n +\t\tstruct expand_data *data,\n +\t\tstruct queued_cmd *cmd,\n -+\t\tint nr)\n ++\t\tsize_t *nr)\n +{\n +\tint i;\n +\n -+\tfor (i = 0; i < nr; i++){\n ++\tif (!opt->buffer_output)\n ++\t\tdie(_(\"flush is only for --buffer mode\"));\n ++\n ++\tfor (i = 0; i < *nr; i++) {\n +\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n +\t\tfree(cmd[i].line);\n +\t}\n +\n ++\t*nr = 0;\n +\tfflush(stdout);\n +}\n +\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +} commands[] = {\n +\t{ \"contents\", parse_cmd_contents, 1},\n +\t{ \"info\", parse_cmd_info, 1},\n ++\t{ \"flush\", NULL, 0},\n +};\n +\n +static void batch_objects_command(struct batch_options *opt,\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\tif (isspace(*input.buf))\n +\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n +\n -+\t\tif (skip_prefix(input.buf, \"flush\", &cmd_end)) {\n -+\t\t\tif (!opt->buffer_output)\n -+\t\t\t\tdie(_(\"flush is only for --buffer mode\"));\n -+\t\t\tif (*cmd_end)\n -+\t\t\t\tdie(_(\"flush takes no arguments\"));\n -+\n -+\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n -+\t\t\tnr = 0;\n -+\t\t\tcontinue;\n -+\t\t}\n -+\n +\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n +\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n +\t\t\t\tcontinue;\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\tif (!cmd)\n +\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n +\n ++\t\tif (!strcmp(cmd->prefix, \"flush\")) {\n ++\t\t\tdispatch_calls(opt, output, data, queued_cmd, &nr);\n ++\t\t\tcontinue;\n ++\t\t}\n ++\n +\t\tif (!opt->buffer_output) {\n +\t\t\tcmd->fn(opt, p, output, data);\n +\t\t\tcontinue;\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\tif (opt->buffer_output &&\n +\t nr &&\n +\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n -+\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n ++\t\tdispatch_calls(opt, output, data, queued_cmd, &nr);\n +\n +\tfree(queued_cmd);\n +\tstrbuf_release(&input);\n @@ builtin/cat-file.c: static int batch_option_callback(const struct option *opt,\n \telse\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \n +@@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *prefix)\n + \t\tN_(\"git cat-file \"),\n + \t\tN_(\"git cat-file (-e | -p) \"),\n + \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] \"),\n +-\t\tN_(\"git cat-file (--batch | --batch-check) [--batch-all-objects]\\n\"\n ++\t\tN_(\"git cat-file (--batch | --batch-check | --batch-command) [--batch-all-objects]\\n\"\n + \t\t \" [--buffer] [--follow-symlinks] [--unordered]\\n\"\n + \t\t \" [--textconv | --filters]\"),\n + \t\tN_(\"git cat-file (--textconv | --filters)\\n\"\n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch-all-objects --batch-c\n +'\n +\n +test_expect_success 'batch-command flush without --buffer' '\n -+\techo \"flush arg\" >cmd &&\n ++\techo \"flush\" >cmd &&\n +\ttest_expect_code 128 git cat-file --batch-command err &&\n +\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n +'\n\n-- \ngitgitgadget\n"},{"id":"448516","messageId":"1a038097bfc1e0e7dc7737e4fb664e8f79a2da50.1644972810.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v7.git.git.1644972810.gitgitgadget@gmail.com","subject":"[PATCH v7 2/4] cat-file: introduce batch_mode enum to replace print_contents","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T00:53:28Z","receivedAt":"2022-02-16T00:53:41Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nA future patch introduces a new --batch-command flag. Including --batch\nand --batch-check, we will have a total of three batch modes. print_contents\nis the only boolean on the batch_options sturct used to distinguish\nbetween the different modes. This makes the code harder to read.\n\nTo reduce potential confusion, replace print_contents with an enum to\nhelp readability and clarity.\n\nHelped-by: Junio C Hamano \nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..5e38af82af1 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,10 +17,15 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n+enum batch_mode {\n+\tBATCH_MODE_CONTENTS,\n+\tBATCH_MODE_INFO,\n+};\n+\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tint print_contents;\n+\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -635,7 +640,14 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\n+\tif (!strcmp(opt->long_name, \"batch\"))\n+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n+\telse if (!strcmp(opt->long_name, \"batch-check\"))\n+\t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse\n+\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n+\n \tbo->format = arg;\n \n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"448517","messageId":"486ee84781679d587f137b9c24fb5377e4b138d8.1644972810.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v7.git.git.1644972810.gitgitgadget@gmail.com","subject":"[PATCH v7 3/4] cat-file: add remove_timestamp helper","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T00:53:29Z","receivedAt":"2022-02-16T00:53:44Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nmaybe_remove_timestamp() takes arguments, but it would be useful to have\na function that reads from stdin and strips the timestamp. This would\nallow tests to pipe data into a function to remove timestamps, and\nwouldn't have to always assign a variable. This is especially helpful\nwhen the data is multiple lines.\n\nKeep maybe_remove_timestamp() the same, but add a remove_timestamp\nhelper that reads from stdin.\n\nThe tests in the next patch will make use of this.\n\nSigned-off-by: John Cai \n---\n t/t1006-cat-file.sh | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..2d52851dadc 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -105,13 +105,18 @@ strlen () {\n }\n \n maybe_remove_timestamp () {\n- if test -z \"$2\"; then\n- echo_without_newline \"$1\"\n- else\n-\techo_without_newline \"$(printf '%s\\n' \"$1\" | sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//')\"\n- fi\n+\tif test -z \"$2\"; then\n+\t\techo_without_newline \"$1\"\n+\telse\n+\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n+\tfi\n }\n \n+remove_timestamp () {\n+\tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n+}\n+\n+\n run_tests () {\n type=$1\n sha1=$2\n-- \ngitgitgadget\n\n"},{"id":"448518","messageId":"aebaf7e3fe1d4448037d2caf03f3de393908ff9a.1644972810.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v7.git.git.1644972810.gitgitgadget@gmail.com","subject":"[PATCH v7 4/4] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T00:53:30Z","receivedAt":"2022-02-16T00:53:50Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] LF\n [arg1] [arg2] LF\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush LF\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n--batch-command also will not allow (B) to flush to stdout until a flush\nis received.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents LF\ninfo LF\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes an argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\nflush LF\ninfo LF\nflush LF\n\nWhen used without --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\ninfo LF\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 42 +++++++++-\n builtin/cat-file.c | 136 ++++++++++++++++++++++++++++++++-\n t/README | 3 +\n t/t1006-cat-file.sh | 107 +++++++++++++++++++++++++-\n 4 files changed, 282 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..70c5b4f12d1 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,33 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+--batch-command=::\n+\tEnter a command mode that reads commands and arguments from stdin. May\n+\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n+\tcase of `--textconv` or `--filters`, the input lines also need to specify\n+\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n+\tfor details.\n++\n+`--batch-command` recognizes the following commands:\n++\n+--\n+contents ::\n+\tPrint object contents for object reference ``. This corresponds to\n+\tthe output of `--batch`.\n+\n+info ::\n+\tPrint object info for object reference ``. This corresponds to the\n+\toutput of `--batch-check`.\n+\n+flush::\n+\tUsed with `--buffer` to execute all preceding commands that were issued\n+\tsince the beginning or since the last flush was issued. When `--buffer`\n+\tis used, no output will come until a `flush` is issued. When `--buffer`\n+\tis not used, commands are flushed each time without issuing `flush`.\n+--\n++\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\n@@ -110,7 +137,7 @@ OPTIONS\n \tthat a process can interactively read and write from\n \t`cat-file`. With this option, the output uses normal stdio\n \tbuffering; this is much more efficient when invoking\n-\t`--batch-check` on a large number of objects.\n+\t`--batch-check` or `--batch-command` on a large number of objects.\n \n --unordered::\n \tWhen `--batch-all-objects` is in use, visit objects in an\n@@ -202,6 +229,13 @@ from stdin, one per line, and print information about them. By default,\n the whole line is considered as an object, as if it were fed to\n linkgit:git-rev-parse[1].\n \n+When `--batch-command` is given, `cat-file` will read commands from stdin,\n+one per line, and print information based on the command given. With\n+`--batch-command`, the `info` command followed by an object will print\n+information about the object the same way `--batch-check` would, and the\n+`contents` command followed by an object prints contents in the same way\n+`--batch` would.\n+\n You can specify the information shown for each object by using a custom\n ``. The `` is copied literally to stdout for each\n object, with placeholders of the form `%(atom)` expanded, followed by a\n@@ -237,9 +271,9 @@ newline. The available atoms are:\n If no format is specified, the default format is `%(objectname)\n %(objecttype) %(objectsize)`.\n \n-If `--batch` is specified, the object information is followed by the\n-object contents (consisting of `%(objectsize)` bytes), followed by a\n-newline.\n+If `--batch` is specified, or if `--batch-command` is used with the `contents`\n+command, the object information is followed by the object contents (consisting\n+of `%(objectsize)` bytes), followed by a newline.\n \n For example, `--batch` without a custom format would produce:\n \ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5e38af82af1..5dd876c5b09 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -20,6 +20,7 @@\n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n \tBATCH_MODE_INFO,\n+\tBATCH_MODE_QUEUE_AND_DISPATCH,\n };\n \n struct batch_options {\n@@ -513,6 +514,128 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tchar *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_CONTENTS;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_INFO;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void dispatch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmd,\n+\t\tsize_t *nr)\n+{\n+\tint i;\n+\n+\tif (!opt->buffer_output)\n+\t\tdie(_(\"flush is only for --buffer mode\"));\n+\n+\tfor (i = 0; i < *nr; i++) {\n+\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\t\tfree(cmd[i].line);\n+\t}\n+\n+\t*nr = 0;\n+\tfflush(stdout);\n+}\n+\n+static const struct parse_cmd {\n+\tconst char *prefix;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+\t{ \"flush\", NULL, 0},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *queued_cmd = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p = NULL, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args) {\n+\t\t\t\tif (*cmd_end != ' ')\n+\t\t\t\t\tdie(_(\"%s requires arguments\"),\n+\t\t\t\t\t commands[i].prefix);\n+\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\t} else if (*cmd_end) {\n+\t\t\t\tdie(_(\"%s takes no arguments\"),\n+\t\t\t\t commands[i].prefix);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!strcmp(cmd->prefix, \"flush\")) {\n+\t\t\tdispatch_calls(opt, output, data, queued_cmd, &nr);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n+\t\tcall.fn = cmd->fn;\n+\t\tcall.line = xstrdup_or_null(p);\n+\t\tqueued_cmd[nr++] = call;\n+\t}\n+\n+\tif (opt->buffer_output &&\n+\t nr &&\n+\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n+\t\tdispatch_calls(opt, output, data, queued_cmd, &nr);\n+\n+\tfree(queued_cmd);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -595,6 +718,10 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -613,6 +740,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -645,6 +773,8 @@ static int batch_option_callback(const struct option *opt,\n \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n \telse if (!strcmp(opt->long_name, \"batch-check\"))\n \t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse if (!strcmp(opt->long_name, \"batch-command\"))\n+\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n \telse\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \n@@ -666,7 +796,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\tN_(\"git cat-file \"),\n \t\tN_(\"git cat-file (-e | -p) \"),\n \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] \"),\n-\t\tN_(\"git cat-file (--batch | --batch-check) [--batch-all-objects]\\n\"\n+\t\tN_(\"git cat-file (--batch | --batch-check | --batch-command) [--batch-all-objects]\\n\"\n \t\t \" [--buffer] [--follow-symlinks] [--unordered]\\n\"\n \t\t \" [--textconv | --filters]\"),\n \t\tN_(\"git cat-file (--textconv | --filters)\\n\"\n@@ -695,6 +825,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/README b/t/README\nindex f48e0542cdc..bcd813b0c59 100644\n--- a/t/README\n+++ b/t/README\n@@ -472,6 +472,9 @@ a test and then fails then the whole test run will abort. This can help to make\n sure the expected tests are executed and not silently skipped when their\n dependency breaks or is simply not present in a new environment.\n \n+GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file\n+--batch-command from flushing to output on exit.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 2d52851dadc..74f0e36b69e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -182,6 +182,24 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ for opt in --buffer --no-buffer\n+ do\n+\ttest -z \"$content\" ||\n+\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n+\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n+\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_write_lines \"info $sha1\" \\\n+\t\t| git cat-file --batch-command $opt >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+ done\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -229,6 +247,22 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success '--batch-command --buffer with flush for blob info' '\n+\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command --buffer without flush for blob info' '\n+\ttouch output &&\n+\ttest_write_lines \"info $hello_sha1\" | \\\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >>output &&\n+\ttest_must_be_empty output\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -272,7 +306,7 @@ test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -378,6 +412,42 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+test_expect_success '--batch-command with multiple info calls gives correct format' '\n+\tcat >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$tree_sha1 tree $tree_size\n+\t$commit_sha1 commit $commit_size\n+\t$tag_sha1 tag $tag_size\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"info $hello_sha1\"\\\n+\t\"info $tree_sha1\"\\\n+\t\"info $commit_sha1\"\\\n+\t\"info $tag_sha1\"\\\n+\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command with multiple command calls gives correct format' '\n+\tremove_timestamp >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$hello_content\n+\t$commit_sha1 commit $commit_size\n+\t$commit_content\n+\t$tag_sha1 tag $tag_size\n+\t$tag_content\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"contents $hello_sha1\"\\\n+\t\"contents $commit_sha1\"\\\n+\t\"contents $tag_sha1\"\\\n+\t\"contents deadbeef\"\\\n+\t\"flush\" | git cat-file --batch-command --buffer | remove_timestamp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -968,5 +1038,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command missing arguments' '\n+\techo \"info\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*info requires arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"448523","messageId":"xmqqfsojy50b.fsf@gitster.g","threadId":"57365","inReplyTo":"aebaf7e3fe1d4448037d2caf03f3de393908ff9a.1644972810.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 4/4] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-16T01:28:04Z","receivedAt":"2022-02-16T01:28:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> +static void dispatch_calls(struct batch_options *opt,\n> +\t\tstruct strbuf *output,\n> +\t\tstruct expand_data *data,\n> +\t\tstruct queued_cmd *cmd,\n> +\t\tsize_t *nr)\n> +{\n> +\tint i;\n> +\n> +\tif (!opt->buffer_output)\n> +\t\tdie(_(\"flush is only for --buffer mode\"));\n> +\n> +\tfor (i = 0; i < *nr; i++) {\n\nIf you updated the max number of items *nr to size_t, don't you need\nto use 'i' with the same type to count up to it?\n\n> +\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n> +\t\tfree(cmd[i].line);\n> +\t}\n> +\n> +\t*nr = 0;\n> +\tfflush(stdout);\n> +}\n\nWouldn't it be easier to reason about what the caller/callee are\nresponsible for, if the function signature looked more like:\n\n\tstatic size_t dispatch_calls(struct batch_options *opt,\n\t\t\t\t ...\n\t\t\t\t struct queued_cmd cmd[], size_t nr)\n\t{\n\t\tsize_t i;\n\n\t\tfor (i = 0; i < nr; i++)\n\t\t\t... do stuff ...;\n\n\t\treturn updated_nr;\n\t}\n\nand make the caller do\n\n\tnr = dispatch_calls(opt, ..., nr);\n\nor if the function *never* leaves anything in the queue when it\nreturns, then\n\n\tstatic void dispatch_calls(struct batch_options *opt,\n\t\t\t\t ...\n\t\t\t\t struct queued_cmd cmd[], size_t nr)\n\t{\n\t\tsize_t i;\n\n\t\tfor (i = 0; i < nr; i++)\n\t\t\t... do stuff ...;\n\t}\n\nand make the caller do\n\n\tdispatch_calls(opt, ..., nr);\n\tnr = 0;\n\ninstead of passing a pointer to nr like the posted patch?\n"},{"id":"448529","messageId":"BE7471F8-0141-49ED-A34B-04D69BE18D96@gmail.com","threadId":"57365","inReplyTo":"xmqqfsojy50b.fsf@gitster.g","subject":"Re: [PATCH v7 4/4] cat-file: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-16T02:48:31Z","receivedAt":"2022-02-16T02:48:36Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"\n\nOn 15 Feb 2022, at 20:28, Junio C Hamano wrote:\n\n> \"John Cai via GitGitGadget\" writes:\n>\n>> +static void dispatch_calls(struct batch_options *opt,\n>> +\t\tstruct strbuf *output,\n>> +\t\tstruct expand_data *data,\n>> +\t\tstruct queued_cmd *cmd,\n>> +\t\tsize_t *nr)\n>> +{\n>> +\tint i;\n>> +\n>> +\tif (!opt->buffer_output)\n>> +\t\tdie(_(\"flush is only for --buffer mode\"));\n>> +\n>> +\tfor (i = 0; i < *nr; i++) {\n>\n> If you updated the max number of items *nr to size_t, don't you need\n> to use 'i' with the same type to count up to it?\n>\n>> +\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n>> +\t\tfree(cmd[i].line);\n>> +\t}\n>> +\n>> +\t*nr = 0;\n>> +\tfflush(stdout);\n>> +}\n>\n> Wouldn't it be easier to reason about what the caller/callee are\n> responsible for, if the function signature looked more like:\n>\n> static size_t dispatch_calls(struct batch_options *opt,\n> \t\t\t ...\n> \t\t\t struct queued_cmd cmd[], size_t nr)\n> {\n> \tsize_t i;\n>\n> \tfor (i = 0; i < nr; i++)\n> \t\t... do stuff ...;\n>\n> \treturn updated_nr;\n> }\n>\n> and make the caller do\n>\n> nr = dispatch_calls(opt, ..., nr);\n>\n> or if the function *never* leaves anything in the queue when it\n> returns, then\n>\n> static void dispatch_calls(struct batch_options *opt,\n> \t\t\t ...\n> \t\t\t struct queued_cmd cmd[], size_t nr)\n> {\n> \tsize_t i;\n>\n> \tfor (i = 0; i < nr; i++)\n> \t\t... do stuff ...;\n> }\n>\n> and make the caller do\n>\n> dispatch_calls(opt, ..., nr);\n> nr = 0;\n>\n> instead of passing a pointer to nr like the posted patch?\n\nYeah, this is what I had before but there was discussion about separation of concerns in [1]. But perhaps it's preferable compared to passing a pointer to nr.\n\n1. https://lore.kernel.org/git/CAPig+cTwLhn1GZ_=6s0FXL0z=Q=p1w9ZGK0hAV8wfK9RsQYjnA@mail.gmail.com/\n\n"},{"id":"448530","messageId":"xmqq4k4zwm65.fsf@gitster.g","threadId":"57365","inReplyTo":"BE7471F8-0141-49ED-A34B-04D69BE18D96@gmail.com","subject":"Re: [PATCH v7 4/4] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-16T03:00:18Z","receivedAt":"2022-02-16T03:00:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Cai writes:\n\n> Yeah, this is what I had before but there was discussion about\n> separation of concerns in [1]. But perhaps it's preferable\n> compared to passing a pointer to nr.\n\nOh, I see.\n\nI do not see any issue with separation of concerns here, actually.\n\nAs long as \"dispatch_calls() consumes all the cmd[] before it\nreturns to the caller\" is clearly understood between the function\nand its caller(s) [*], clearing of \"nr\" the caller has is entirely\ncaller's problem. It becomes needed only because this caller\ndecides to reuse cmd[] array.\n\n\tSide note: you do have a comment before the function to tell\n\twhat to expect out of the helper for its callers, right?\n\nIf it were just \"accumulate many cmd[] and call the function once\"\ncaller, it would care to maintain the correct \"nr\" only up to the\npoint where the function is called (because pair is the\nway the function takes the list of commands and it needs a correct\n\"nr\"), but not after making the call, as the only thing left to do\nwould be to free the cmd[] array itself, which does not even need\n\"nr\".\n\n"},{"id":"448531","messageId":"CAPig+cQUwErXaZR-xZ8u9iz4kpEkE4G5QaxNRCuUx_Xt0rQ_-g@mail.gmail.com","threadId":"57365","inReplyTo":"BE7471F8-0141-49ED-A34B-04D69BE18D96@gmail.com","subject":"Re: [PATCH v7 4/4] cat-file: add --batch-command mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-16T03:01:07Z","receivedAt":"2022-02-16T03:01:21Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Feb 15, 2022 at 9:48 PM John Cai wrote:\n> On 15 Feb 2022, at 20:28, Junio C Hamano wrote:\n> > and make the caller do\n> >\n> > dispatch_calls(opt, ..., nr);\n> > nr = 0;\n> >\n> > instead of passing a pointer to nr like the posted patch?\n>\n> Yeah, this is what I had before but there was discussion about separation of concerns in [1]. But perhaps it's preferable compared to passing a pointer to nr.\n>\n> 1. https://lore.kernel.org/git/CAPig+cTwLhn1GZ_=6s0FXL0z=Q=p1w9ZGK0hAV8wfK9RsQYjnA@mail.gmail.com/\n\nMy biggest concern when mentioning it during review was that if a\ncaller forgets to do `nr = 0`, then a sequence such as:\n\n dispatch_calls(...);\n ...\n dispatch_calls(...);\n\nwill send dangling pointers to the command handlers in the second\ndispatch_calls() invocation because the first call to dispatch_calls()\ndid free(cmd[i].line). In that sense, it's an accident waiting to\nhappen if people modifying this code in the future aren't paying close\nattention.\n\nThat said, I don't feel too strongly about it and mentioned in my\nreview that it might be \"good enough\" as-is (with the caller having to\nremember to `nr = 0`) since it's a local helper function.\n"},{"id":"448535","messageId":"CAPig+cSEO407rp3_ufS_DV1XJnkkb++6MP8UJhpj5aQP04gZ-Q@mail.gmail.com","threadId":"57365","inReplyTo":"xmqq4k4zwm65.fsf@gitster.g","subject":"Re: [PATCH v7 4/4] cat-file: add --batch-command mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-16T03:17:13Z","receivedAt":"2022-02-16T03:17:28Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Feb 15, 2022 at 10:00 PM Junio C Hamano wrote:\n> John Cai writes:\n> > Yeah, this is what I had before but there was discussion about\n> > separation of concerns in [1]. But perhaps it's preferable\n> > compared to passing a pointer to nr.\n>\n> Oh, I see.\n>\n> I do not see any issue with separation of concerns here, actually.\n>\n> As long as \"dispatch_calls() consumes all the cmd[] before it\n> returns to the caller\" is clearly understood between the function\n> and its caller(s) [*], clearing of \"nr\" the caller has is entirely\n> caller's problem. It becomes needed only because this caller\n> decides to reuse cmd[] array.\n>\n> Side note: you do have a comment before the function to tell\n> what to expect out of the helper for its callers, right?\n\nTrue, it is the caller's problem due to reusing the array, though\ndispatch_calls() freeing `line` muddies things slightly. A comment\nbefore the function could indeed be helpful, especially letting\ncallers know that `line` is being freed.\n\nIt probably also would be wise to replace the free() call with\nFREE_AND_NULL() in order to address the dangling-pointer concern.\n"},{"id":"448598","messageId":"pull.1212.v8.git.git.1645023740.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v7.git.git.1644972810.gitgitgadget@gmail.com","subject":"[PATCH v8 0/4] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T15:02:16Z","receivedAt":"2022-02-16T15:02:27Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has four parts:\n\n 1. preparation patch to rename a variable\n 2. adding an enum to keep track of batch modes\n 3. add a remove_timestamp() helper that takes stdin and removes timestamps\n 4. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v7\n\n * revert back to having caller set nr to 0\n * add comment before dispatch_calls to clarify usage of helper\n * rename prefix->name\n\nChanges since v6 (thanks to Eric's feedback)\n\n * allow command parsing logic to handle the case of flush as well\n * fixed documentation by adding --batch-command to the synopsis and\n adjusting tick marks\n * set nr=0 within helper function\n\nChanges since v5\n\n * replaced flush tests that used fifo pipes to using a GIT_TEST_ env\n variable to control whether or not --batch-command flushes on exit.\n * added remove_timestamp helper in tests.\n * added documentation to show format can be used with --batch-command\n\nChanges since v4\n\n * added Phillip's suggested test for testing flush. This should have\n addressed the flaky test that was hanging. I tested it on my side and\n wasn't able to reproduce the deadlock.\n * plugged some holes in the logic that parsed the command and arguments,\n thanks to Eric's feedback\n * fixed verbiage in commit messages per Christian's feedback\n * clarified places in documentation that should mention --batch-command per\n Eric's feedback\n\nChanges since v3 (thanks to Junio's feedback):\n\n * added cascading logic in batch_options_callback()\n * free memory for queued call input lines\n * do not throw error when flushing an empty queue\n * renamed cmds array to singular queued_cmd\n * fixed flaky test that failed --stress\n\nChanges since v2:\n\n * added enum to keep track of which batch mode we are in (thanks to Junio's\n feedback)\n * fixed array allocation logic (thanks to Junio's feedback)\n * added code to flush commands when --batch-commands receives an EOF and\n exits (thanks to Phillip's feedback)\n * fixed docs formatting (thanks to Jonathan's feedback)\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (4):\n cat-file: rename cmdmode to transform_mode\n cat-file: introduce batch_mode enum to replace print_contents\n cat-file: add remove_timestamp helper\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 42 +++++++-\n builtin/cat-file.c | 174 ++++++++++++++++++++++++++++++---\n t/README | 3 +\n t/t1006-cat-file.sh | 122 +++++++++++++++++++++--\n 4 files changed, 319 insertions(+), 22 deletions(-)\n\n\nbase-commit: 38062e73e009f27ea192d50481fcb5e7b0e9d6eb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v8\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v8\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v7:\n\n 1: fa6294387ab = 1: fa6294387ab cat-file: rename cmdmode to transform_mode\n 2: 1a038097bfc = 2: 1a038097bfc cat-file: introduce batch_mode enum to replace print_contents\n 3: 486ee847816 = 3: 486ee847816 cat-file: add remove_timestamp helper\n 4: aebaf7e3fe1 ! 4: 8edf80574b8 cat-file: add --batch-command mode\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\tbatch_one_object(line, output, opt, data);\n +}\n +\n ++/* Loop through each queued_cmd, dispatch the function, free the\n ++ * memory associated with line so the cmd array can be reused.\n ++ * Callers must set nr back to 0 in order to reuse the cmd array.\n ++ */\n +static void dispatch_calls(struct batch_options *opt,\n +\t\tstruct strbuf *output,\n +\t\tstruct expand_data *data,\n +\t\tstruct queued_cmd *cmd,\n -+\t\tsize_t *nr)\n ++\t\tint nr)\n +{\n +\tint i;\n +\n +\tif (!opt->buffer_output)\n +\t\tdie(_(\"flush is only for --buffer mode\"));\n +\n -+\tfor (i = 0; i < *nr; i++) {\n ++\tfor (i = 0; i < nr; i++) {\n +\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n -+\t\tfree(cmd[i].line);\n ++\t\tFREE_AND_NULL(cmd[i].line);\n +\t}\n +\n -+\t*nr = 0;\n +\tfflush(stdout);\n +}\n +\n +static const struct parse_cmd {\n -+\tconst char *prefix;\n ++\tconst char *name;\n +\tparse_cmd_fn_t fn;\n +\tunsigned takes_args;\n +} commands[] = {\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n +\n +\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n -+\t\t\tif (!skip_prefix(input.buf, commands[i].prefix, &cmd_end))\n ++\t\t\tif (!skip_prefix(input.buf, commands[i].name, &cmd_end))\n +\t\t\t\tcontinue;\n +\n +\t\t\tcmd = &commands[i];\n +\t\t\tif (cmd->takes_args) {\n +\t\t\t\tif (*cmd_end != ' ')\n +\t\t\t\t\tdie(_(\"%s requires arguments\"),\n -+\t\t\t\t\t commands[i].prefix);\n ++\t\t\t\t\t commands[i].name);\n +\n +\t\t\t\tp = cmd_end + 1;\n +\t\t\t} else if (*cmd_end) {\n +\t\t\t\tdie(_(\"%s takes no arguments\"),\n -+\t\t\t\t commands[i].prefix);\n ++\t\t\t\t commands[i].name);\n +\t\t\t}\n +\n +\t\t\tbreak;\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\t\tif (!cmd)\n +\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n +\n -+\t\tif (!strcmp(cmd->prefix, \"flush\")) {\n -+\t\t\tdispatch_calls(opt, output, data, queued_cmd, &nr);\n ++\t\tif (!strcmp(cmd->name, \"flush\")) {\n ++\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n ++\t\t\tnr = 0;\n +\t\t\tcontinue;\n +\t\t}\n +\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\tif (opt->buffer_output &&\n +\t nr &&\n +\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n -+\t\tdispatch_calls(opt, output, data, queued_cmd, &nr);\n ++\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n +\n +\tfree(queued_cmd);\n +\tstrbuf_release(&input);\n\n-- \ngitgitgadget\n"},{"id":"448599","messageId":"1a038097bfc1e0e7dc7737e4fb664e8f79a2da50.1645023740.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v8.git.git.1645023740.gitgitgadget@gmail.com","subject":"[PATCH v8 2/4] cat-file: introduce batch_mode enum to replace print_contents","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T15:02:18Z","receivedAt":"2022-02-16T15:02:28Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nA future patch introduces a new --batch-command flag. Including --batch\nand --batch-check, we will have a total of three batch modes. print_contents\nis the only boolean on the batch_options sturct used to distinguish\nbetween the different modes. This makes the code harder to read.\n\nTo reduce potential confusion, replace print_contents with an enum to\nhelp readability and clarity.\n\nHelped-by: Junio C Hamano \nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..5e38af82af1 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,10 +17,15 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n+enum batch_mode {\n+\tBATCH_MODE_CONTENTS,\n+\tBATCH_MODE_INFO,\n+};\n+\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tint print_contents;\n+\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -635,7 +640,14 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\n+\tif (!strcmp(opt->long_name, \"batch\"))\n+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n+\telse if (!strcmp(opt->long_name, \"batch-check\"))\n+\t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse\n+\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n+\n \tbo->format = arg;\n \n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"448600","messageId":"fa6294387ab223d56e6ab66448f1ebe1ec370cd7.1645023740.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v8.git.git.1645023740.gitgitgadget@gmail.com","subject":"[PATCH v8 1/4] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T15:02:17Z","receivedAt":"2022-02-16T15:02:31Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIn the next patch, we will add an enum on the batch_options struct that\nindicates which type of batch operation will be used: --batch,\n--batch-check and the soon to be --batch-command that will read\ncommands from stdin. --batch-command mode might get confused with\nthe cmdmode flag.\n\nThere is value in renaming cmdmode in any case. cmdmode refers to how\nthe result output of the blob will be transformed, either according to\n--filter or --textconv. So transform_mode is a more descriptive name\nfor the flag.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"448601","messageId":"486ee84781679d587f137b9c24fb5377e4b138d8.1645023740.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v8.git.git.1645023740.gitgitgadget@gmail.com","subject":"[PATCH v8 3/4] cat-file: add remove_timestamp helper","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T15:02:19Z","receivedAt":"2022-02-16T15:02:37Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nmaybe_remove_timestamp() takes arguments, but it would be useful to have\na function that reads from stdin and strips the timestamp. This would\nallow tests to pipe data into a function to remove timestamps, and\nwouldn't have to always assign a variable. This is especially helpful\nwhen the data is multiple lines.\n\nKeep maybe_remove_timestamp() the same, but add a remove_timestamp\nhelper that reads from stdin.\n\nThe tests in the next patch will make use of this.\n\nSigned-off-by: John Cai \n---\n t/t1006-cat-file.sh | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..2d52851dadc 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -105,13 +105,18 @@ strlen () {\n }\n \n maybe_remove_timestamp () {\n- if test -z \"$2\"; then\n- echo_without_newline \"$1\"\n- else\n-\techo_without_newline \"$(printf '%s\\n' \"$1\" | sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//')\"\n- fi\n+\tif test -z \"$2\"; then\n+\t\techo_without_newline \"$1\"\n+\telse\n+\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n+\tfi\n }\n \n+remove_timestamp () {\n+\tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n+}\n+\n+\n run_tests () {\n type=$1\n sha1=$2\n-- \ngitgitgadget\n\n"},{"id":"448602","messageId":"8edf80574b8dd72ca8489df6e480cf6820b98070.1645023740.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v8.git.git.1645023740.gitgitgadget@gmail.com","subject":"[PATCH v8 4/4] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T15:02:20Z","receivedAt":"2022-02-16T15:02:39Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] LF\n [arg1] [arg2] LF\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush LF\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n--batch-command also will not allow (B) to flush to stdout until a flush\nis received.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents LF\ninfo LF\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes an argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\nflush LF\ninfo LF\nflush LF\n\nWhen used without --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\ninfo LF\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 42 +++++++++-\n builtin/cat-file.c | 140 ++++++++++++++++++++++++++++++++-\n t/README | 3 +\n t/t1006-cat-file.sh | 107 ++++++++++++++++++++++++-\n 4 files changed, 286 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..70c5b4f12d1 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,33 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+--batch-command=::\n+\tEnter a command mode that reads commands and arguments from stdin. May\n+\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n+\tcase of `--textconv` or `--filters`, the input lines also need to specify\n+\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n+\tfor details.\n++\n+`--batch-command` recognizes the following commands:\n++\n+--\n+contents ::\n+\tPrint object contents for object reference ``. This corresponds to\n+\tthe output of `--batch`.\n+\n+info ::\n+\tPrint object info for object reference ``. This corresponds to the\n+\toutput of `--batch-check`.\n+\n+flush::\n+\tUsed with `--buffer` to execute all preceding commands that were issued\n+\tsince the beginning or since the last flush was issued. When `--buffer`\n+\tis used, no output will come until a `flush` is issued. When `--buffer`\n+\tis not used, commands are flushed each time without issuing `flush`.\n+--\n++\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\n@@ -110,7 +137,7 @@ OPTIONS\n \tthat a process can interactively read and write from\n \t`cat-file`. With this option, the output uses normal stdio\n \tbuffering; this is much more efficient when invoking\n-\t`--batch-check` on a large number of objects.\n+\t`--batch-check` or `--batch-command` on a large number of objects.\n \n --unordered::\n \tWhen `--batch-all-objects` is in use, visit objects in an\n@@ -202,6 +229,13 @@ from stdin, one per line, and print information about them. By default,\n the whole line is considered as an object, as if it were fed to\n linkgit:git-rev-parse[1].\n \n+When `--batch-command` is given, `cat-file` will read commands from stdin,\n+one per line, and print information based on the command given. With\n+`--batch-command`, the `info` command followed by an object will print\n+information about the object the same way `--batch-check` would, and the\n+`contents` command followed by an object prints contents in the same way\n+`--batch` would.\n+\n You can specify the information shown for each object by using a custom\n ``. The `` is copied literally to stdout for each\n object, with placeholders of the form `%(atom)` expanded, followed by a\n@@ -237,9 +271,9 @@ newline. The available atoms are:\n If no format is specified, the default format is `%(objectname)\n %(objecttype) %(objectsize)`.\n \n-If `--batch` is specified, the object information is followed by the\n-object contents (consisting of `%(objectsize)` bytes), followed by a\n-newline.\n+If `--batch` is specified, or if `--batch-command` is used with the `contents`\n+command, the object information is followed by the object contents (consisting\n+of `%(objectsize)` bytes), followed by a newline.\n \n For example, `--batch` without a custom format would produce:\n \ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5e38af82af1..39d486dd737 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -20,6 +20,7 @@\n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n \tBATCH_MODE_INFO,\n+\tBATCH_MODE_QUEUE_AND_DISPATCH,\n };\n \n struct batch_options {\n@@ -513,6 +514,132 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tchar *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_CONTENTS;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_INFO;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+/* Loop through each queued_cmd, dispatch the function, free the\n+ * memory associated with line so the cmd array can be reused.\n+ * Callers must set nr back to 0 in order to reuse the cmd array.\n+ */\n+static void dispatch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmd,\n+\t\tint nr)\n+{\n+\tint i;\n+\n+\tif (!opt->buffer_output)\n+\t\tdie(_(\"flush is only for --buffer mode\"));\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\t\tFREE_AND_NULL(cmd[i].line);\n+\t}\n+\n+\tfflush(stdout);\n+}\n+\n+static const struct parse_cmd {\n+\tconst char *name;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+\t{ \"flush\", NULL, 0},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *queued_cmd = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p = NULL, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].name, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args) {\n+\t\t\t\tif (*cmd_end != ' ')\n+\t\t\t\t\tdie(_(\"%s requires arguments\"),\n+\t\t\t\t\t commands[i].name);\n+\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\t} else if (*cmd_end) {\n+\t\t\t\tdie(_(\"%s takes no arguments\"),\n+\t\t\t\t commands[i].name);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!strcmp(cmd->name, \"flush\")) {\n+\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\t\t\tnr = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n+\t\tcall.fn = cmd->fn;\n+\t\tcall.line = xstrdup_or_null(p);\n+\t\tqueued_cmd[nr++] = call;\n+\t}\n+\n+\tif (opt->buffer_output &&\n+\t nr &&\n+\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n+\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\n+\tfree(queued_cmd);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -595,6 +722,10 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -613,6 +744,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -645,6 +777,8 @@ static int batch_option_callback(const struct option *opt,\n \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n \telse if (!strcmp(opt->long_name, \"batch-check\"))\n \t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse if (!strcmp(opt->long_name, \"batch-command\"))\n+\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n \telse\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \n@@ -666,7 +800,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\tN_(\"git cat-file \"),\n \t\tN_(\"git cat-file (-e | -p) \"),\n \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] \"),\n-\t\tN_(\"git cat-file (--batch | --batch-check) [--batch-all-objects]\\n\"\n+\t\tN_(\"git cat-file (--batch | --batch-check | --batch-command) [--batch-all-objects]\\n\"\n \t\t \" [--buffer] [--follow-symlinks] [--unordered]\\n\"\n \t\t \" [--textconv | --filters]\"),\n \t\tN_(\"git cat-file (--textconv | --filters)\\n\"\n@@ -695,6 +829,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/README b/t/README\nindex f48e0542cdc..bcd813b0c59 100644\n--- a/t/README\n+++ b/t/README\n@@ -472,6 +472,9 @@ a test and then fails then the whole test run will abort. This can help to make\n sure the expected tests are executed and not silently skipped when their\n dependency breaks or is simply not present in a new environment.\n \n+GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file\n+--batch-command from flushing to output on exit.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 2d52851dadc..74f0e36b69e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -182,6 +182,24 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ for opt in --buffer --no-buffer\n+ do\n+\ttest -z \"$content\" ||\n+\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n+\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n+\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_write_lines \"info $sha1\" \\\n+\t\t| git cat-file --batch-command $opt >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+ done\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -229,6 +247,22 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success '--batch-command --buffer with flush for blob info' '\n+\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command --buffer without flush for blob info' '\n+\ttouch output &&\n+\ttest_write_lines \"info $hello_sha1\" | \\\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >>output &&\n+\ttest_must_be_empty output\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -272,7 +306,7 @@ test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -378,6 +412,42 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+test_expect_success '--batch-command with multiple info calls gives correct format' '\n+\tcat >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$tree_sha1 tree $tree_size\n+\t$commit_sha1 commit $commit_size\n+\t$tag_sha1 tag $tag_size\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"info $hello_sha1\"\\\n+\t\"info $tree_sha1\"\\\n+\t\"info $commit_sha1\"\\\n+\t\"info $tag_sha1\"\\\n+\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command with multiple command calls gives correct format' '\n+\tremove_timestamp >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$hello_content\n+\t$commit_sha1 commit $commit_size\n+\t$commit_content\n+\t$tag_sha1 tag $tag_size\n+\t$tag_content\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"contents $hello_sha1\"\\\n+\t\"contents $commit_sha1\"\\\n+\t\"contents $tag_sha1\"\\\n+\t\"contents deadbeef\"\\\n+\t\"flush\" | git cat-file --batch-command --buffer | remove_timestamp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -968,5 +1038,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command missing arguments' '\n+\techo \"info\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*info requires arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"448619","messageId":"xmqqwnhuu410.fsf@gitster.g","threadId":"57365","inReplyTo":"8edf80574b8dd72ca8489df6e480cf6820b98070.1645023740.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 4/4] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-16T17:15:07Z","receivedAt":"2022-02-16T17:15:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> +/* Loop through each queued_cmd, dispatch the function, free the\n> + * memory associated with line so the cmd array can be reused.\n> + * Callers must set nr back to 0 in order to reuse the cmd array.\n> + */\n\nStyle.\n\n\t/*\n\t * Our multi-line comments look like this;\n\t * slash-asterisk on the opening line and\n\t * asterisk-slash on the closing line sit\n\t * on their own lines.\n\t */\n\nI am not sure if that is an accurate description. The caller does\nnot have to reuse the cmd array. The only thing they have to know\nabout this function is that the .line member is freed after the\nfunction is done with it, so they do not have to free the member\nthemselves.\n\nAs it seems that this code structure and division of responsibility\nbetween the caller and the callee is confusing even to the author of\nthis code, it may make sense to make the caller responsible for\nfreeing.\n\nThen the caller becomes\n\n> +\t\tif (!strcmp(cmd->name, \"flush\")) {\n> +\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n\n\t\t\tfor (i = 0; i < nr; i++)\n\t\t\t\tfree(queued_cmd[i].line);\n\n> +\t\t\tnr = 0;\n> +\t\t\tcontinue;\n> +\t\t}\n\nwhich is not too bad. And then we'd free the array itself at the\nend ...\n\n> ...\n> +\t\tcall.line = xstrdup_or_null(p);\n> +\t\tqueued_cmd[nr++] = call;\n> +\t}\n> +\n> +\tif (opt->buffer_output &&\n> +\t nr &&\n> +\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n> +\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n> +\n> +\tfree(queued_cmd);\n\n... which may be easier to see what is going on.\n\nThanks.\n"},{"id":"448621","messageId":"CAPig+cQKRytC0xQBCMss0S6tAoVft8XhcPCws9x4PMx_STfcHA@mail.gmail.com","threadId":"57365","inReplyTo":"xmqqwnhuu410.fsf@gitster.g","subject":"Re: [PATCH v8 4/4] cat-file: add --batch-command mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-16T17:25:41Z","receivedAt":"2022-02-16T17:25:55Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Feb 16, 2022 at 12:15 PM Junio C Hamano wrote:\n> \"John Cai via GitGitGadget\" writes:\n> As it seems that this code structure and division of responsibility\n> between the caller and the callee is confusing even to the author of\n> this code, it may make sense to make the caller responsible for\n> freeing.\n>\n> Then the caller becomes\n>\n> > + if (!strcmp(cmd->name, \"flush\")) {\n> > + dispatch_calls(opt, output, data, queued_cmd, nr);\n>\n> for (i = 0; i < nr; i++)\n> free(queued_cmd[i].line);\n>\n> > + nr = 0;\n> > + continue;\n> > + }\n>\n> which is not too bad. And then we'd free the array itself at the\n> end ...\n>\n> > ...\n> > + call.line = xstrdup_or_null(p);\n> > + queued_cmd[nr++] = call;\n> > + }\n> > +\n> > + if (opt->buffer_output &&\n> > + nr &&\n> > + !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n> > + dispatch_calls(opt, output, data, queued_cmd, nr);\n> > +\n> > + free(queued_cmd);\n>\n> ... which may be easier to see what is going on.\n\nI agree that it is easier to see what is going on when the caller is\nresponsible for freeing `line`. It _may_ make sense to factor out the\nfree-line-loop to a separate function since the caller will have to do\nso after both calls to dispatch_calls(), not just the one inside the\nstrbuf_getline() loop, but also the one after the loop. A separate\nfunction might be overkill for this two-line loop; on the other hand,\nit may clue in future readers that resource management needs to be\ntaken into account.\n"},{"id":"448631","messageId":"D2B37C47-677F-430A-9917-441D9FD647BC@gmail.com","threadId":"57365","inReplyTo":"CAPig+cQKRytC0xQBCMss0S6tAoVft8XhcPCws9x4PMx_STfcHA@mail.gmail.com","subject":"Re: [PATCH v8 4/4] cat-file: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-16T20:30:16Z","receivedAt":"2022-02-16T20:30:24Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"\n\nOn 16 Feb 2022, at 12:25, Eric Sunshine wrote:\n\n> On Wed, Feb 16, 2022 at 12:15 PM Junio C Hamano wrote:\n>> \"John Cai via GitGitGadget\" writes:\n>> As it seems that this code structure and division of responsibility\n>> between the caller and the callee is confusing even to the author of\n>> this code, it may make sense to make the caller responsible for\n>> freeing.\n>>\n>> Then the caller becomes\n>>\n>>> + if (!strcmp(cmd->name, \"flush\")) {\n>>> + dispatch_calls(opt, output, data, queued_cmd, nr);\n>>\n>> for (i = 0; i < nr; i++)\n>> free(queued_cmd[i].line);\n>>\n>>> + nr = 0;\n>>> + continue;\n>>> + }\n>>\n>> which is not too bad. And then we'd free the array itself at the\n>> end ...\n>>\n>>> ...\n>>> + call.line = xstrdup_or_null(p);\n>>> + queued_cmd[nr++] = call;\n>>> + }\n>>> +\n>>> + if (opt->buffer_output &&\n>>> + nr &&\n>>> + !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n>>> + dispatch_calls(opt, output, data, queued_cmd, nr);\n>>> +\n>>> + free(queued_cmd);\n>>\n>> ... which may be easier to see what is going on.\n>\n> I agree that it is easier to see what is going on when the caller is\n> responsible for freeing `line`. It _may_ make sense to factor out the\n> free-line-loop to a separate function since the caller will have to do\n> so after both calls to dispatch_calls(), not just the one inside the\n> strbuf_getline() loop, but also the one after the loop. A separate\n> function might be overkill for this two-line loop; on the other hand,\n> it may clue in future readers that resource management needs to be\n> taken into account.\n\nBoth of these suggestions sound good to me. Thanks for the help down to these details. These are\nall valuable learning points!\n"},{"id":"448632","messageId":"pull.1212.v9.git.git.1645045157.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v8.git.git.1645023740.gitgitgadget@gmail.com","subject":"[PATCH v9 0/4] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T20:59:13Z","receivedAt":"2022-02-16T20:59:23Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has four parts:\n\n 1. preparation patch to rename a variable\n 2. adding an enum to keep track of batch modes\n 3. add a remove_timestamp() helper that takes stdin and removes timestamps\n 4. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v8\n\n * have caller free line through a helper function for the sake of\n separation of concerns\n\nChanges since v7\n\n * revert back to having caller set nr to 0\n * add comment before dispatch_calls to clarify usage of helper\n * rename prefix->name\n\nChanges since v6 (thanks to Eric's feedback)\n\n * allow command parsing logic to handle the case of flush as well\n * fixed documentation by adding --batch-command to the synopsis and\n adjusting tick marks\n * set nr=0 within helper function\n\nChanges since v5\n\n * replaced flush tests that used fifo pipes to using a GIT_TEST_ env\n variable to control whether or not --batch-command flushes on exit.\n * added remove_timestamp helper in tests.\n * added documentation to show format can be used with --batch-command\n\nChanges since v4\n\n * added Phillip's suggested test for testing flush. This should have\n addressed the flaky test that was hanging. I tested it on my side and\n wasn't able to reproduce the deadlock.\n * plugged some holes in the logic that parsed the command and arguments,\n thanks to Eric's feedback\n * fixed verbiage in commit messages per Christian's feedback\n * clarified places in documentation that should mention --batch-command per\n Eric's feedback\n\nChanges since v3 (thanks to Junio's feedback):\n\n * added cascading logic in batch_options_callback()\n * free memory for queued call input lines\n * do not throw error when flushing an empty queue\n * renamed cmds array to singular queued_cmd\n * fixed flaky test that failed --stress\n\nChanges since v2:\n\n * added enum to keep track of which batch mode we are in (thanks to Junio's\n feedback)\n * fixed array allocation logic (thanks to Junio's feedback)\n * added code to flush commands when --batch-commands receives an EOF and\n exits (thanks to Phillip's feedback)\n * fixed docs formatting (thanks to Jonathan's feedback)\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (4):\n cat-file: rename cmdmode to transform_mode\n cat-file: introduce batch_mode enum to replace print_contents\n cat-file: add remove_timestamp helper\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 42 +++++++-\n builtin/cat-file.c | 181 ++++++++++++++++++++++++++++++---\n t/README | 3 +\n t/t1006-cat-file.sh | 122 ++++++++++++++++++++--\n 4 files changed, 326 insertions(+), 22 deletions(-)\n\n\nbase-commit: b80121027d1247a0754b3cc46897fee75c050b44\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v9\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v9\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v8:\n\n 1: fa6294387ab = 1: 76d6e4fe517 cat-file: rename cmdmode to transform_mode\n 2: 1a038097bfc = 2: 12084a335cb cat-file: introduce batch_mode enum to replace print_contents\n 3: 486ee847816 = 3: bf74b6cd75b cat-file: add remove_timestamp helper\n 4: 8edf80574b8 ! 4: dbe194f8a85 cat-file: add --batch-command mode\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\tbatch_one_object(line, output, opt, data);\n +}\n +\n -+/* Loop through each queued_cmd, dispatch the function, free the\n -+ * memory associated with line so the cmd array can be reused.\n -+ * Callers must set nr back to 0 in order to reuse the cmd array.\n -+ */\n +static void dispatch_calls(struct batch_options *opt,\n +\t\tstruct strbuf *output,\n +\t\tstruct expand_data *data,\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\tif (!opt->buffer_output)\n +\t\tdie(_(\"flush is only for --buffer mode\"));\n +\n -+\tfor (i = 0; i < nr; i++) {\n ++\tfor (i = 0; i < nr; i++)\n +\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n -+\t\tFREE_AND_NULL(cmd[i].line);\n -+\t}\n +\n +\tfflush(stdout);\n +}\n +\n ++static void free_cmds(struct queued_cmd *cmd, int nr)\n ++{\n ++\tint i;\n ++\n ++\tfor (i = 0; i < nr; i++)\n ++\t\tFREE_AND_NULL(cmd[i].line);\n ++}\n ++\n ++\n +static const struct parse_cmd {\n +\tconst char *name;\n +\tparse_cmd_fn_t fn;\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\n +\t\tif (!strcmp(cmd->name, \"flush\")) {\n +\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n ++\t\t\tfree_cmds(queued_cmd, nr);\n +\t\t\tnr = 0;\n +\t\t\tcontinue;\n +\t\t}\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\n +\tif (opt->buffer_output &&\n +\t nr &&\n -+\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0))\n ++\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0)) {\n +\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n ++\t\tfree_cmds(queued_cmd, nr);\n ++\t}\n +\n +\tfree(queued_cmd);\n +\tstrbuf_release(&input);\n @@ builtin/cat-file.c: static int batch_objects(struct batch_options *opt)\n +\t\tbatch_objects_command(opt, &output, &data);\n +\t\tgoto cleanup;\n +\t}\n ++\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n\n-- \ngitgitgadget\n"},{"id":"448633","messageId":"76d6e4fe517ed78dc86ca70d6da5aa67f06bd5cf.1645045157.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v9.git.git.1645045157.gitgitgadget@gmail.com","subject":"[PATCH v9 1/4] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T20:59:14Z","receivedAt":"2022-02-16T20:59:25Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIn the next patch, we will add an enum on the batch_options struct that\nindicates which type of batch operation will be used: --batch,\n--batch-check and the soon to be --batch-command that will read\ncommands from stdin. --batch-command mode might get confused with\nthe cmdmode flag.\n\nThere is value in renaming cmdmode in any case. cmdmode refers to how\nthe result output of the blob will be transformed, either according to\n--filter or --textconv. So transform_mode is a more descriptive name\nfor the flag.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"448634","messageId":"12084a335cb9128db5d122dea9f74edc86b64595.1645045157.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v9.git.git.1645045157.gitgitgadget@gmail.com","subject":"[PATCH v9 2/4] cat-file: introduce batch_mode enum to replace print_contents","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T20:59:15Z","receivedAt":"2022-02-16T20:59:32Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nA future patch introduces a new --batch-command flag. Including --batch\nand --batch-check, we will have a total of three batch modes. print_contents\nis the only boolean on the batch_options sturct used to distinguish\nbetween the different modes. This makes the code harder to read.\n\nTo reduce potential confusion, replace print_contents with an enum to\nhelp readability and clarity.\n\nHelped-by: Junio C Hamano \nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..5e38af82af1 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,10 +17,15 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n+enum batch_mode {\n+\tBATCH_MODE_CONTENTS,\n+\tBATCH_MODE_INFO,\n+};\n+\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tint print_contents;\n+\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -635,7 +640,14 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\n+\tif (!strcmp(opt->long_name, \"batch\"))\n+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n+\telse if (!strcmp(opt->long_name, \"batch-check\"))\n+\t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse\n+\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n+\n \tbo->format = arg;\n \n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"448635","messageId":"bf74b6cd75bd886c1b5954693beeaccdfd2e51ec.1645045157.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v9.git.git.1645045157.gitgitgadget@gmail.com","subject":"[PATCH v9 3/4] cat-file: add remove_timestamp helper","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T20:59:16Z","receivedAt":"2022-02-16T20:59:33Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nmaybe_remove_timestamp() takes arguments, but it would be useful to have\na function that reads from stdin and strips the timestamp. This would\nallow tests to pipe data into a function to remove timestamps, and\nwouldn't have to always assign a variable. This is especially helpful\nwhen the data is multiple lines.\n\nKeep maybe_remove_timestamp() the same, but add a remove_timestamp\nhelper that reads from stdin.\n\nThe tests in the next patch will make use of this.\n\nSigned-off-by: John Cai \n---\n t/t1006-cat-file.sh | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..2d52851dadc 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -105,13 +105,18 @@ strlen () {\n }\n \n maybe_remove_timestamp () {\n- if test -z \"$2\"; then\n- echo_without_newline \"$1\"\n- else\n-\techo_without_newline \"$(printf '%s\\n' \"$1\" | sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//')\"\n- fi\n+\tif test -z \"$2\"; then\n+\t\techo_without_newline \"$1\"\n+\telse\n+\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n+\tfi\n }\n \n+remove_timestamp () {\n+\tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n+}\n+\n+\n run_tests () {\n type=$1\n sha1=$2\n-- \ngitgitgadget\n\n"},{"id":"448636","messageId":"dbe194f8a8592dd67d6ba3a11cb7b1c81b789cec.1645045157.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v9.git.git.1645045157.gitgitgadget@gmail.com","subject":"[PATCH v9 4/4] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-16T20:59:17Z","receivedAt":"2022-02-16T20:59:34Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] LF\n [arg1] [arg2] LF\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush LF\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n--batch-command also will not allow (B) to flush to stdout until a flush\nis received.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents LF\ninfo LF\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes an argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\nflush LF\ninfo LF\nflush LF\n\nWhen used without --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\ninfo LF\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 42 +++++++++-\n builtin/cat-file.c | 147 ++++++++++++++++++++++++++++++++-\n t/README | 3 +\n t/t1006-cat-file.sh | 107 +++++++++++++++++++++++-\n 4 files changed, 293 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..70c5b4f12d1 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,33 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+--batch-command=::\n+\tEnter a command mode that reads commands and arguments from stdin. May\n+\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n+\tcase of `--textconv` or `--filters`, the input lines also need to specify\n+\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n+\tfor details.\n++\n+`--batch-command` recognizes the following commands:\n++\n+--\n+contents ::\n+\tPrint object contents for object reference ``. This corresponds to\n+\tthe output of `--batch`.\n+\n+info ::\n+\tPrint object info for object reference ``. This corresponds to the\n+\toutput of `--batch-check`.\n+\n+flush::\n+\tUsed with `--buffer` to execute all preceding commands that were issued\n+\tsince the beginning or since the last flush was issued. When `--buffer`\n+\tis used, no output will come until a `flush` is issued. When `--buffer`\n+\tis not used, commands are flushed each time without issuing `flush`.\n+--\n++\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\n@@ -110,7 +137,7 @@ OPTIONS\n \tthat a process can interactively read and write from\n \t`cat-file`. With this option, the output uses normal stdio\n \tbuffering; this is much more efficient when invoking\n-\t`--batch-check` on a large number of objects.\n+\t`--batch-check` or `--batch-command` on a large number of objects.\n \n --unordered::\n \tWhen `--batch-all-objects` is in use, visit objects in an\n@@ -202,6 +229,13 @@ from stdin, one per line, and print information about them. By default,\n the whole line is considered as an object, as if it were fed to\n linkgit:git-rev-parse[1].\n \n+When `--batch-command` is given, `cat-file` will read commands from stdin,\n+one per line, and print information based on the command given. With\n+`--batch-command`, the `info` command followed by an object will print\n+information about the object the same way `--batch-check` would, and the\n+`contents` command followed by an object prints contents in the same way\n+`--batch` would.\n+\n You can specify the information shown for each object by using a custom\n ``. The `` is copied literally to stdout for each\n object, with placeholders of the form `%(atom)` expanded, followed by a\n@@ -237,9 +271,9 @@ newline. The available atoms are:\n If no format is specified, the default format is `%(objectname)\n %(objecttype) %(objectsize)`.\n \n-If `--batch` is specified, the object information is followed by the\n-object contents (consisting of `%(objectsize)` bytes), followed by a\n-newline.\n+If `--batch` is specified, or if `--batch-command` is used with the `contents`\n+command, the object information is followed by the object contents (consisting\n+of `%(objectsize)` bytes), followed by a newline.\n \n For example, `--batch` without a custom format would produce:\n \ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5e38af82af1..3dc960e9f85 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -20,6 +20,7 @@\n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n \tBATCH_MODE_INFO,\n+\tBATCH_MODE_QUEUE_AND_DISPATCH,\n };\n \n struct batch_options {\n@@ -513,6 +514,138 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tchar *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_CONTENTS;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_INFO;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void dispatch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmd,\n+\t\tint nr)\n+{\n+\tint i;\n+\n+\tif (!opt->buffer_output)\n+\t\tdie(_(\"flush is only for --buffer mode\"));\n+\n+\tfor (i = 0; i < nr; i++)\n+\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\n+\tfflush(stdout);\n+}\n+\n+static void free_cmds(struct queued_cmd *cmd, int nr)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < nr; i++)\n+\t\tFREE_AND_NULL(cmd[i].line);\n+}\n+\n+\n+static const struct parse_cmd {\n+\tconst char *name;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+\t{ \"flush\", NULL, 0},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *queued_cmd = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p = NULL, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].name, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args) {\n+\t\t\t\tif (*cmd_end != ' ')\n+\t\t\t\t\tdie(_(\"%s requires arguments\"),\n+\t\t\t\t\t commands[i].name);\n+\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\t} else if (*cmd_end) {\n+\t\t\t\tdie(_(\"%s takes no arguments\"),\n+\t\t\t\t commands[i].name);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!strcmp(cmd->name, \"flush\")) {\n+\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\t\t\tfree_cmds(queued_cmd, nr);\n+\t\t\tnr = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n+\t\tcall.fn = cmd->fn;\n+\t\tcall.line = xstrdup_or_null(p);\n+\t\tqueued_cmd[nr++] = call;\n+\t}\n+\n+\tif (opt->buffer_output &&\n+\t nr &&\n+\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0)) {\n+\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\t\tfree_cmds(queued_cmd, nr);\n+\t}\n+\n+\tfree(queued_cmd);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -595,6 +728,11 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n+\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -613,6 +751,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -645,6 +784,8 @@ static int batch_option_callback(const struct option *opt,\n \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n \telse if (!strcmp(opt->long_name, \"batch-check\"))\n \t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse if (!strcmp(opt->long_name, \"batch-command\"))\n+\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n \telse\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \n@@ -666,7 +807,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\tN_(\"git cat-file \"),\n \t\tN_(\"git cat-file (-e | -p) \"),\n \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] \"),\n-\t\tN_(\"git cat-file (--batch | --batch-check) [--batch-all-objects]\\n\"\n+\t\tN_(\"git cat-file (--batch | --batch-check | --batch-command) [--batch-all-objects]\\n\"\n \t\t \" [--buffer] [--follow-symlinks] [--unordered]\\n\"\n \t\t \" [--textconv | --filters]\"),\n \t\tN_(\"git cat-file (--textconv | --filters)\\n\"\n@@ -695,6 +836,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/README b/t/README\nindex f48e0542cdc..bcd813b0c59 100644\n--- a/t/README\n+++ b/t/README\n@@ -472,6 +472,9 @@ a test and then fails then the whole test run will abort. This can help to make\n sure the expected tests are executed and not silently skipped when their\n dependency breaks or is simply not present in a new environment.\n \n+GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file\n+--batch-command from flushing to output on exit.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 2d52851dadc..74f0e36b69e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -182,6 +182,24 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ for opt in --buffer --no-buffer\n+ do\n+\ttest -z \"$content\" ||\n+\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n+\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n+\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_write_lines \"info $sha1\" \\\n+\t\t| git cat-file --batch-command $opt >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+ done\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n@@ -229,6 +247,22 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success '--batch-command --buffer with flush for blob info' '\n+\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command --buffer without flush for blob info' '\n+\ttouch output &&\n+\ttest_write_lines \"info $hello_sha1\" | \\\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >>output &&\n+\ttest_must_be_empty output\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -272,7 +306,7 @@ test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -378,6 +412,42 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+test_expect_success '--batch-command with multiple info calls gives correct format' '\n+\tcat >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$tree_sha1 tree $tree_size\n+\t$commit_sha1 commit $commit_size\n+\t$tag_sha1 tag $tag_size\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"info $hello_sha1\"\\\n+\t\"info $tree_sha1\"\\\n+\t\"info $commit_sha1\"\\\n+\t\"info $tag_sha1\"\\\n+\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command with multiple command calls gives correct format' '\n+\tremove_timestamp >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$hello_content\n+\t$commit_sha1 commit $commit_size\n+\t$commit_content\n+\t$tag_sha1 tag $tag_size\n+\t$tag_content\n+\tdeadbeef missing\n+\tEOF\n+\n+\ttest_write_lines \"contents $hello_sha1\"\\\n+\t\"contents $commit_sha1\"\\\n+\t\"contents $tag_sha1\"\\\n+\t\"contents deadbeef\"\\\n+\t\"flush\" | git cat-file --batch-command --buffer | remove_timestamp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -968,5 +1038,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command missing arguments' '\n+\techo \"info\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*info requires arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"448762","messageId":"095278d4-217b-7d97-5364-9cd491b34ed0@gmail.com","threadId":"57365","inReplyTo":"dbe194f8a8592dd67d6ba3a11cb7b1c81b789cec.1645045157.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v9 4/4] cat-file: add --batch-command mode","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-18T11:26:10Z","receivedAt":"2022-02-18T11:26:18Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John\n\nThis is looking good. I think the only thing that is missing (and which \nI should have realized earlier) is that there are no tests for valid or \ninvalid format arguments to --batch-command. I haven't checked but there \nmust be some other tests in the t1006 that we can piggy back on to add \nthat. I've left some stylistic comments below but I don't fell strongly \nabout them apart from the README comment so please don't feel obliged to \nact on them, it's looking pretty good as is.\n\nBest Wishes\n\nPhillip\n\nOn 16/02/2022 20:59, John Cai via GitGitGadget wrote:\n> From: John Cai \n> \n> Add a new flag --batch-command that accepts commands and arguments\n> from stdin, similar to git-update-ref --stdin.\n> \n> At GitLab, we use a pair of long running cat-file processes when\n> accessing object content. One for iterating over object metadata with\n> --batch-check, and the other to grab object contents with --batch.\n> \n> However, if we had --batch-command, we wouldn't need to keep both\n> processes around, and instead just have one --batch-command process\n> where we can flip between getting object info, and getting object\n> contents. Since we have a pair of cat-file processes per repository,\n> this means we can get rid of roughly half of long lived git cat-file\n> processes. Given there are many repositories being accessed at any given\n> time, this can lead to huge savings.\n> \n> git cat-file --batch-command\n> \n> will enter an interactive command mode whereby the user can enter in\n> commands and their arguments that get queued in memory:\n> \n> [arg1] [arg2] LF\n> [arg1] [arg2] LF\n> \n> When --buffer mode is used, commands will be queued in memory until a\n> flush command is issued that execute them:\n> \n> flush LF\n> \n> The reason for a flush command is that when a consumer process (A)\n> talks to a git cat-file process (B) and interactively writes to and\n> reads from it in --buffer mode, (A) needs to be able to control when\n> the buffer is flushed to stdout.\n> \n> Currently, from (A)'s perspective, the only way is to either\n> \n> 1. kill (B)'s process\n> 2. send an invalid object to stdin.\n> \n> 1. is not ideal from a performance perspective as it will require\n> spawning a new cat-file process each time, and 2. is hacky and not a\n> good long term solution.\n> \n> With this mechanism of queueing up commands and letting (A) issue a\n> flush command, process (A) can control when the buffer is flushed and\n> can guarantee it will receive all of the output when in --buffer mode.\n> --batch-command also will not allow (B) to flush to stdout until a flush\n> is received.\n> \n> This patch adds the basic structure for adding command which can be\n> extended in the future to add more commands. It also adds the following\n> two commands (on top of the flush command):\n> \n> contents LF\n> info LF\n> \n> The contents command takes an argument and prints out the object\n> contents.\n> \n> The info command takes an argument and prints out the object\n> metadata.\n> \n> These can be used in the following way with --buffer:\n> \n> info LF\n> contents LF\n> contents LF\n> info LF\n> flush LF\n> info LF\n> flush LF\n> \n> When used without --buffer:\n> \n> info LF\n> contents LF\n> contents LF\n> info LF\n> info LF\n> \n> Helped-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: John Cai \n> ---\n> Documentation/git-cat-file.txt | 42 +++++++++-\n> builtin/cat-file.c | 147 ++++++++++++++++++++++++++++++++-\n> t/README | 3 +\n> t/t1006-cat-file.sh | 107 +++++++++++++++++++++++-\n> 4 files changed, 293 insertions(+), 6 deletions(-)\n> \n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> index bef76f4dd06..70c5b4f12d1 100644\n> --- a/Documentation/git-cat-file.txt\n> +++ b/Documentation/git-cat-file.txt\n> @@ -96,6 +96,33 @@ OPTIONS\n> \tneed to specify the path, separated by whitespace. See the\n> \tsection `BATCH OUTPUT` below for details.\n> \n> +--batch-command::\n> +--batch-command=::\n> +\tEnter a command mode that reads commands and arguments from stdin. May\n> +\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n> +\tcase of `--textconv` or `--filters`, the input lines also need to specify\n> +\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n> +\tfor details.\n> ++\n> +`--batch-command` recognizes the following commands:\n> ++\n> +--\n> +contents ::\n> +\tPrint object contents for object reference ``. This corresponds to\n> +\tthe output of `--batch`.\n> +\n> +info ::\n> +\tPrint object info for object reference ``. This corresponds to the\n> +\toutput of `--batch-check`.\n> +\n> +flush::\n> +\tUsed with `--buffer` to execute all preceding commands that were issued\n> +\tsince the beginning or since the last flush was issued. When `--buffer`\n> +\tis used, no output will come until a `flush` is issued. When `--buffer`\n> +\tis not used, commands are flushed each time without issuing `flush`.\n> +--\n> ++\n> +\n> --batch-all-objects::\n> \tInstead of reading a list of objects on stdin, perform the\n> \trequested batch operation on all objects in the repository and\n> @@ -110,7 +137,7 @@ OPTIONS\n> \tthat a process can interactively read and write from\n> \t`cat-file`. With this option, the output uses normal stdio\n> \tbuffering; this is much more efficient when invoking\n> -\t`--batch-check` on a large number of objects.\n> +\t`--batch-check` or `--batch-command` on a large number of objects.\n> \n> --unordered::\n> \tWhen `--batch-all-objects` is in use, visit objects in an\n> @@ -202,6 +229,13 @@ from stdin, one per line, and print information about them. By default,\n> the whole line is considered as an object, as if it were fed to\n> linkgit:git-rev-parse[1].\n> \n> +When `--batch-command` is given, `cat-file` will read commands from stdin,\n> +one per line, and print information based on the command given. With\n> +`--batch-command`, the `info` command followed by an object will print\n> +information about the object the same way `--batch-check` would, and the\n> +`contents` command followed by an object prints contents in the same way\n> +`--batch` would.\n> +\n> You can specify the information shown for each object by using a custom\n> ``. The `` is copied literally to stdout for each\n> object, with placeholders of the form `%(atom)` expanded, followed by a\n> @@ -237,9 +271,9 @@ newline. The available atoms are:\n> If no format is specified, the default format is `%(objectname)\n> %(objecttype) %(objectsize)`.\n> \n> -If `--batch` is specified, the object information is followed by the\n> -object contents (consisting of `%(objectsize)` bytes), followed by a\n> -newline.\n> +If `--batch` is specified, or if `--batch-command` is used with the `contents`\n> +command, the object information is followed by the object contents (consisting\n> +of `%(objectsize)` bytes), followed by a newline.\n> \n> For example, `--batch` without a custom format would produce:\n> \n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 5e38af82af1..3dc960e9f85 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -20,6 +20,7 @@\n> enum batch_mode {\n> \tBATCH_MODE_CONTENTS,\n> \tBATCH_MODE_INFO,\n> +\tBATCH_MODE_QUEUE_AND_DISPATCH,\n> };\n> \n> struct batch_options {\n> @@ -513,6 +514,138 @@ static int batch_unordered_packed(const struct object_id *oid,\n> \t\t\t\t data);\n> }\n> \n> +typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n> +\t\t\t struct strbuf *, struct expand_data *);\n> +\n> +struct queued_cmd {\n> +\tparse_cmd_fn_t fn;\n> +\tchar *line;\n> +};\n> +\n> +static void parse_cmd_contents(struct batch_options *opt,\n> +\t\t\t const char *line,\n> +\t\t\t struct strbuf *output,\n> +\t\t\t struct expand_data *data)\n> +{\n> +\topt->batch_mode = BATCH_MODE_CONTENTS;\n> +\tbatch_one_object(line, output, opt, data);\n> +}\n> +\n> +static void parse_cmd_info(struct batch_options *opt,\n> +\t\t\t const char *line,\n> +\t\t\t struct strbuf *output,\n> +\t\t\t struct expand_data *data)\n> +{\n> +\topt->batch_mode = BATCH_MODE_INFO;\n> +\tbatch_one_object(line, output, opt, data);\n> +}\n> +\n> +static void dispatch_calls(struct batch_options *opt,\n> +\t\tstruct strbuf *output,\n> +\t\tstruct expand_data *data,\n> +\t\tstruct queued_cmd *cmd,\n> +\t\tint nr)\n> +{\n> +\tint i;\n> +\n> +\tif (!opt->buffer_output)\n> +\t\tdie(_(\"flush is only for --buffer mode\"));\n> +\n> +\tfor (i = 0; i < nr; i++)\n> +\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n> +\n> +\tfflush(stdout);\n> +}\n> +\n> +static void free_cmds(struct queued_cmd *cmd, int nr)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < nr; i++)\n> +\t\tFREE_AND_NULL(cmd[i].line);\n> +}\n> +\n> +\n> +static const struct parse_cmd {\n> +\tconst char *name;\n> +\tparse_cmd_fn_t fn;\n> +\tunsigned takes_args;\n> +} commands[] = {\n> +\t{ \"contents\", parse_cmd_contents, 1},\n> +\t{ \"info\", parse_cmd_info, 1},\n> +\t{ \"flush\", NULL, 0},\n> +};\n> +\n> +static void batch_objects_command(struct batch_options *opt,\n> +\t\t\t\t struct strbuf *output,\n> +\t\t\t\t struct expand_data *data)\n> +{\n> +\tstruct strbuf input = STRBUF_INIT;\n> +\tstruct queued_cmd *queued_cmd = NULL;\n> +\tsize_t alloc = 0, nr = 0;\n> +\n> +\twhile (!strbuf_getline(&input, stdin)) {\n> +\t\tint i;\n> +\t\tconst struct parse_cmd *cmd = NULL;\n> +\t\tconst char *p = NULL, *cmd_end;\n> +\t\tstruct queued_cmd call = {0};\n> +\n> +\t\tif (!input.len)\n> +\t\t\tdie(_(\"empty command in input\"));\n> +\t\tif (isspace(*input.buf))\n> +\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n> +\n> +\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n> +\t\t\tif (!skip_prefix(input.buf, commands[i].name, &cmd_end))\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\tcmd = &commands[i];\n> +\t\t\tif (cmd->takes_args) {\n> +\t\t\t\tif (*cmd_end != ' ')\n> +\t\t\t\t\tdie(_(\"%s requires arguments\"),\n> +\t\t\t\t\t commands[i].name);\n> +\n> +\t\t\t\tp = cmd_end + 1;\n> +\t\t\t} else if (*cmd_end) {\n> +\t\t\t\tdie(_(\"%s takes no arguments\"),\n> +\t\t\t\t commands[i].name);\n> +\t\t\t}\n> +\n> +\t\t\tbreak;\n> +\t\t}\n> +\n> +\t\tif (!cmd)\n> +\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n> +\n> +\t\tif (!strcmp(cmd->name, \"flush\")) {\n> +\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n> +\t\t\tfree_cmds(queued_cmd, nr);\n> +\t\t\tnr = 0;\n\nIt'd be nice if free_cmds() zeroed nr for us rather than having to \nremember to do it separately as the two are intimately linked.\n\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (!opt->buffer_output) {\n> +\t\t\tcmd->fn(opt, p, output, data);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n> +\t\tcall.fn = cmd->fn;\n> +\t\tcall.line = xstrdup_or_null(p);\n> +\t\tqueued_cmd[nr++] = call;\n\nI found this a bit confusing to follow with all the \"continue\"s for me \nit would be easier to follow if this was written as\n\tif (!strcmp(cmd->name, \"flush\") {\n\t\t...\n\t} else if (!opt->buffer_output) {\n\t\t...\n\t} else {\n\t\tALLOC_GROW ...\n\t}\n\n> +\t}\n> +\n> +\tif (opt->buffer_output &&\n> +\t nr &&\n> +\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0)) {\n> +\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n> +\t\tfree_cmds(queued_cmd, nr);\n> +\t}\n> +\n> +\tfree(queued_cmd);\n> +\tstrbuf_release(&input);\n> +}\n> +\n> static int batch_objects(struct batch_options *opt)\n> {\n> \tstruct strbuf input = STRBUF_INIT;\n> @@ -595,6 +728,11 @@ static int batch_objects(struct batch_options *opt)\n> \tsave_warning = warn_on_object_refname_ambiguity;\n> \twarn_on_object_refname_ambiguity = 0;\n> \n> +\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n> +\t\tbatch_objects_command(opt, &output, &data);\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> \twhile (strbuf_getline(&input, stdin) != EOF) {\n> \t\tif (data.split_on_whitespace) {\n> \t\t\t/*\n> @@ -613,6 +751,7 @@ static int batch_objects(struct batch_options *opt)\n> \t\tbatch_one_object(input.buf, &output, opt, &data);\n> \t}\n> \n> + cleanup:\n> \tstrbuf_release(&input);\n> \tstrbuf_release(&output);\n> \twarn_on_object_refname_ambiguity = save_warning;\n> @@ -645,6 +784,8 @@ static int batch_option_callback(const struct option *opt,\n> \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n> \telse if (!strcmp(opt->long_name, \"batch-check\"))\n> \t\tbo->batch_mode = BATCH_MODE_INFO;\n> +\telse if (!strcmp(opt->long_name, \"batch-command\"))\n> +\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n> \telse\n> \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n> \n> @@ -666,7 +807,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n> \t\tN_(\"git cat-file \"),\n> \t\tN_(\"git cat-file (-e | -p) \"),\n> \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] \"),\n> -\t\tN_(\"git cat-file (--batch | --batch-check) [--batch-all-objects]\\n\"\n> +\t\tN_(\"git cat-file (--batch | --batch-check | --batch-command) [--batch-all-objects]\\n\"\n> \t\t \" [--buffer] [--follow-symlinks] [--unordered]\\n\"\n> \t\t \" [--textconv | --filters]\"),\n> \t\tN_(\"git cat-file (--textconv | --filters)\\n\"\n> @@ -695,6 +836,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n> \t\t\tN_(\"like --batch, but don't emit \"),\n> \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> \t\t\tbatch_option_callback),\n> +\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n> +\t\t\tN_(\"read commands from stdin\"),\n> +\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> +\t\t\tbatch_option_callback),\n> \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n> \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n> \t\t/* Batch-specific options */\n> diff --git a/t/README b/t/README\n> index f48e0542cdc..bcd813b0c59 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -472,6 +472,9 @@ a test and then fails then the whole test run will abort. This can help to make\n> sure the expected tests are executed and not silently skipped when their\n> dependency breaks or is simply not present in a new environment.\n> \n> +GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file\n> +--batch-command from flushing to output on exit.\n\nI don't think you need to document this here. Looking at the other \nvariables this is a list of things one can set to change the behavior of \nthe tests when they are run. GIT_TEST_CAT_FILE_NO_FLUSH is not in that \ncategory - we don't want anyone setting it when they run the tests, it's \njust an implementation detail.\n\n> +\n> Naming Tests\n> ------------\n> \n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 2d52851dadc..74f0e36b69e 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -182,6 +182,24 @@ $content\"\n> \ttest_cmp expect actual\n> '\n> \n> + for opt in --buffer --no-buffer\n> + do\n> +\ttest -z \"$content\" ||\n> +\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n> +\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n> +\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n> +\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n> +\t\ttest_cmp expect actual\n> +\t'\n> +\n> +\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n> +\t\techo \"$sha1 $type $size\" >expect &&\n> +\t\ttest_write_lines \"info $sha1\" \\\n> +\t\t| git cat-file --batch-command $opt >actual &&\n> +\t\ttest_cmp expect actual\n> +\t'\n> + done\n> +\n> test_expect_success \"custom --batch-check format\" '\n> \techo \"$type $sha1\" >expect &&\n> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n> @@ -229,6 +247,22 @@ test_expect_success \"setup\" '\n> \n> run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n> \n> +test_expect_success '--batch-command --buffer with flush for blob info' '\n> +\techo \"$hello_sha1 blob $hello_size\" >expect &&\n> +\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n\nYou don't need a '\\' after a '|', however it might be better to use the \nstyle from the tests above where the '|' is on the beginning of the next \nline.\n\n> +\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n> +\tgit cat-file --batch-command --buffer >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success '--batch-command --buffer without flush for blob info' '\n> +\ttouch output &&\n> +\ttest_write_lines \"info $hello_sha1\" | \\\n> +\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n> +\tgit cat-file --batch-command --buffer >>output &&\n> +\ttest_must_be_empty output\n> +'\n> +\n> test_expect_success '--batch-check without %(rest) considers whole line' '\n> \techo \"$hello_sha1 blob $hello_size\" >expect &&\n> \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n> @@ -272,7 +306,7 @@ test_expect_success \\\n> \"Reach a blob from a tag pointing to it\" \\\n> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n> \n> -for batch in batch batch-check\n> +for batch in batch batch-check batch-command\n> do\n> for opt in t s e p\n> do\n> @@ -378,6 +412,42 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n> '\n> \n> +test_expect_success '--batch-command with multiple info calls gives correct format' '\n> +\tcat >expect <<-EOF &&\n> +\t$hello_sha1 blob $hello_size\n> +\t$tree_sha1 tree $tree_size\n> +\t$commit_sha1 commit $commit_size\n> +\t$tag_sha1 tag $tag_size\n> +\tdeadbeef missing\n> +\tEOF\n> +\n> +\ttest_write_lines \"info $hello_sha1\"\\\n> +\t\"info $tree_sha1\"\\\n> +\t\"info $commit_sha1\"\\\n> +\t\"info $tag_sha1\"\\\n> +\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n\nThis is quite noisy with all the \" and \\, using a here document instead \nwould match our usual style.\n\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success '--batch-command with multiple command calls gives correct format' '\n> +\tremove_timestamp >expect <<-EOF &&\n> +\t$hello_sha1 blob $hello_size\n> +\t$hello_content\n> +\t$commit_sha1 commit $commit_size\n> +\t$commit_content\n> +\t$tag_sha1 tag $tag_size\n> +\t$tag_content\n> +\tdeadbeef missing\n> +\tEOF\n> +\n> +\ttest_write_lines \"contents $hello_sha1\"\\\n> +\t\"contents $commit_sha1\"\\\n> +\t\"contents $tag_sha1\"\\\n> +\t\"contents deadbeef\"\\\n> +\t\"flush\" | git cat-file --batch-command --buffer | remove_timestamp >actual &&\n\nThis loses the exit code of the command we're trying to test, it would \nbe better to have\n\tgit cat-file ... >actual-raw &&\n\tremove_timestamp actual\n\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_expect_success 'setup blobs which are likely to delta' '\n> \ttest-tool genrandom foo 10240 >foo &&\n> \t{ cat foo && echo plus; } >foo-plus &&\n> @@ -968,5 +1038,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n> \techo \"$orig commit $orig_size\" >expect &&\n> \ttest_cmp expect actual\n> '\n> +test_expect_success 'batch-command empty command' '\n> +\techo \"\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*empty command in input.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command whitespace before command' '\n> +\techo \" info deadbeef\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*whitespace before command.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command unknown command' '\n> +\techo unknown_command >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*unknown command.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command missing arguments' '\n> +\techo \"info\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*info requires arguments.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush with arguments' '\n> +\techo \"flush arg\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n> +\tgrep \"^fatal:.*flush takes no arguments.*\" err\n> +'\n> +\n> +test_expect_success 'batch-command flush without --buffer' '\n> +\techo \"flush\" >cmd &&\n> +\ttest_expect_code 128 git cat-file --batch-command err &&\n> +\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n> +'\n> \n> test_done\n\n"},{"id":"448768","messageId":"ECBE48BF-156B-4D8A-9CDE-30AE84BF3630@gmail.com","threadId":"57365","inReplyTo":"095278d4-217b-7d97-5364-9cd491b34ed0@gmail.com","subject":"Re: [PATCH v9 4/4] cat-file: add --batch-command mode","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-18T16:53:56Z","receivedAt":"2022-02-18T16:54:02Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Phillip,\n\nOn 18 Feb 2022, at 6:26, Phillip Wood wrote:\n\n> Hi John\n>\n> This is looking good. I think the only thing that is missing (and which I should have realized earlier) is that there are no tests for valid or invalid format arguments to --batch-command. I haven't checked but there must be some other tests in the t1006 that we can piggy back on to add that. I've left some stylistic comments below but I don't fell strongly about them apart from the README comment so please don't feel obliged to act on them, it's looking pretty good as is.\n>\n> Best Wishes\n>\n> Phillip\n>\n> On 16/02/2022 20:59, John Cai via GitGitGadget wrote:\n>> From: John Cai \n>>\n>> Add a new flag --batch-command that accepts commands and arguments\n>> from stdin, similar to git-update-ref --stdin.\n>>\n>> At GitLab, we use a pair of long running cat-file processes when\n>> accessing object content. One for iterating over object metadata with\n>> --batch-check, and the other to grab object contents with --batch.\n>>\n>> However, if we had --batch-command, we wouldn't need to keep both\n>> processes around, and instead just have one --batch-command process\n>> where we can flip between getting object info, and getting object\n>> contents. Since we have a pair of cat-file processes per repository,\n>> this means we can get rid of roughly half of long lived git cat-file\n>> processes. Given there are many repositories being accessed at any given\n>> time, this can lead to huge savings.\n>>\n>> git cat-file --batch-command\n>>\n>> will enter an interactive command mode whereby the user can enter in\n>> commands and their arguments that get queued in memory:\n>>\n>> [arg1] [arg2] LF\n>> [arg1] [arg2] LF\n>>\n>> When --buffer mode is used, commands will be queued in memory until a\n>> flush command is issued that execute them:\n>>\n>> flush LF\n>>\n>> The reason for a flush command is that when a consumer process (A)\n>> talks to a git cat-file process (B) and interactively writes to and\n>> reads from it in --buffer mode, (A) needs to be able to control when\n>> the buffer is flushed to stdout.\n>>\n>> Currently, from (A)'s perspective, the only way is to either\n>>\n>> 1. kill (B)'s process\n>> 2. send an invalid object to stdin.\n>>\n>> 1. is not ideal from a performance perspective as it will require\n>> spawning a new cat-file process each time, and 2. is hacky and not a\n>> good long term solution.\n>>\n>> With this mechanism of queueing up commands and letting (A) issue a\n>> flush command, process (A) can control when the buffer is flushed and\n>> can guarantee it will receive all of the output when in --buffer mode.\n>> --batch-command also will not allow (B) to flush to stdout until a flush\n>> is received.\n>>\n>> This patch adds the basic structure for adding command which can be\n>> extended in the future to add more commands. It also adds the following\n>> two commands (on top of the flush command):\n>>\n>> contents LF\n>> info LF\n>>\n>> The contents command takes an argument and prints out the object\n>> contents.\n>>\n>> The info command takes an argument and prints out the object\n>> metadata.\n>>\n>> These can be used in the following way with --buffer:\n>>\n>> info LF\n>> contents LF\n>> contents LF\n>> info LF\n>> flush LF\n>> info LF\n>> flush LF\n>>\n>> When used without --buffer:\n>>\n>> info LF\n>> contents LF\n>> contents LF\n>> info LF\n>> info LF\n>>\n>> Helped-by: Ævar Arnfjörð Bjarmason \n>> Signed-off-by: John Cai \n>> ---\n>> Documentation/git-cat-file.txt | 42 +++++++++-\n>> builtin/cat-file.c | 147 ++++++++++++++++++++++++++++++++-\n>> t/README | 3 +\n>> t/t1006-cat-file.sh | 107 +++++++++++++++++++++++-\n>> 4 files changed, 293 insertions(+), 6 deletions(-)\n>>\n>> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n>> index bef76f4dd06..70c5b4f12d1 100644\n>> --- a/Documentation/git-cat-file.txt\n>> +++ b/Documentation/git-cat-file.txt\n>> @@ -96,6 +96,33 @@ OPTIONS\n>> \tneed to specify the path, separated by whitespace. See the\n>> \tsection `BATCH OUTPUT` below for details.\n>> +--batch-command::\n>> +--batch-command=::\n>> +\tEnter a command mode that reads commands and arguments from stdin. May\n>> +\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n>> +\tcase of `--textconv` or `--filters`, the input lines also need to specify\n>> +\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n>> +\tfor details.\n>> ++\n>> +`--batch-command` recognizes the following commands:\n>> ++\n>> +--\n>> +contents ::\n>> +\tPrint object contents for object reference ``. This corresponds to\n>> +\tthe output of `--batch`.\n>> +\n>> +info ::\n>> +\tPrint object info for object reference ``. This corresponds to the\n>> +\toutput of `--batch-check`.\n>> +\n>> +flush::\n>> +\tUsed with `--buffer` to execute all preceding commands that were issued\n>> +\tsince the beginning or since the last flush was issued. When `--buffer`\n>> +\tis used, no output will come until a `flush` is issued. When `--buffer`\n>> +\tis not used, commands are flushed each time without issuing `flush`.\n>> +--\n>> ++\n>> +\n>> --batch-all-objects::\n>> \tInstead of reading a list of objects on stdin, perform the\n>> \trequested batch operation on all objects in the repository and\n>> @@ -110,7 +137,7 @@ OPTIONS\n>> \tthat a process can interactively read and write from\n>> \t`cat-file`. With this option, the output uses normal stdio\n>> \tbuffering; this is much more efficient when invoking\n>> -\t`--batch-check` on a large number of objects.\n>> +\t`--batch-check` or `--batch-command` on a large number of objects.\n>> --unordered::\n>> \tWhen `--batch-all-objects` is in use, visit objects in an\n>> @@ -202,6 +229,13 @@ from stdin, one per line, and print information about them. By default,\n>> the whole line is considered as an object, as if it were fed to\n>> linkgit:git-rev-parse[1].\n>> +When `--batch-command` is given, `cat-file` will read commands from stdin,\n>> +one per line, and print information based on the command given. With\n>> +`--batch-command`, the `info` command followed by an object will print\n>> +information about the object the same way `--batch-check` would, and the\n>> +`contents` command followed by an object prints contents in the same way\n>> +`--batch` would.\n>> +\n>> You can specify the information shown for each object by using a custom\n>> ``. The `` is copied literally to stdout for each\n>> object, with placeholders of the form `%(atom)` expanded, followed by a\n>> @@ -237,9 +271,9 @@ newline. The available atoms are:\n>> If no format is specified, the default format is `%(objectname)\n>> %(objecttype) %(objectsize)`.\n>> -If `--batch` is specified, the object information is followed by the\n>> -object contents (consisting of `%(objectsize)` bytes), followed by a\n>> -newline.\n>> +If `--batch` is specified, or if `--batch-command` is used with the `contents`\n>> +command, the object information is followed by the object contents (consisting\n>> +of `%(objectsize)` bytes), followed by a newline.\n>> For example, `--batch` without a custom format would produce:\n>> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n>> index 5e38af82af1..3dc960e9f85 100644\n>> --- a/builtin/cat-file.c\n>> +++ b/builtin/cat-file.c\n>> @@ -20,6 +20,7 @@\n>> enum batch_mode {\n>> \tBATCH_MODE_CONTENTS,\n>> \tBATCH_MODE_INFO,\n>> +\tBATCH_MODE_QUEUE_AND_DISPATCH,\n>> };\n>> struct batch_options {\n>> @@ -513,6 +514,138 @@ static int batch_unordered_packed(const struct object_id *oid,\n>> \t\t\t\t data);\n>> }\n>> +typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n>> +\t\t\t struct strbuf *, struct expand_data *);\n>> +\n>> +struct queued_cmd {\n>> +\tparse_cmd_fn_t fn;\n>> +\tchar *line;\n>> +};\n>> +\n>> +static void parse_cmd_contents(struct batch_options *opt,\n>> +\t\t\t const char *line,\n>> +\t\t\t struct strbuf *output,\n>> +\t\t\t struct expand_data *data)\n>> +{\n>> +\topt->batch_mode = BATCH_MODE_CONTENTS;\n>> +\tbatch_one_object(line, output, opt, data);\n>> +}\n>> +\n>> +static void parse_cmd_info(struct batch_options *opt,\n>> +\t\t\t const char *line,\n>> +\t\t\t struct strbuf *output,\n>> +\t\t\t struct expand_data *data)\n>> +{\n>> +\topt->batch_mode = BATCH_MODE_INFO;\n>> +\tbatch_one_object(line, output, opt, data);\n>> +}\n>> +\n>> +static void dispatch_calls(struct batch_options *opt,\n>> +\t\tstruct strbuf *output,\n>> +\t\tstruct expand_data *data,\n>> +\t\tstruct queued_cmd *cmd,\n>> +\t\tint nr)\n>> +{\n>> +\tint i;\n>> +\n>> +\tif (!opt->buffer_output)\n>> +\t\tdie(_(\"flush is only for --buffer mode\"));\n>> +\n>> +\tfor (i = 0; i < nr; i++)\n>> +\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n>> +\n>> +\tfflush(stdout);\n>> +}\n>> +\n>> +static void free_cmds(struct queued_cmd *cmd, int nr)\n>> +{\n>> +\tint i;\n>> +\n>> +\tfor (i = 0; i < nr; i++)\n>> +\t\tFREE_AND_NULL(cmd[i].line);\n>> +}\n>> +\n>> +\n>> +static const struct parse_cmd {\n>> +\tconst char *name;\n>> +\tparse_cmd_fn_t fn;\n>> +\tunsigned takes_args;\n>> +} commands[] = {\n>> +\t{ \"contents\", parse_cmd_contents, 1},\n>> +\t{ \"info\", parse_cmd_info, 1},\n>> +\t{ \"flush\", NULL, 0},\n>> +};\n>> +\n>> +static void batch_objects_command(struct batch_options *opt,\n>> +\t\t\t\t struct strbuf *output,\n>> +\t\t\t\t struct expand_data *data)\n>> +{\n>> +\tstruct strbuf input = STRBUF_INIT;\n>> +\tstruct queued_cmd *queued_cmd = NULL;\n>> +\tsize_t alloc = 0, nr = 0;\n>> +\n>> +\twhile (!strbuf_getline(&input, stdin)) {\n>> +\t\tint i;\n>> +\t\tconst struct parse_cmd *cmd = NULL;\n>> +\t\tconst char *p = NULL, *cmd_end;\n>> +\t\tstruct queued_cmd call = {0};\n>> +\n>> +\t\tif (!input.len)\n>> +\t\t\tdie(_(\"empty command in input\"));\n>> +\t\tif (isspace(*input.buf))\n>> +\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n>> +\n>> +\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n>> +\t\t\tif (!skip_prefix(input.buf, commands[i].name, &cmd_end))\n>> +\t\t\t\tcontinue;\n>> +\n>> +\t\t\tcmd = &commands[i];\n>> +\t\t\tif (cmd->takes_args) {\n>> +\t\t\t\tif (*cmd_end != ' ')\n>> +\t\t\t\t\tdie(_(\"%s requires arguments\"),\n>> +\t\t\t\t\t commands[i].name);\n>> +\n>> +\t\t\t\tp = cmd_end + 1;\n>> +\t\t\t} else if (*cmd_end) {\n>> +\t\t\t\tdie(_(\"%s takes no arguments\"),\n>> +\t\t\t\t commands[i].name);\n>> +\t\t\t}\n>> +\n>> +\t\t\tbreak;\n>> +\t\t}\n>> +\n>> +\t\tif (!cmd)\n>> +\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n>> +\n>> +\t\tif (!strcmp(cmd->name, \"flush\")) {\n>> +\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n>> +\t\t\tfree_cmds(queued_cmd, nr);\n>> +\t\t\tnr = 0;\n>\n> It'd be nice if free_cmds() zeroed nr for us rather than having to remember to do it separately as the two are intimately linked.\n\nThis does feel cleaner. Before there was a version where I did this inside of\ndispatch_calls and there was feedback that this wasn't clean. But now that\nfree_cmds prepares the queued_cmd array for reuse, then it may make sense to do\nit inside. Though honestly from the back and forth around this, I'm not too sure\nwhat the best thing to do stylistically would be.\n\n>\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\tif (!opt->buffer_output) {\n>> +\t\t\tcmd->fn(opt, p, output, data);\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n>> +\t\tcall.fn = cmd->fn;\n>> +\t\tcall.line = xstrdup_or_null(p);\n>> +\t\tqueued_cmd[nr++] = call;\n>\n> I found this a bit confusing to follow with all the \"continue\"s for me it would be easier to follow if this was written as\n> if (!strcmp(cmd->name, \"flush\") {\n> \t...\n> } else if (!opt->buffer_output) {\n> \t...\n> } else {\n> \tALLOC_GROW ...\n> }\n\nGood point. I think this would be easier to follow.\n>\n>> +\t}\n>> +\n>> +\tif (opt->buffer_output &&\n>> +\t nr &&\n>> +\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0)) {\n>> +\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n>> +\t\tfree_cmds(queued_cmd, nr);\n>> +\t}\n>> +\n>> +\tfree(queued_cmd);\n>> +\tstrbuf_release(&input);\n>> +}\n>> +\n>> static int batch_objects(struct batch_options *opt)\n>> {\n>> \tstruct strbuf input = STRBUF_INIT;\n>> @@ -595,6 +728,11 @@ static int batch_objects(struct batch_options *opt)\n>> \tsave_warning = warn_on_object_refname_ambiguity;\n>> \twarn_on_object_refname_ambiguity = 0;\n>> +\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n>> +\t\tbatch_objects_command(opt, &output, &data);\n>> +\t\tgoto cleanup;\n>> +\t}\n>> +\n>> \twhile (strbuf_getline(&input, stdin) != EOF) {\n>> \t\tif (data.split_on_whitespace) {\n>> \t\t\t/*\n>> @@ -613,6 +751,7 @@ static int batch_objects(struct batch_options *opt)\n>> \t\tbatch_one_object(input.buf, &output, opt, &data);\n>> \t}\n>> + cleanup:\n>> \tstrbuf_release(&input);\n>> \tstrbuf_release(&output);\n>> \twarn_on_object_refname_ambiguity = save_warning;\n>> @@ -645,6 +784,8 @@ static int batch_option_callback(const struct option *opt,\n>> \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n>> \telse if (!strcmp(opt->long_name, \"batch-check\"))\n>> \t\tbo->batch_mode = BATCH_MODE_INFO;\n>> +\telse if (!strcmp(opt->long_name, \"batch-command\"))\n>> +\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n>> \telse\n>> \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n>> @@ -666,7 +807,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n>> \t\tN_(\"git cat-file \"),\n>> \t\tN_(\"git cat-file (-e | -p) \"),\n>> \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] \"),\n>> -\t\tN_(\"git cat-file (--batch | --batch-check) [--batch-all-objects]\\n\"\n>> +\t\tN_(\"git cat-file (--batch | --batch-check | --batch-command) [--batch-all-objects]\\n\"\n>> \t\t \" [--buffer] [--follow-symlinks] [--unordered]\\n\"\n>> \t\t \" [--textconv | --filters]\"),\n>> \t\tN_(\"git cat-file (--textconv | --filters)\\n\"\n>> @@ -695,6 +836,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n>> \t\t\tN_(\"like --batch, but don't emit \"),\n>> \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n>> \t\t\tbatch_option_callback),\n>> +\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n>> +\t\t\tN_(\"read commands from stdin\"),\n>> +\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n>> +\t\t\tbatch_option_callback),\n>> \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n>> \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n>> \t\t/* Batch-specific options */\n>> diff --git a/t/README b/t/README\n>> index f48e0542cdc..bcd813b0c59 100644\n>> --- a/t/README\n>> +++ b/t/README\n>> @@ -472,6 +472,9 @@ a test and then fails then the whole test run will abort. This can help to make\n>> sure the expected tests are executed and not silently skipped when their\n>> dependency breaks or is simply not present in a new environment.\n>> +GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file\n>> +--batch-command from flushing to output on exit.\n>\n> I don't think you need to document this here. Looking at the other variables this is a list of things one can set to change the behavior of the tests when they are run. GIT_TEST_CAT_FILE_NO_FLUSH is not in that category - we don't want anyone setting it when they run the tests, it's just an implementation detail.\n\nokay sounds good.\n\n>\n>> +\n>> Naming Tests\n>> ------------\n>> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n>> index 2d52851dadc..74f0e36b69e 100755\n>> --- a/t/t1006-cat-file.sh\n>> +++ b/t/t1006-cat-file.sh\n>> @@ -182,6 +182,24 @@ $content\"\n>> \ttest_cmp expect actual\n>> '\n>> + for opt in --buffer --no-buffer\n>> + do\n>> +\ttest -z \"$content\" ||\n>> +\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n>> +\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n>> +\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n>> +\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t'\n>> +\n>> +\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n>> +\t\techo \"$sha1 $type $size\" >expect &&\n>> +\t\ttest_write_lines \"info $sha1\" \\\n>> +\t\t| git cat-file --batch-command $opt >actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t'\n>> + done\n>> +\n>> test_expect_success \"custom --batch-check format\" '\n>> \techo \"$type $sha1\" >expect &&\n>> \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n>> @@ -229,6 +247,22 @@ test_expect_success \"setup\" '\n>> run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n>> +test_expect_success '--batch-command --buffer with flush for blob info' '\n>> +\techo \"$hello_sha1 blob $hello_size\" >expect &&\n>> +\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n>\n> You don't need a '\\' after a '|', however it might be better to use the style from the tests above where the '|' is on the beginning of the next line.\n\ngood to know. I'll adjust this.\n>\n>> +\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n>> +\tgit cat-file --batch-command --buffer >actual &&\n>> +\ttest_cmp expect actual\n>> +'\n>> +\n>> +test_expect_success '--batch-command --buffer without flush for blob info' '\n>> +\ttouch output &&\n>> +\ttest_write_lines \"info $hello_sha1\" | \\\n>> +\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n>> +\tgit cat-file --batch-command --buffer >>output &&\n>> +\ttest_must_be_empty output\n>> +'\n>> +\n>> test_expect_success '--batch-check without %(rest) considers whole line' '\n>> \techo \"$hello_sha1 blob $hello_size\" >expect &&\n>> \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n>> @@ -272,7 +306,7 @@ test_expect_success \\\n>> \"Reach a blob from a tag pointing to it\" \\\n>> \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n>> -for batch in batch batch-check\n>> +for batch in batch batch-check batch-command\n>> do\n>> for opt in t s e p\n>> do\n>> @@ -378,6 +412,42 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n>> \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n>> '\n>> +test_expect_success '--batch-command with multiple info calls gives correct format' '\n>> +\tcat >expect <<-EOF &&\n>> +\t$hello_sha1 blob $hello_size\n>> +\t$tree_sha1 tree $tree_size\n>> +\t$commit_sha1 commit $commit_size\n>> +\t$tag_sha1 tag $tag_size\n>> +\tdeadbeef missing\n>> +\tEOF\n>> +\n>> +\ttest_write_lines \"info $hello_sha1\"\\\n>> +\t\"info $tree_sha1\"\\\n>> +\t\"info $commit_sha1\"\\\n>> +\t\"info $tag_sha1\"\\\n>> +\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n>\n> This is quite noisy with all the \" and \\, using a here document instead would match our usual style.\n\nThat's a good suggestion. I can change this into a here doc\n\n>\n>> +\ttest_cmp expect actual\n>> +'\n>> +\n>> +test_expect_success '--batch-command with multiple command calls gives correct format' '\n>> +\tremove_timestamp >expect <<-EOF &&\n>> +\t$hello_sha1 blob $hello_size\n>> +\t$hello_content\n>> +\t$commit_sha1 commit $commit_size\n>> +\t$commit_content\n>> +\t$tag_sha1 tag $tag_size\n>> +\t$tag_content\n>> +\tdeadbeef missing\n>> +\tEOF\n>> +\n>> +\ttest_write_lines \"contents $hello_sha1\"\\\n>> +\t\"contents $commit_sha1\"\\\n>> +\t\"contents $tag_sha1\"\\\n>> +\t\"contents deadbeef\"\\\n>> +\t\"flush\" | git cat-file --batch-command --buffer | remove_timestamp >actual &&\n>\n> This loses the exit code of the command we're trying to test, it would be better to have\n> git cat-file ... >actual-raw &&\n> remove_timestamp actual\n>\n\nI hadn't considered this effect. Thanks for pointing that out!\n\n>> +\ttest_cmp expect actual\n>> +'\n>> +\n>> test_expect_success 'setup blobs which are likely to delta' '\n>> \ttest-tool genrandom foo 10240 >foo &&\n>> \t{ cat foo && echo plus; } >foo-plus &&\n>> @@ -968,5 +1038,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n>> \techo \"$orig commit $orig_size\" >expect &&\n>> \ttest_cmp expect actual\n>> '\n>> +test_expect_success 'batch-command empty command' '\n>> +\techo \"\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*empty command in input.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command whitespace before command' '\n>> +\techo \" info deadbeef\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*whitespace before command.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command unknown command' '\n>> +\techo unknown_command >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*unknown command.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command missing arguments' '\n>> +\techo \"info\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*info requires arguments.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command flush with arguments' '\n>> +\techo \"flush arg\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n>> +\tgrep \"^fatal:.*flush takes no arguments.*\" err\n>> +'\n>> +\n>> +test_expect_success 'batch-command flush without --buffer' '\n>> +\techo \"flush\" >cmd &&\n>> +\ttest_expect_code 128 git cat-file --batch-command err &&\n>> +\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n>> +'\n>> test_done\n"},{"id":"448774","messageId":"xmqqtucw2io4.fsf@gitster.g","threadId":"57365","inReplyTo":"095278d4-217b-7d97-5364-9cd491b34ed0@gmail.com","subject":"Re: [PATCH v9 4/4] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-18T17:23:07Z","receivedAt":"2022-02-18T17:23:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> This is looking good. I think the only thing that is missing (and\n> which I should have realized earlier) is that there are no tests for\n> valid or invalid format arguments to --batch-command. I haven't\n> checked but there must be some other tests in the t1006 that we can\n> piggy back on to add that. I've left some stylistic comments below but\n> I don't fell strongly about them apart from the README comment so\n> please don't feel obliged to act on them, it's looking pretty good as\n> is.\n\nYes, I agree that this is mostly good. I also agree with all the\npoints you raised in your review, including the stylistic ones.\n"},{"id":"448775","messageId":"xmqqpmnk2i8u.fsf@gitster.g","threadId":"57365","inReplyTo":"ECBE48BF-156B-4D8A-9CDE-30AE84BF3630@gmail.com","subject":"Re: [PATCH v9 4/4] cat-file: add --batch-command mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-18T17:32:17Z","receivedAt":"2022-02-18T17:32:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Cai writes:\n\n>>> +\t\tif (!strcmp(cmd->name, \"flush\")) {\n>>> +\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n>>> +\t\t\tfree_cmds(queued_cmd, nr);\n>>> +\t\t\tnr = 0;\n>>\n>> It'd be nice if free_cmds() zeroed nr for us rather than having to remember to do it separately as the two are intimately linked.\n>\n> This does feel cleaner. Before there was a version where I did this inside of\n> dispatch_calls and there was feedback that this wasn't clean. But now that\n> free_cmds prepares the queued_cmd array for reuse, then it may make sense to do\n> it inside. Though honestly from the back and forth around this, I'm not too sure\n> what the best thing to do stylistically would be.\n\nI am not sure about style, but at the semantic level, free_cmds()\nthat \"frees\" the queued_cmd by releasing the resources it holds and\nresets its counter to zero would be a more complete \"does one thing\nand one thing well\" helper function.\n\n>>> +test_expect_success '--batch-command --buffer with flush for blob info' '\n>>> +\techo \"$hello_sha1 blob $hello_size\" >expect &&\n>>> +\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n>>\n>> You don't need a '\\' after a '|', however it might be better to use the style from the tests above where the '|' is on the beginning of the next line.\n\nPlease don't do\n\n\tproducer \\\n\t| consumer\n\ninstead, write\n\n\tproducer |\n\tconsumer\n\nWith two fewer bytes, and is far more common, judging from the\noutput of\n\n $ git grep -e '^[ ]*| [A-Za-z]' t\n\ni.e. indent with whitespace or tab, pipe, space and alpha (i.e. the\nbeginning of the command, possibly a single-shot environment\nassignment).\n"},{"id":"448777","messageId":"76d6e4fe517ed78dc86ca70d6da5aa67f06bd5cf.1645208594.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v10.git.git.1645208594.gitgitgadget@gmail.com","subject":"[PATCH v10 1/4] cat-file: rename cmdmode to transform_mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-18T18:23:11Z","receivedAt":"2022-02-18T18:23:22Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIn the next patch, we will add an enum on the batch_options struct that\nindicates which type of batch operation will be used: --batch,\n--batch-check and the soon to be --batch-command that will read\ncommands from stdin. --batch-command mode might get confused with\nthe cmdmode flag.\n\nThere is value in renaming cmdmode in any case. cmdmode refers to how\nthe result output of the blob will be transformed, either according to\n--filter or --textconv. So transform_mode is a more descriptive name\nfor the flag.\n\nRename cmdmode to transform_mode in cat-file.c\n\nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 7b3f42950ec..5f015e71096 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -24,7 +24,7 @@ struct batch_options {\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n-\tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n };\n \n@@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n \t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n+\t\tif (opt->transform_mode) {\n \t\t\tchar *contents;\n \t\t\tunsigned long size;\n \n \t\t\tif (!data->rest)\n \t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n \n-\t\t\tif (opt->cmdmode == 'w') {\n+\t\t\tif (opt->transform_mode == 'w') {\n \t\t\t\tif (filter_object(data->rest, 0100644, oid,\n \t\t\t\t\t\t &contents, &size))\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n+\t\t\t} else if (opt->transform_mode == 'c') {\n \t\t\t\tenum object_type type;\n \t\t\t\tif (!textconv_object(the_repository,\n \t\t\t\t\t\t data->rest, 0100644, oid,\n@@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\t\t\tdie(\"could not convert '%s' %s\",\n \t\t\t\t\t oid_to_hex(oid), data->rest);\n \t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n+\t\t\t\tBUG(\"invalid transform_mode: %c\", opt->transform_mode);\n \t\t\tbatch_write(opt, contents, size);\n \t\t\tfree(contents);\n \t\t} else {\n@@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt)\n \tstrbuf_expand(&output, opt->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (opt->transform_mode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n@@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t/* Return early if we're in batch mode? */\n \tif (batch.enabled) {\n \t\tif (opt_cw)\n-\t\t\tbatch.cmdmode = opt;\n+\t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n \t\t\t\t usage, options, opt);\n-- \ngitgitgadget\n\n"},{"id":"448778","messageId":"12084a335cb9128db5d122dea9f74edc86b64595.1645208594.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v10.git.git.1645208594.gitgitgadget@gmail.com","subject":"[PATCH v10 2/4] cat-file: introduce batch_mode enum to replace print_contents","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-18T18:23:12Z","receivedAt":"2022-02-18T18:23:24Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nA future patch introduces a new --batch-command flag. Including --batch\nand --batch-check, we will have a total of three batch modes. print_contents\nis the only boolean on the batch_options sturct used to distinguish\nbetween the different modes. This makes the code harder to read.\n\nTo reduce potential confusion, replace print_contents with an enum to\nhelp readability and clarity.\n\nHelped-by: Junio C Hamano \nSigned-off-by: John Cai \n---\n builtin/cat-file.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5f015e71096..5e38af82af1 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -17,10 +17,15 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n \n+enum batch_mode {\n+\tBATCH_MODE_CONTENTS,\n+\tBATCH_MODE_INFO,\n+};\n+\n struct batch_options {\n \tint enabled;\n \tint follow_symlinks;\n-\tint print_contents;\n+\tenum batch_mode batch_mode;\n \tint buffer_output;\n \tint all_objects;\n \tint unordered;\n@@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name,\n \tstrbuf_addch(scratch, '\\n');\n \tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS) {\n \t\tprint_object_or_die(opt, data);\n \t\tbatch_write(opt, \"\\n\", 1);\n \t}\n@@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt)\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (opt->batch_mode == BATCH_MODE_CONTENTS)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n@@ -635,7 +640,14 @@ static int batch_option_callback(const struct option *opt,\n \t}\n \n \tbo->enabled = 1;\n-\tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n+\n+\tif (!strcmp(opt->long_name, \"batch\"))\n+\t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n+\telse if (!strcmp(opt->long_name, \"batch-check\"))\n+\t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse\n+\t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n+\n \tbo->format = arg;\n \n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"448779","messageId":"pull.1212.v10.git.git.1645208594.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v9.git.git.1645045157.gitgitgadget@gmail.com","subject":"[PATCH v10 0/4] Add cat-file --batch-command flag","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-18T18:23:10Z","receivedAt":"2022-02-18T18:23:25Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"The feature proposal of adding a command interface to cat-file was first\ndiscussed in [A]. In [B], Taylor expressed the need for a fuller proposal\nbefore moving forward with a new flag. An RFC was created [C] and the idea\nwas discussed more thoroughly, and overall it seemed like it was headed in\nthe right direction.\n\nThis patch series consolidates the feedback from these different threads.\n\nThis patch series has four parts:\n\n 1. preparation patch to rename a variable\n 2. adding an enum to keep track of batch modes\n 3. add a remove_timestamp() helper that takes stdin and removes timestamps\n 4. logic to handle --batch-command flag, adding contents, info, flush\n commands\n\nChanges since v9\n\n * add test to exercise format for batch-command\n * minor semantic improvements\n * removed README entry for environment variable used in test\n\nChanges since v8\n\n * have caller free line through a helper function for the sake of\n separation of concerns\n\nChanges since v7\n\n * revert back to having caller set nr to 0\n * add comment before dispatch_calls to clarify usage of helper\n * rename prefix->name\n\nChanges since v6 (thanks to Eric's feedback)\n\n * allow command parsing logic to handle the case of flush as well\n * fixed documentation by adding --batch-command to the synopsis and\n adjusting tick marks\n * set nr=0 within helper function\n\nChanges since v5\n\n * replaced flush tests that used fifo pipes to using a GIT_TEST_ env\n variable to control whether or not --batch-command flushes on exit.\n * added remove_timestamp helper in tests.\n * added documentation to show format can be used with --batch-command\n\nChanges since v4\n\n * added Phillip's suggested test for testing flush. This should have\n addressed the flaky test that was hanging. I tested it on my side and\n wasn't able to reproduce the deadlock.\n * plugged some holes in the logic that parsed the command and arguments,\n thanks to Eric's feedback\n * fixed verbiage in commit messages per Christian's feedback\n * clarified places in documentation that should mention --batch-command per\n Eric's feedback\n\nChanges since v3 (thanks to Junio's feedback):\n\n * added cascading logic in batch_options_callback()\n * free memory for queued call input lines\n * do not throw error when flushing an empty queue\n * renamed cmds array to singular queued_cmd\n * fixed flaky test that failed --stress\n\nChanges since v2:\n\n * added enum to keep track of which batch mode we are in (thanks to Junio's\n feedback)\n * fixed array allocation logic (thanks to Junio's feedback)\n * added code to flush commands when --batch-commands receives an EOF and\n exits (thanks to Phillip's feedback)\n * fixed docs formatting (thanks to Jonathan's feedback)\n\nChanges since v1:\n\n * simplified \"session\" mechanism. \"flush\" will execute all commands that\n were entered in since the last \"flush\" when in --buffer mode\n * when not in --buffer mode, each command is executed and flushed each time\n * rename cmdmode to transform_mode instead of just mode\n * simplified command parsing logic\n * changed rename of cmdmode to transform_mode\n * clarified verbiage in commit messages\n\nA. https://lore.kernel.org/git/xmqqk0hitnkc.fsf@gitster.g/ B.\nhttps://lore.kernel.org/git/YehomwNiIs0l83W7@nand.local/ C.\nhttps://lore.kernel.org/git/e75ba9ea-fdda-6e9f-4dd6-24190117d93b@gmail.com/\n\nJohn Cai (4):\n cat-file: rename cmdmode to transform_mode\n cat-file: introduce batch_mode enum to replace print_contents\n cat-file: add remove_timestamp helper\n cat-file: add --batch-command mode\n\n Documentation/git-cat-file.txt | 42 +++++++-\n builtin/cat-file.c | 178 ++++++++++++++++++++++++++++++---\n t/t1006-cat-file.sh | 135 +++++++++++++++++++++++--\n 3 files changed, 333 insertions(+), 22 deletions(-)\n\n\nbase-commit: b80121027d1247a0754b3cc46897fee75c050b44\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1212%2Fjohn-cai%2Fjc-cat-file-batch-command-v10\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1212/john-cai/jc-cat-file-batch-command-v10\nPull-Request: https://github.com/git/git/pull/1212\n\nRange-diff vs v9:\n\n 1: 76d6e4fe517 = 1: 76d6e4fe517 cat-file: rename cmdmode to transform_mode\n 2: 12084a335cb = 2: 12084a335cb cat-file: introduce batch_mode enum to replace print_contents\n 3: bf74b6cd75b = 3: bf74b6cd75b cat-file: add remove_timestamp helper\n 4: dbe194f8a85 ! 4: c6ea1214062 cat-file: add --batch-command mode\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\tfflush(stdout);\n +}\n +\n -+static void free_cmds(struct queued_cmd *cmd, int nr)\n ++static void free_cmds(struct queued_cmd *cmd, size_t *nr)\n +{\n -+\tint i;\n ++\tsize_t i;\n +\n -+\tfor (i = 0; i < nr; i++)\n ++\tfor (i = 0; i < *nr; i++)\n +\t\tFREE_AND_NULL(cmd[i].line);\n ++\n ++\t*nr = 0;\n +}\n +\n +\n @@ builtin/cat-file.c: static int batch_unordered_packed(const struct object_id *oi\n +\n +\t\tif (!strcmp(cmd->name, \"flush\")) {\n +\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n -+\t\t\tfree_cmds(queued_cmd, nr);\n -+\t\t\tnr = 0;\n -+\t\t\tcontinue;\n -+\t\t}\n -+\n -+\t\tif (!opt->buffer_output) {\n ++\t\t\tfree_cmds(queued_cmd, &nr);\n ++\t\t} else if (!opt->buffer_output) {\n +\t\t\tcmd->fn(opt, p, output, data);\n -+\t\t\tcontinue;\n ++\t\t} else {\n ++\t\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n ++\t\t\tcall.fn = cmd->fn;\n ++\t\t\tcall.line = xstrdup_or_null(p);\n ++\t\t\tqueued_cmd[nr++] = call;\n +\t\t}\n -+\n -+\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n -+\t\tcall.fn = cmd->fn;\n -+\t\tcall.line = xstrdup_or_null(p);\n -+\t\tqueued_cmd[nr++] = call;\n +\t}\n +\n +\tif (opt->buffer_output &&\n +\t nr &&\n +\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0)) {\n +\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n -+\t\tfree_cmds(queued_cmd, nr);\n ++\t\tfree_cmds(queued_cmd, &nr);\n +\t}\n +\n +\tfree(queued_cmd);\n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *pr\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\n \n - ## t/README ##\n -@@ t/README: a test and then fails then the whole test run will abort. This can help to make\n - sure the expected tests are executed and not silently skipped when their\n - dependency breaks or is simply not present in a new environment.\n - \n -+GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file\n -+--batch-command from flushing to output on exit.\n -+\n - Naming Tests\n - ------------\n - \n -\n ## t/t1006-cat-file.sh ##\n @@ t/t1006-cat-file.sh: $content\"\n \ttest_cmp expect actual\n @@ t/t1006-cat-file.sh: $content\"\n +\ttest -z \"$content\" ||\n +\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n +\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n -+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" \\\n -+\t\t| git cat-file --batch-command $opt)\" $no_ts >actual &&\n ++\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" |\n ++\t\tgit cat-file --batch-command $opt)\" $no_ts >actual &&\n +\t\ttest_cmp expect actual\n +\t'\n +\n +\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n +\t\techo \"$sha1 $type $size\" >expect &&\n -+\t\ttest_write_lines \"info $sha1\" \\\n -+\t\t| git cat-file --batch-command $opt >actual &&\n ++\t\ttest_write_lines \"info $sha1\" |\n ++\t\tgit cat-file --batch-command $opt >actual &&\n +\t\ttest_cmp expect actual\n +\t'\n + done\n @@ t/t1006-cat-file.sh: $content\"\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n + \ttest_cmp expect actual\n + '\n + \n ++ test_expect_success \"custom --batch-command format\" '\n ++\techo \"$type $sha1\" >expect &&\n ++\techo \"info $sha1\" | git cat-file --batch-command=\"%(objecttype) %(objectname)\" >actual &&\n ++\ttest_cmp expect actual\n ++ '\n ++\n + test_expect_success '--batch-check with %(rest)' '\n + \techo \"$type this is some extra content\" >expect &&\n + \techo \"$sha1 this is some extra content\" |\n @@ t/t1006-cat-file.sh: test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n +test_expect_success '--batch-command --buffer with flush for blob info' '\n +\techo \"$hello_sha1 blob $hello_size\" >expect &&\n -+\ttest_write_lines \"info $hello_sha1\" \"flush\" | \\\n ++\ttest_write_lines \"info $hello_sha1\" \"flush\" |\n +\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n +\tgit cat-file --batch-command --buffer >actual &&\n +\ttest_cmp expect actual\n @@ t/t1006-cat-file.sh: test_expect_success \"setup\" '\n +\n +test_expect_success '--batch-command --buffer without flush for blob info' '\n +\ttouch output &&\n -+\ttest_write_lines \"info $hello_sha1\" | \\\n ++\ttest_write_lines \"info $hello_sha1\" |\n +\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n +\tgit cat-file --batch-command --buffer >>output &&\n +\ttest_must_be_empty output\n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n +\tdeadbeef missing\n +\tEOF\n +\n -+\ttest_write_lines \"info $hello_sha1\"\\\n -+\t\"info $tree_sha1\"\\\n -+\t\"info $commit_sha1\"\\\n -+\t\"info $tag_sha1\"\\\n -+\t\"info deadbeef\" | git cat-file --batch-command --buffer >actual &&\n ++\tgit cat-file --batch-command --buffer >actual <<-EOF &&\n ++\tinfo $hello_sha1\n ++\tinfo $tree_sha1\n ++\tinfo $commit_sha1\n ++\tinfo $tag_sha1\n ++\tinfo deadbeef\n ++\tEOF\n ++\n +\ttest_cmp expect actual\n +'\n +\n @@ t/t1006-cat-file.sh: test_expect_success \"--batch-check with multiple sha1s give\n +\tdeadbeef missing\n +\tEOF\n +\n -+\ttest_write_lines \"contents $hello_sha1\"\\\n -+\t\"contents $commit_sha1\"\\\n -+\t\"contents $tag_sha1\"\\\n -+\t\"contents deadbeef\"\\\n -+\t\"flush\" | git cat-file --batch-command --buffer | remove_timestamp >actual &&\n ++\tgit cat-file --batch-command --buffer >actual_raw <<-EOF &&\n ++\tcontents $hello_sha1\n ++\tcontents $commit_sha1\n ++\tcontents $tag_sha1\n ++\tcontents deadbeef\n ++\tflush\n ++\tEOF\n ++\n ++\tremove_timestamp actual &&\n +\ttest_cmp expect actual\n +'\n +\n\n-- \ngitgitgadget\n"},{"id":"448780","messageId":"bf74b6cd75bd886c1b5954693beeaccdfd2e51ec.1645208594.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v10.git.git.1645208594.gitgitgadget@gmail.com","subject":"[PATCH v10 3/4] cat-file: add remove_timestamp helper","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-18T18:23:13Z","receivedAt":"2022-02-18T18:23:33Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nmaybe_remove_timestamp() takes arguments, but it would be useful to have\na function that reads from stdin and strips the timestamp. This would\nallow tests to pipe data into a function to remove timestamps, and\nwouldn't have to always assign a variable. This is especially helpful\nwhen the data is multiple lines.\n\nKeep maybe_remove_timestamp() the same, but add a remove_timestamp\nhelper that reads from stdin.\n\nThe tests in the next patch will make use of this.\n\nSigned-off-by: John Cai \n---\n t/t1006-cat-file.sh | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 145eee11df9..2d52851dadc 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -105,13 +105,18 @@ strlen () {\n }\n \n maybe_remove_timestamp () {\n- if test -z \"$2\"; then\n- echo_without_newline \"$1\"\n- else\n-\techo_without_newline \"$(printf '%s\\n' \"$1\" | sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//')\"\n- fi\n+\tif test -z \"$2\"; then\n+\t\techo_without_newline \"$1\"\n+\telse\n+\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n+\tfi\n }\n \n+remove_timestamp () {\n+\tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n+}\n+\n+\n run_tests () {\n type=$1\n sha1=$2\n-- \ngitgitgadget\n\n"},{"id":"448781","messageId":"c6ea1214062381e2c669959e9084adead5600510.1645208594.git.gitgitgadget@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v10.git.git.1645208594.gitgitgadget@gmail.com","subject":"[PATCH v10 4/4] cat-file: add --batch-command mode","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-18T18:23:14Z","receivedAt":"2022-02-18T18:23:36Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nAdd a new flag --batch-command that accepts commands and arguments\nfrom stdin, similar to git-update-ref --stdin.\n\nAt GitLab, we use a pair of long running cat-file processes when\naccessing object content. One for iterating over object metadata with\n--batch-check, and the other to grab object contents with --batch.\n\nHowever, if we had --batch-command, we wouldn't need to keep both\nprocesses around, and instead just have one --batch-command process\nwhere we can flip between getting object info, and getting object\ncontents. Since we have a pair of cat-file processes per repository,\nthis means we can get rid of roughly half of long lived git cat-file\nprocesses. Given there are many repositories being accessed at any given\ntime, this can lead to huge savings.\n\ngit cat-file --batch-command\n\nwill enter an interactive command mode whereby the user can enter in\ncommands and their arguments that get queued in memory:\n\n [arg1] [arg2] LF\n [arg1] [arg2] LF\n\nWhen --buffer mode is used, commands will be queued in memory until a\nflush command is issued that execute them:\n\nflush LF\n\nThe reason for a flush command is that when a consumer process (A)\ntalks to a git cat-file process (B) and interactively writes to and\nreads from it in --buffer mode, (A) needs to be able to control when\nthe buffer is flushed to stdout.\n\nCurrently, from (A)'s perspective, the only way is to either\n\n1. kill (B)'s process\n2. send an invalid object to stdin.\n\n1. is not ideal from a performance perspective as it will require\nspawning a new cat-file process each time, and 2. is hacky and not a\ngood long term solution.\n\nWith this mechanism of queueing up commands and letting (A) issue a\nflush command, process (A) can control when the buffer is flushed and\ncan guarantee it will receive all of the output when in --buffer mode.\n--batch-command also will not allow (B) to flush to stdout until a flush\nis received.\n\nThis patch adds the basic structure for adding command which can be\nextended in the future to add more commands. It also adds the following\ntwo commands (on top of the flush command):\n\ncontents LF\ninfo LF\n\nThe contents command takes an argument and prints out the object\ncontents.\n\nThe info command takes an argument and prints out the object\nmetadata.\n\nThese can be used in the following way with --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\nflush LF\ninfo LF\nflush LF\n\nWhen used without --buffer:\n\ninfo LF\ncontents LF\ncontents LF\ninfo LF\ninfo LF\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: John Cai \n---\n Documentation/git-cat-file.txt | 42 +++++++++-\n builtin/cat-file.c | 144 ++++++++++++++++++++++++++++++++-\n t/t1006-cat-file.sh | 120 ++++++++++++++++++++++++++-\n 3 files changed, 300 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex bef76f4dd06..70c5b4f12d1 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -96,6 +96,33 @@ OPTIONS\n \tneed to specify the path, separated by whitespace. See the\n \tsection `BATCH OUTPUT` below for details.\n \n+--batch-command::\n+--batch-command=::\n+\tEnter a command mode that reads commands and arguments from stdin. May\n+\tonly be combined with `--buffer`, `--textconv` or `--filters`. In the\n+\tcase of `--textconv` or `--filters`, the input lines also need to specify\n+\tthe path, separated by whitespace. See the section `BATCH OUTPUT` below\n+\tfor details.\n++\n+`--batch-command` recognizes the following commands:\n++\n+--\n+contents ::\n+\tPrint object contents for object reference ``. This corresponds to\n+\tthe output of `--batch`.\n+\n+info ::\n+\tPrint object info for object reference ``. This corresponds to the\n+\toutput of `--batch-check`.\n+\n+flush::\n+\tUsed with `--buffer` to execute all preceding commands that were issued\n+\tsince the beginning or since the last flush was issued. When `--buffer`\n+\tis used, no output will come until a `flush` is issued. When `--buffer`\n+\tis not used, commands are flushed each time without issuing `flush`.\n+--\n++\n+\n --batch-all-objects::\n \tInstead of reading a list of objects on stdin, perform the\n \trequested batch operation on all objects in the repository and\n@@ -110,7 +137,7 @@ OPTIONS\n \tthat a process can interactively read and write from\n \t`cat-file`. With this option, the output uses normal stdio\n \tbuffering; this is much more efficient when invoking\n-\t`--batch-check` on a large number of objects.\n+\t`--batch-check` or `--batch-command` on a large number of objects.\n \n --unordered::\n \tWhen `--batch-all-objects` is in use, visit objects in an\n@@ -202,6 +229,13 @@ from stdin, one per line, and print information about them. By default,\n the whole line is considered as an object, as if it were fed to\n linkgit:git-rev-parse[1].\n \n+When `--batch-command` is given, `cat-file` will read commands from stdin,\n+one per line, and print information based on the command given. With\n+`--batch-command`, the `info` command followed by an object will print\n+information about the object the same way `--batch-check` would, and the\n+`contents` command followed by an object prints contents in the same way\n+`--batch` would.\n+\n You can specify the information shown for each object by using a custom\n ``. The `` is copied literally to stdout for each\n object, with placeholders of the form `%(atom)` expanded, followed by a\n@@ -237,9 +271,9 @@ newline. The available atoms are:\n If no format is specified, the default format is `%(objectname)\n %(objecttype) %(objectsize)`.\n \n-If `--batch` is specified, the object information is followed by the\n-object contents (consisting of `%(objectsize)` bytes), followed by a\n-newline.\n+If `--batch` is specified, or if `--batch-command` is used with the `contents`\n+command, the object information is followed by the object contents (consisting\n+of `%(objectsize)` bytes), followed by a newline.\n \n For example, `--batch` without a custom format would produce:\n \ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5e38af82af1..e75e110302e 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -20,6 +20,7 @@\n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n \tBATCH_MODE_INFO,\n+\tBATCH_MODE_QUEUE_AND_DISPATCH,\n };\n \n struct batch_options {\n@@ -513,6 +514,135 @@ static int batch_unordered_packed(const struct object_id *oid,\n \t\t\t\t data);\n }\n \n+typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n+\t\t\t struct strbuf *, struct expand_data *);\n+\n+struct queued_cmd {\n+\tparse_cmd_fn_t fn;\n+\tchar *line;\n+};\n+\n+static void parse_cmd_contents(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_CONTENTS;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void parse_cmd_info(struct batch_options *opt,\n+\t\t\t const char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data)\n+{\n+\topt->batch_mode = BATCH_MODE_INFO;\n+\tbatch_one_object(line, output, opt, data);\n+}\n+\n+static void dispatch_calls(struct batch_options *opt,\n+\t\tstruct strbuf *output,\n+\t\tstruct expand_data *data,\n+\t\tstruct queued_cmd *cmd,\n+\t\tint nr)\n+{\n+\tint i;\n+\n+\tif (!opt->buffer_output)\n+\t\tdie(_(\"flush is only for --buffer mode\"));\n+\n+\tfor (i = 0; i < nr; i++)\n+\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\n+\tfflush(stdout);\n+}\n+\n+static void free_cmds(struct queued_cmd *cmd, size_t *nr)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < *nr; i++)\n+\t\tFREE_AND_NULL(cmd[i].line);\n+\n+\t*nr = 0;\n+}\n+\n+\n+static const struct parse_cmd {\n+\tconst char *name;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+\t{ \"flush\", NULL, 0},\n+};\n+\n+static void batch_objects_command(struct batch_options *opt,\n+\t\t\t\t struct strbuf *output,\n+\t\t\t\t struct expand_data *data)\n+{\n+\tstruct strbuf input = STRBUF_INIT;\n+\tstruct queued_cmd *queued_cmd = NULL;\n+\tsize_t alloc = 0, nr = 0;\n+\n+\twhile (!strbuf_getline(&input, stdin)) {\n+\t\tint i;\n+\t\tconst struct parse_cmd *cmd = NULL;\n+\t\tconst char *p = NULL, *cmd_end;\n+\t\tstruct queued_cmd call = {0};\n+\n+\t\tif (!input.len)\n+\t\t\tdie(_(\"empty command in input\"));\n+\t\tif (isspace(*input.buf))\n+\t\t\tdie(_(\"whitespace before command: '%s'\"), input.buf);\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(commands); i++) {\n+\t\t\tif (!skip_prefix(input.buf, commands[i].name, &cmd_end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tcmd = &commands[i];\n+\t\t\tif (cmd->takes_args) {\n+\t\t\t\tif (*cmd_end != ' ')\n+\t\t\t\t\tdie(_(\"%s requires arguments\"),\n+\t\t\t\t\t commands[i].name);\n+\n+\t\t\t\tp = cmd_end + 1;\n+\t\t\t} else if (*cmd_end) {\n+\t\t\t\tdie(_(\"%s takes no arguments\"),\n+\t\t\t\t commands[i].name);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!cmd)\n+\t\t\tdie(_(\"unknown command: '%s'\"), input.buf);\n+\n+\t\tif (!strcmp(cmd->name, \"flush\")) {\n+\t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\t\t\tfree_cmds(queued_cmd, &nr);\n+\t\t} else if (!opt->buffer_output) {\n+\t\t\tcmd->fn(opt, p, output, data);\n+\t\t} else {\n+\t\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n+\t\t\tcall.fn = cmd->fn;\n+\t\t\tcall.line = xstrdup_or_null(p);\n+\t\t\tqueued_cmd[nr++] = call;\n+\t\t}\n+\t}\n+\n+\tif (opt->buffer_output &&\n+\t nr &&\n+\t !git_env_bool(\"GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT\", 0)) {\n+\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n+\t\tfree_cmds(queued_cmd, &nr);\n+\t}\n+\n+\tfree(queued_cmd);\n+\tstrbuf_release(&input);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -595,6 +725,11 @@ static int batch_objects(struct batch_options *opt)\n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n \n+\tif (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) {\n+\t\tbatch_objects_command(opt, &output, &data);\n+\t\tgoto cleanup;\n+\t}\n+\n \twhile (strbuf_getline(&input, stdin) != EOF) {\n \t\tif (data.split_on_whitespace) {\n \t\t\t/*\n@@ -613,6 +748,7 @@ static int batch_objects(struct batch_options *opt)\n \t\tbatch_one_object(input.buf, &output, opt, &data);\n \t}\n \n+ cleanup:\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -645,6 +781,8 @@ static int batch_option_callback(const struct option *opt,\n \t\tbo->batch_mode = BATCH_MODE_CONTENTS;\n \telse if (!strcmp(opt->long_name, \"batch-check\"))\n \t\tbo->batch_mode = BATCH_MODE_INFO;\n+\telse if (!strcmp(opt->long_name, \"batch-command\"))\n+\t\tbo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH;\n \telse\n \t\tBUG(\"%s given to batch-option-callback\", opt->long_name);\n \n@@ -666,7 +804,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\tN_(\"git cat-file \"),\n \t\tN_(\"git cat-file (-e | -p) \"),\n \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] \"),\n-\t\tN_(\"git cat-file (--batch | --batch-check) [--batch-all-objects]\\n\"\n+\t\tN_(\"git cat-file (--batch | --batch-check | --batch-command) [--batch-all-objects]\\n\"\n \t\t \" [--buffer] [--follow-symlinks] [--unordered]\\n\"\n \t\t \" [--textconv | --filters]\"),\n \t\tN_(\"git cat-file (--textconv | --filters)\\n\"\n@@ -695,6 +833,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"like --batch, but don't emit \"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\tbatch_option_callback),\n+\t\tOPT_CALLBACK_F(0, \"batch-command\", &batch, N_(\"format\"),\n+\t\t\tN_(\"read commands from stdin\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\tbatch_option_callback),\n \t\tOPT_CMDMODE(0, \"batch-all-objects\", &opt,\n \t\t\t N_(\"with --batch[-check]: ignores stdin, batches all known objects\"), 'b'),\n \t\t/* Batch-specific options */\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 2d52851dadc..1b852076944 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -182,12 +182,36 @@ $content\"\n \ttest_cmp expect actual\n '\n \n+ for opt in --buffer --no-buffer\n+ do\n+\ttest -z \"$content\" ||\n+\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n+\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n+\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" |\n+\t\tgit cat-file --batch-command $opt)\" $no_ts >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"--batch-command $opt output of $type info is correct\" '\n+\t\techo \"$sha1 $type $size\" >expect &&\n+\t\ttest_write_lines \"info $sha1\" |\n+\t\tgit cat-file --batch-command $opt >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+ done\n+\n test_expect_success \"custom --batch-check format\" '\n \techo \"$type $sha1\" >expect &&\n \techo $sha1 | git cat-file --batch-check=\"%(objecttype) %(objectname)\" >actual &&\n \ttest_cmp expect actual\n '\n \n+ test_expect_success \"custom --batch-command format\" '\n+\techo \"$type $sha1\" >expect &&\n+\techo \"info $sha1\" | git cat-file --batch-command=\"%(objecttype) %(objectname)\" >actual &&\n+\ttest_cmp expect actual\n+ '\n+\n test_expect_success '--batch-check with %(rest)' '\n \techo \"$type this is some extra content\" >expect &&\n \techo \"$sha1 this is some extra content\" |\n@@ -229,6 +253,22 @@ test_expect_success \"setup\" '\n \n run_tests 'blob' $hello_sha1 $hello_size \"$hello_content\" \"$hello_content\"\n \n+test_expect_success '--batch-command --buffer with flush for blob info' '\n+\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\ttest_write_lines \"info $hello_sha1\" \"flush\" |\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command --buffer without flush for blob info' '\n+\ttouch output &&\n+\ttest_write_lines \"info $hello_sha1\" |\n+\tGIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \\\n+\tgit cat-file --batch-command --buffer >>output &&\n+\ttest_must_be_empty output\n+'\n+\n test_expect_success '--batch-check without %(rest) considers whole line' '\n \techo \"$hello_sha1 blob $hello_size\" >expect &&\n \tgit update-index --add --cacheinfo 100644 $hello_sha1 \"white space\" &&\n@@ -272,7 +312,7 @@ test_expect_success \\\n \"Reach a blob from a tag pointing to it\" \\\n \"test '$hello_content' = \\\"\\$(git cat-file blob $tag_sha1)\\\"\"\n \n-for batch in batch batch-check\n+for batch in batch batch-check batch-command\n do\n for opt in t s e p\n do\n@@ -378,6 +418,49 @@ test_expect_success \"--batch-check with multiple sha1s gives correct format\" '\n \"$(echo_without_newline \"$batch_check_input\" | git cat-file --batch-check)\"\n '\n \n+test_expect_success '--batch-command with multiple info calls gives correct format' '\n+\tcat >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$tree_sha1 tree $tree_size\n+\t$commit_sha1 commit $commit_size\n+\t$tag_sha1 tag $tag_size\n+\tdeadbeef missing\n+\tEOF\n+\n+\tgit cat-file --batch-command --buffer >actual <<-EOF &&\n+\tinfo $hello_sha1\n+\tinfo $tree_sha1\n+\tinfo $commit_sha1\n+\tinfo $tag_sha1\n+\tinfo deadbeef\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--batch-command with multiple command calls gives correct format' '\n+\tremove_timestamp >expect <<-EOF &&\n+\t$hello_sha1 blob $hello_size\n+\t$hello_content\n+\t$commit_sha1 commit $commit_size\n+\t$commit_content\n+\t$tag_sha1 tag $tag_size\n+\t$tag_content\n+\tdeadbeef missing\n+\tEOF\n+\n+\tgit cat-file --batch-command --buffer >actual_raw <<-EOF &&\n+\tcontents $hello_sha1\n+\tcontents $commit_sha1\n+\tcontents $tag_sha1\n+\tcontents deadbeef\n+\tflush\n+\tEOF\n+\n+\tremove_timestamp actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'setup blobs which are likely to delta' '\n \ttest-tool genrandom foo 10240 >foo &&\n \t{ cat foo && echo plus; } >foo-plus &&\n@@ -968,5 +1051,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace'\n \techo \"$orig commit $orig_size\" >expect &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'batch-command empty command' '\n+\techo \"\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*empty command in input.*\" err\n+'\n+\n+test_expect_success 'batch-command whitespace before command' '\n+\techo \" info deadbeef\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*whitespace before command.*\" err\n+'\n+\n+test_expect_success 'batch-command unknown command' '\n+\techo unknown_command >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*unknown command.*\" err\n+'\n+\n+test_expect_success 'batch-command missing arguments' '\n+\techo \"info\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*info requires arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush with arguments' '\n+\techo \"flush arg\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command --buffer err &&\n+\tgrep \"^fatal:.*flush takes no arguments.*\" err\n+'\n+\n+test_expect_success 'batch-command flush without --buffer' '\n+\techo \"flush\" >cmd &&\n+\ttest_expect_code 128 git cat-file --batch-command err &&\n+\tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n+'\n \n test_done\n-- \ngitgitgadget\n"},{"id":"448795","messageId":"xmqqy228yng0.fsf@gitster.g","threadId":"57365","inReplyTo":"pull.1212.v10.git.git.1645208594.gitgitgadget@gmail.com","subject":"Re: [PATCH v10 0/4] Add cat-file --batch-command flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-18T19:38:55Z","receivedAt":"2022-02-18T19:39:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> The feature proposal of adding a command interface to cat-file was first\n> discussed in [A]. In [B], Taylor expressed the need for a fuller proposal\n> before moving forward with a new flag. An RFC was created [C] and the idea\n> was discussed more thoroughly, and overall it seemed like it was headed in\n> the right direction.\n>\n> This patch series consolidates the feedback from these different threads.\n>\n> This patch series has four parts:\n>\n> 1. preparation patch to rename a variable\n> 2. adding an enum to keep track of batch modes\n> 3. add a remove_timestamp() helper that takes stdin and removes timestamps\n> 4. logic to handle --batch-command flag, adding contents, info, flush\n> commands\n>\n> Changes since v9\n>\n> * add test to exercise format for batch-command\n> * minor semantic improvements\n> * removed README entry for environment variable used in test\n\nBoth the changes relative to the previous round and relative to\n'master' look good to me.\n\nThanks.\n"},{"id":"448842","messageId":"220219.86a6ene548.gmgdl@evledraar.gmail.com","threadId":"57365","inReplyTo":"bf74b6cd75bd886c1b5954693beeaccdfd2e51ec.1645208594.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v10 3/4] cat-file: add remove_timestamp helper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-19T06:33:52Z","receivedAt":"2022-02-19T06:35:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Feb 18 2022, John Cai via GitGitGadget wrote:\n\n> From: John Cai \n>\n> maybe_remove_timestamp() takes arguments, but it would be useful to have\n> a function that reads from stdin and strips the timestamp. This would\n> allow tests to pipe data into a function to remove timestamps, and\n> wouldn't have to always assign a variable. This is especially helpful\n> when the data is multiple lines.\n>\n> Keep maybe_remove_timestamp() the same, but add a remove_timestamp\n> helper that reads from stdin.\n>\n> The tests in the next patch will make use of this.\n>\n> Signed-off-by: John Cai \n> ---\n> t/t1006-cat-file.sh | 15 ++++++++++-----\n> 1 file changed, 10 insertions(+), 5 deletions(-)\n>\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 145eee11df9..2d52851dadc 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -105,13 +105,18 @@ strlen () {\n> }\n> \n> maybe_remove_timestamp () {\n> - if test -z \"$2\"; then\n> - echo_without_newline \"$1\"\n> - else\n> -\techo_without_newline \"$(printf '%s\\n' \"$1\" | sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//')\"\n> - fi\n> +\tif test -z \"$2\"; then\n> +\t\techo_without_newline \"$1\"\n> +\telse\n> +\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n> +\tfi\n> }\n> \n> +remove_timestamp () {\n> +\tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n> +}\n> +\n> +\n> run_tests () {\n> type=$1\n> sha1=$2\n\nI may have missed some previous discussions, but is there a reason this\necho_without_newline dance is needed? At this point this on top passes\nall tests for me on both dash and bash:\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 2d52851dadc..8266a939f99 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -104,18 +104,19 @@ strlen () {\n echo_without_newline \"$1\" | wc -c | sed -e 's/^ *//'\n }\n \n-maybe_remove_timestamp () {\n-\tif test -z \"$2\"; then\n-\t\techo_without_newline \"$1\"\n-\telse\n-\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n-\tfi\n-}\n-\n remove_timestamp () {\n \tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n }\n \n+maybe_remove_timestamp () {\n+\tif test -n \"$2\"\n+\tthen\n+\t\techo \"$1\" | remove_timestamp\n+\t\treturn 0\n+\tfi\n+\n+\techo \"$1\"\n+}\n \n run_tests () {\n type=$1\n\nThe move is another comment, if we're adding a remove_timestamp() let's\ndefine it before maybe_remove_timestamp() which uses it, even though in\nthis case we can get away with it...\n"},{"id":"448843","messageId":"220219.865ypbe51h.gmgdl@evledraar.gmail.com","threadId":"57365","inReplyTo":"c6ea1214062381e2c669959e9084adead5600510.1645208594.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v10 4/4] cat-file: add --batch-command mode","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-19T06:35:30Z","receivedAt":"2022-02-19T06:36:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Feb 18 2022, John Cai via GitGitGadget wrote:\n\n> From: John Cai \n> [....]\n> + for opt in --buffer --no-buffer\n> + do\n> +\ttest -z \"$content\" ||\n> +\t\ttest_expect_success \"--batch-command $opt output of $type content is correct\" '\n> +\t\tmaybe_remove_timestamp \"$batch_output\" $no_ts >expect &&\n> +\t\tmaybe_remove_timestamp \"$(test_write_lines \"contents $sha1\" |\n> +\t\tgit cat-file --batch-command $opt)\" $no_ts >actual &&\n> +\t\ttest_cmp expect actual\n> +\t'\n> +\n> [...]\n> +test_expect_success '--batch-command with multiple command calls gives correct format' '\n> +\tremove_timestamp >expect <<-EOF &&\n> +\t$hello_sha1 blob $hello_size\n> +\t$hello_content\n> +\t$commit_sha1 commit $commit_size\n> +\t$commit_content\n> +\t$tag_sha1 tag $tag_size\n> +\t$tag_content\n> +\tdeadbeef missing\n> +\tEOF\n> +\n> +\tgit cat-file --batch-command --buffer >actual_raw <<-EOF &&\n> +\tcontents $hello_sha1\n> +\tcontents $commit_sha1\n> +\tcontents $tag_sha1\n> +\tcontents deadbeef\n> +\tflush\n> +\tEOF\n> +\n> +\tremove_timestamp actual &&\n> +\ttest_cmp expect actual\n> +'\n\nRe my comment on 3/4; I then tried my suggested change to\nmaybe_remove_timestamp on this patch, and it also works on this\ncommit...\n"},{"id":"449108","messageId":"9b9f1ac5-d5d9-0116-477f-0477c27bc55e@gmail.com","threadId":"57365","inReplyTo":"pull.1212.v10.git.git.1645208594.gitgitgadget@gmail.com","subject":"Re: [PATCH v10 0/4] Add cat-file --batch-command flag","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-22T11:07:41Z","receivedAt":"2022-02-22T11:07:47Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 18/02/2022 18:23, John Cai via GitGitGadget wrote:\n> The feature proposal of adding a command interface to cat-file was first\n> discussed in [A]. In [B], Taylor expressed the need for a fuller proposal\n> before moving forward with a new flag. An RFC was created [C] and the idea\n> was discussed more thoroughly, and overall it seemed like it was headed in\n> the right direction.\n> \n> This patch series consolidates the feedback from these different threads.\n> \n> This patch series has four parts:\n> \n> 1. preparation patch to rename a variable\n> 2. adding an enum to keep track of batch modes\n> 3. add a remove_timestamp() helper that takes stdin and removes timestamps\n> 4. logic to handle --batch-command flag, adding contents, info, flush\n> commands\n> \n> Changes since v9\n> \n> * add test to exercise format for batch-command\n> * minor semantic improvements\n> * removed README entry for environment variable used in test\n\nThe range-diff looks good to me.\n\nBest Wishes\n\nPhillip\n"},{"id":"449125","messageId":"7905D997-249D-4DDF-A97D-0667499AE786@gmail.com","threadId":"57365","inReplyTo":"220219.86a6ene548.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v10 3/4] cat-file: add remove_timestamp helper","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-22T03:31:14Z","receivedAt":"2022-02-22T13:20:46Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Ævar,\n\nOn 19 Feb 2022, at 1:33, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Feb 18 2022, John Cai via GitGitGadget wrote:\n>\n>> From: John Cai \n>>\n>> maybe_remove_timestamp() takes arguments, but it would be useful to have\n>> a function that reads from stdin and strips the timestamp. This would\n>> allow tests to pipe data into a function to remove timestamps, and\n>> wouldn't have to always assign a variable. This is especially helpful\n>> when the data is multiple lines.\n>>\n>> Keep maybe_remove_timestamp() the same, but add a remove_timestamp\n>> helper that reads from stdin.\n>>\n>> The tests in the next patch will make use of this.\n>>\n>> Signed-off-by: John Cai \n>> ---\n>> t/t1006-cat-file.sh | 15 ++++++++++-----\n>> 1 file changed, 10 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n>> index 145eee11df9..2d52851dadc 100755\n>> --- a/t/t1006-cat-file.sh\n>> +++ b/t/t1006-cat-file.sh\n>> @@ -105,13 +105,18 @@ strlen () {\n>> }\n>>\n>> maybe_remove_timestamp () {\n>> - if test -z \"$2\"; then\n>> - echo_without_newline \"$1\"\n>> - else\n>> -\techo_without_newline \"$(printf '%s\\n' \"$1\" | sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//')\"\n>> - fi\n>> +\tif test -z \"$2\"; then\n>> +\t\techo_without_newline \"$1\"\n>> +\telse\n>> +\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n>> +\tfi\n>> }\n>>\n>> +remove_timestamp () {\n>> +\tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n>> +}\n>> +\n>> +\n>> run_tests () {\n>> type=$1\n>> sha1=$2\n>\n> I may have missed some previous discussions, but is there a reason this\n> echo_without_newline dance is needed? At this point this on top passes\n> all tests for me on both dash and bash:\n>\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 2d52851dadc..8266a939f99 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -104,18 +104,19 @@ strlen () {\n> echo_without_newline \"$1\" | wc -c | sed -e 's/^ *//'\n> }\n>\n> -maybe_remove_timestamp () {\n> -\tif test -z \"$2\"; then\n> -\t\techo_without_newline \"$1\"\n> -\telse\n> -\t\techo_without_newline \"$(printf '%s\\n' \"$1\" | remove_timestamp)\"\n> -\tfi\n> -}\n> -\n> remove_timestamp () {\n> \tsed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//'\n> }\n>\n> +maybe_remove_timestamp () {\n> +\tif test -n \"$2\"\n> +\tthen\n> +\t\techo \"$1\" | remove_timestamp\n> +\t\treturn 0\n> +\tfi\n> +\n> +\techo \"$1\"\n> +}\n>\n> run_tests () {\n> type=$1\n>\n> The move is another comment, if we're adding a remove_timestamp() let's\n> define it before maybe_remove_timestamp() which uses it, even though in\n> this case we can get away with it...\n\nThanks for these suggestions! I'll adjust 3/4 to include these changes.\n"}]}