{"thread":{"id":"56095","subject":"[PATCH 02/19] cat-file: merge two block into one","startedAt":"2021-07-12T11:47:09Z","lastAt":"2021-07-15T15:40:50Z","messageCount":52,"participants":["ZheNing Hu via GitGitGadget","Christian Couder","ZheNing Hu","Philip Oakley","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":19},"messages":[{"id":"429722","messageId":"1332006006f4596122aa05afcb2b35f3ee781e1a.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 02/19] cat-file: merge two block into one","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:42Z","receivedAt":"2021-07-12T11:47:09Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nThere are two \"if (opt->all_objects)\" blocks next\nto each other, merge them into one to provide better\nreadability.\n\nHelped-by: Jeff King \nSigned-off-by: ZheNing Hu \nAcked-by: Jeff King \nSigned-off-by: Junio C Hamano \n---\n builtin/cat-file.c | 5 +----\n 1 file changed, 1 insertion(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 02461bb5ea6..243fe6844bc 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -520,14 +520,11 @@ static int batch_objects(struct batch_options *opt)\n \t\tdata.info.typep = &data.type;\n \n \tif (opt->all_objects) {\n+\t\tstruct object_cb_data cb;\n \t\tstruct object_info empty = OBJECT_INFO_INIT;\n \n \t\tif (!memcmp(&data.info, &empty, sizeof(empty)))\n \t\t\tdata.skip_object_info = 1;\n-\t}\n-\n-\tif (opt->all_objects) {\n-\t\tstruct object_cb_data cb;\n \n \t\tif (has_promisor_remote())\n \t\t\twarning(\"This repository uses promisor remotes. Some objects may not be loaded.\");\n-- \ngitgitgadget\n\n"},{"id":"429723","messageId":"9aef8882bd1e5e9bdd401d92449190f462310e40.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 01/19] cat-file: handle trivial --batch format with --batch-all-objects","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:41Z","receivedAt":"2021-07-12T11:47:09Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nThe --batch code to print an object assumes we found out the type of\nthe object from calling oid_object_info_extended(). This is true for\nthe default format, but even in a custom format, we manually modify\nthe object_info struct to ask for the type.\n\nThis assumption was broken by 845de33a5b (cat-file: avoid noop calls\nto sha1_object_info_extended, 2016-05-18). That commit skips the call\nto oid_object_info_extended() entirely when --batch-all-objects is in\nuse, and the custom format does not include any placeholders that\nrequire calling it.\n\nOr when the custom format only include placeholders like %(objectname) or\n%(rest), oid_object_info_extended() will not get the type of the object.\n\nThis results in an error when we try to confirm that the type didn't\nchange:\n\n$ git cat-file --batch=batman --batch-all-objects\nbatman\nfatal: object 000023961a0c02d6e21dc51ea3484ff71abf1c74 changed type!?\n\nand also has other subtle effects (e.g., we'd fail to stream a blob,\nsince we don't realize it's a blob in the first place).\n\nWe can fix this by flipping the order of the setup. The check for \"do\nwe need to get the object info\" must come _after_ we've decided\nwhether we need to look up the type.\n\nHelped-by: Jeff King \nSigned-off-by: ZheNing Hu \nAcked-by: Jeff King \nSigned-off-by: Junio C Hamano \n---\n builtin/cat-file.c | 13 +++++++------\n t/t1006-cat-file.sh | 22 ++++++++++++++++++++++\n 2 files changed, 29 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5ebf13359e8..02461bb5ea6 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -512,12 +512,6 @@ static int batch_objects(struct batch_options *opt)\n \tif (opt->cmdmode)\n \t\tdata.split_on_whitespace = 1;\n \n-\tif (opt->all_objects) {\n-\t\tstruct object_info empty = OBJECT_INFO_INIT;\n-\t\tif (!memcmp(&data.info, &empty, sizeof(empty)))\n-\t\t\tdata.skip_object_info = 1;\n-\t}\n-\n \t/*\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n@@ -525,6 +519,13 @@ static int batch_objects(struct batch_options *opt)\n \tif (opt->print_contents)\n \t\tdata.info.typep = &data.type;\n \n+\tif (opt->all_objects) {\n+\t\tstruct object_info empty = OBJECT_INFO_INIT;\n+\n+\t\tif (!memcmp(&data.info, &empty, sizeof(empty)))\n+\t\t\tdata.skip_object_info = 1;\n+\t}\n+\n \tif (opt->all_objects) {\n \t\tstruct object_cb_data cb;\n \ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 5d2dc99b74a..18b3779ccb6 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -586,4 +586,26 @@ test_expect_success 'cat-file --unordered works' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'set up object list for --batch-all-objects tests' '\n+\tgit -C all-two cat-file --batch-all-objects --batch-check=\"%(objectname)\" >objects\n+'\n+\n+test_expect_success 'cat-file --batch=\"%(objectname)\" with --batch-all-objects will work' '\n+\tgit -C all-two cat-file --batch=\"%(objectname)\" expect &&\n+\tgit -C all-two cat-file --batch-all-objects --batch=\"%(objectname)\" >actual &&\n+\tcmp expect actual\n+'\n+\n+test_expect_success 'cat-file --batch=\"%(rest)\" with --batch-all-objects will work' '\n+\tgit -C all-two cat-file --batch=\"%(rest)\" expect &&\n+\tgit -C all-two cat-file --batch-all-objects --batch=\"%(rest)\" >actual &&\n+\tcmp expect actual\n+'\n+\n+test_expect_success 'cat-file --batch=\"batman\" with --batch-all-objects will work' '\n+\tgit -C all-two cat-file --batch=\"batman\" expect &&\n+\tgit -C all-two cat-file --batch-all-objects --batch=\"batman\" >actual &&\n+\tcmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"429724","messageId":"0dacc60bbcc9ae4376f58c967f43bd0c2396fd7e.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 03/19] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:43Z","receivedAt":"2021-07-12T11:47:09Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nOnly tag and commit objects use `grab_sub_body_contents()` to grab\nobject contents in the current codebase. We want to teach the\nfunction to also handle blobs and trees to get their raw data,\nwithout parsing a blob (whose contents looks like a commit or a tag)\nincorrectly as a commit or a tag.\n\nSkip the block of code that is specific to handling commits and tags\nearly when the given object is of a wrong type to help later\naddition to handle other types of objects in this function.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Junio C Hamano \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4db0e40ff4c..5cee6512fba 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n+static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n {\n \tint i;\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n+\tvoid *buf = data->content;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n@@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (strcmp(name, \"body\") &&\n-\t\t !starts_with(name, \"subject\") &&\n-\t\t !starts_with(name, \"trailers\") &&\n-\t\t !starts_with(name, \"contents\"))\n+\n+\t\tif ((data->type != OBJ_TAG &&\n+\t\t data->type != OBJ_COMMIT) ||\n+\t\t (strcmp(name, \"body\") &&\n+\t\t !starts_with(name, \"subject\") &&\n+\t\t !starts_with(name, \"trailers\") &&\n+\t\t !starts_with(name, \"contents\")))\n \t\t\tcontinue;\n \t\tif (!subpos)\n \t\t\tfind_subpos(buf,\n@@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n+static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n {\n+\tvoid *buf = data->content;\n+\n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n@@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi->content);\n+\t\tgrab_values(ref->value, deref, *obj, oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n-- \ngitgitgadget\n\n"},{"id":"429725","messageId":"pull.993.git.1626090419.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":null,"subject":"[PATCH 00/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:40Z","receivedAt":"2021-07-12T11:47:09Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"This patch series make cat-file reuse ref-filter logic.\n\nChange from last version:\n\n 1. Declare buf_size in if (atom_type == ATOM_RAW) block.\n 2. Modify the code style of the test.\n 3. Delete \"use_textconv\" and \"use_filter\" flag. Instead, add member\n cat_file_cmdmode to struct ref_array_item.\n 4. Add function reject_atom() to enhance the readability of the code.\n 5. Create p1006-cat-file.sh for performance regression testing.\n 6. Use a \"fast path\" to output object data to reduce the performance\n degradation of cat-file --batch with the suggest of Ævar Arnfjörð\n Bjarmason.\n\nZheNing Hu (19):\n cat-file: handle trivial --batch format with --batch-all-objects\n cat-file: merge two block into one\n [GSOC] ref-filter: add obj-type check in grab contents\n [GSOC] ref-filter: add %(raw) atom\n [GSOC] ref-filter: --format=%(raw) re-support --perl\n [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n [GSOC] ref-filter: add %(rest) atom\n [GSOC] ref-filter: pass get_object() return value to their callers\n [GSOC] ref-filter: introduce free_ref_array_item_value() function\n [GSOC] ref-filter: introduce reject_atom()\n [GSOC] ref-filter: modify the error message and value in get_object\n [GSOC] cat-file: add has_object_file() check\n [GSOC] cat-file: change batch_objects parameter name\n [GSOC] cat-file: reuse ref-filter logic\n [GSOC] cat-file: reuse err buf in batch_object_write()\n [GSOC] cat-file: re-implement --textconv, --filters options\n [GSOC] ref-filter: remove grab_oid() function\n [GSOC] cat-file: create p1006-cat-file.sh\n [GSOC] cat-file: use fast path when using default_format\n\n Documentation/git-cat-file.txt | 6 +\n Documentation/git-for-each-ref.txt | 9 +\n builtin/cat-file.c | 308 +++++++++----------------\n builtin/tag.c | 2 +-\n quote.c | 17 ++\n quote.h | 1 +\n ref-filter.c | 346 +++++++++++++++++++++--------\n ref-filter.h | 13 +-\n t/perf/p1006-cat-file.sh | 28 +++\n t/t1006-cat-file.sh | 273 +++++++++++++++++++++++\n t/t3203-branch-output.sh | 4 +\n t/t6300-for-each-ref.sh | 235 ++++++++++++++++++++\n t/t6301-for-each-ref-errors.sh | 2 +-\n t/t7004-tag.sh | 4 +\n t/t7030-verify-tag.sh | 4 +\n 15 files changed, 955 insertions(+), 297 deletions(-)\n create mode 100755 t/perf/p1006-cat-file.sh\n\n\nbase-commit: d486ca60a51c9cb1fe068803c3f540724e95e83a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-993%2Fadlternative%2Fcat-file-batch-refactor-2-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-993/adlternative/cat-file-batch-refactor-2-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/993\n-- \ngitgitgadget\n"},{"id":"429726","messageId":"460c807c6ab9010ae166fe77e54d5435960f8e37.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 07/19] [GSOC] ref-filter: add %(rest) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:47Z","receivedAt":"2021-07-12T11:47:11Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nIn order to let \"cat-file --batch=%(rest)\" use the ref-filter\ninterface, add %(rest) atom for ref-filter. \"git for-each-ref\",\n\"git branch\", \"git tag\" and \"git verify-tag\" will reject %(rest)\nby default.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 21 +++++++++++++++++++++\n ref-filter.h | 5 ++++-\n t/t3203-branch-output.sh | 4 ++++\n t/t6300-for-each-ref.sh | 4 ++++\n t/t7004-tag.sh | 4 ++++\n t/t7030-verify-tag.sh | 4 ++++\n 6 files changed, 41 insertions(+), 1 deletion(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex c8e561a3687..ee6c906f22d 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -157,6 +157,7 @@ enum atom_type {\n \tATOM_IF,\n \tATOM_THEN,\n \tATOM_ELSE,\n+\tATOM_REST,\n };\n \n /*\n@@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"%%(rest) does not take arguments\"));\n+\tformat->use_rest = 1;\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n@@ -615,6 +625,7 @@ static struct {\n \t[ATOM_IF] = { \"if\", SOURCE_NONE, FIELD_STR, if_atom_parser },\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n+\t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -1010,6 +1021,9 @@ int verify_ref_format(struct ref_format *format)\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n \n+\t\tif (used_atom[at].atom_type == ATOM_REST)\n+\t\t\tdie(\"this command reject atom %%(%.*s)\", (int)(ep - sp - 2), sp + 2);\n+\n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n \t\t format->quote_style == QUOTE_SHELL ||\n \t\t format->quote_style == QUOTE_TCL) &&\n@@ -1928,6 +1942,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_REST) {\n+\t\t\tif (ref->rest)\n+\t\t\t\tv->s = xstrdup(ref->rest);\n+\t\t\telse\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2145,6 +2165,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n+\tref->rest = NULL;\n \n \treturn ref;\n }\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 74fb423fc89..c15dee8d6b9 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -38,6 +38,7 @@ struct ref_sorting {\n \n struct ref_array_item {\n \tstruct object_id objectname;\n+\tconst char *rest;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n@@ -76,14 +77,16 @@ struct ref_format {\n \t * verify_ref_format() afterwards to finalize.\n \t */\n \tconst char *format;\n+\tconst char *rest;\n \tint quote_style;\n+\tint use_rest;\n \tint use_color;\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n };\n \n-#define REF_FORMAT_INIT { NULL, 0, -1 }\n+#define REF_FORMAT_INIT { .use_color = -1 }\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\ndiff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh\nindex 5325b9f67a0..6e94c6db7b5 100755\n--- a/t/t3203-branch-output.sh\n+++ b/t/t3203-branch-output.sh\n@@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git branch with --format=%(rest) must fail' '\n+\ttest_must_fail git branch --format=\"%(rest)\" >actual\n+'\n+\n test_expect_success 'worktree colors correct' '\n \tcat >expect <<-EOF &&\n \t* (HEAD detached from fromtag)\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 3d15d0a5360..0d2e062f791 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' '\n \ttest_cmp expect actual.clean\n '\n \n+test_expect_success 'basic atom: rest must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(rest)\" refs/heads/main\n+'\n+\n test_expect_success 'trailer parsing not fooled by --- line' '\n \tgit commit --allow-empty -F - <<-\\EOF &&\n \tthis is the subject\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 2f72c5c6883..082be85dffc 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git tag -l with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git tag -l --format=\"%(rest)\" \"v1*\"\n+'\n+\n test_expect_success \"set up color tests\" '\n \techo \"v1.0\" >expect.color &&\n \techo \"v1.0\" >expect.bare &&\ndiff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh\nindex 3cefde9602b..10faa645157 100755\n--- a/t/t7030-verify-tag.sh\n+++ b/t/t7030-verify-tag.sh\n@@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success GPG 'verifying tag with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git verify-tag --format=\"%(rest)\" \"fourth-signed\"\n+'\n+\n test_expect_success GPG 'verifying a forged tag with --format should fail silently' '\n \ttest_must_fail git verify-tag --format=\"tagname : %(tag)\" $(cat forged1.tag) >actual-forged &&\n \ttest_must_be_empty actual-forged\n-- \ngitgitgadget\n\n"},{"id":"429727","messageId":"e1aca51d500736722295b4c371c9698c168261cd.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 08/19] [GSOC] ref-filter: pass get_object() return value to their callers","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:48Z","receivedAt":"2021-07-12T11:47:12Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause in the refactor of `git cat-file --batch` later,\noid_object_info_extended() in get_object() will be used to obtain\nthe info of an object with it's oid. When the object cannot be\nobtained in the git repository, `cat-file --batch` expects to output\n\" missing\" and continue the next oid query instead of letting\nGit exit. In other error conditions, Git should exit normally. So we\ncan achieve this function by passing the return value of get_object().\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 17 +++++++++++------\n 1 file changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex ee6c906f22d..3189872188a 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1817,6 +1817,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n {\n \tstruct object *obj;\n \tint i;\n+\tint ret;\n \tstruct object_info empty = OBJECT_INFO_INIT;\n \n \tCALLOC_ARRAY(ref->value, used_atom_cnt);\n@@ -1973,8 +1974,9 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \n \n \toi.oid = ref->objectname;\n-\tif (get_object(ref, 0, &obj, &oi, err))\n-\t\treturn -1;\n+\tret = get_object(ref, 0, &obj, &oi, err);\n+\tif (ret)\n+\t\treturn ret;\n \n \t/*\n \t * If there is no atom that wants to know about tagged\n@@ -2006,8 +2008,10 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,\n \t\t\t struct atom_value **v, struct strbuf *err)\n {\n \tif (!ref->value) {\n-\t\tif (populate_value(ref, err))\n-\t\t\treturn -1;\n+\t\tint ret = populate_value(ref, err);\n+\n+\t\tif (ret)\n+\t\t\treturn ret;\n \t\tfill_missing_values(ref->value);\n \t}\n \t*v = &ref->value[atom];\n@@ -2581,6 +2585,7 @@ int format_ref_array_item(struct ref_array_item *info,\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n+\tint ret;\n \n \tstate.quote_style = format->quote_style;\n \tpush_stack_element(&state.stack);\n@@ -2593,10 +2598,10 @@ int format_ref_array_item(struct ref_array_item *info,\n \t\tif (cp < sp)\n \t\t\tappend_literal(cp, sp, &state);\n \t\tpos = parse_ref_filter_atom(format, sp + 2, ep, error_buf);\n-\t\tif (pos < 0 || get_ref_atom_value(info, pos, &atomv, error_buf) ||\n+\t\tif (pos < 0 || (ret = get_ref_atom_value(info, pos, &atomv, error_buf)) ||\n \t\t atomv->handler(atomv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\n-\t\t\treturn -1;\n+\t\t\treturn ret ? ret : -1;\n \t\t}\n \t}\n \tif (*cp) {\n-- \ngitgitgadget\n\n"},{"id":"429728","messageId":"2d34ad3bc77a1a08e5f31545d4a345706541d700.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 06/19] [GSOC] ref-filter: use non-const ref_format in *_atom_parser()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:46Z","receivedAt":"2021-07-12T11:47:12Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse non-const ref_format in *_atom_parser(), which can help us\nmodify the members of ref_format in *_atom_parser().\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/tag.c | 2 +-\n ref-filter.c | 44 ++++++++++++++++++++++----------------------\n ref-filter.h | 4 ++--\n 3 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 82fcfc09824..452558ec957 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref,\n \t\t const struct object_id *oid, void *cb_data)\n {\n \tint flags;\n-\tconst struct ref_format *format = cb_data;\n+\tstruct ref_format *format = cb_data;\n \tflags = GPG_VERIFY_VERBOSE;\n \n \tif (format->format)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex ba9ab35d7ec..c8e561a3687 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -226,7 +226,7 @@ static int strbuf_addf_ret(struct strbuf *sb, int ret, const char *fmt, ...)\n \treturn ret;\n }\n \n-static int color_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int color_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *color_value, struct strbuf *err)\n {\n \tif (!color_value)\n@@ -264,7 +264,7 @@ static int refname_atom_parser_internal(struct refname_atom *atom, const char *a\n \treturn 0;\n }\n \n-static int remote_ref_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int remote_ref_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct string_list params = STRING_LIST_INIT_DUP;\n@@ -311,7 +311,7 @@ static int remote_ref_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objecttype_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -323,7 +323,7 @@ static int objecttype_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objectsize_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -343,7 +343,7 @@ static int objectsize_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int deltabase_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int deltabase_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -355,7 +355,7 @@ static int deltabase_atom_parser(const struct ref_format *format, struct used_at\n \treturn 0;\n }\n \n-static int body_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int body_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -364,7 +364,7 @@ static int body_atom_parser(const struct ref_format *format, struct used_atom *a\n \treturn 0;\n }\n \n-static int subject_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int subject_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -376,7 +376,7 @@ static int subject_atom_parser(const struct ref_format *format, struct used_atom\n \treturn 0;\n }\n \n-static int trailers_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int trailers_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tatom->u.contents.trailer_opts.no_divider = 1;\n@@ -402,7 +402,7 @@ static int trailers_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int contents_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int contents_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -430,7 +430,7 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int raw_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -442,7 +442,7 @@ static int raw_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int oid_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -461,7 +461,7 @@ static int oid_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int person_email_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int person_email_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -475,7 +475,7 @@ static int person_email_atom_parser(const struct ref_format *format, struct used\n \treturn 0;\n }\n \n-static int refname_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int refname_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \treturn refname_atom_parser_internal(&atom->u.refname, arg, atom->name, err);\n@@ -492,7 +492,7 @@ static align_type parse_align_position(const char *s)\n \treturn -1;\n }\n \n-static int align_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int align_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct align *align = &atom->u.align;\n@@ -544,7 +544,7 @@ static int align_atom_parser(const struct ref_format *format, struct used_atom *\n \treturn 0;\n }\n \n-static int if_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -559,7 +559,7 @@ static int if_atom_parser(const struct ref_format *format, struct used_atom *ato\n \treturn 0;\n }\n \n-static int head_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n \tatom->u.head = resolve_refdup(\"HEAD\", RESOLVE_REF_READING, NULL, NULL);\n@@ -570,7 +570,7 @@ static struct {\n \tconst char *name;\n \tinfo_source source;\n \tcmp_type cmp_type;\n-\tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n+\tint (*parser)(struct ref_format *format, struct used_atom *atom,\n \t\t const char *arg, struct strbuf *err);\n } valid_atom[] = {\n \t[ATOM_REFNAME] = { \"refname\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -649,7 +649,7 @@ struct atom_value {\n /*\n * Used to parse format string and sort specifiers\n */\n-static int parse_ref_filter_atom(const struct ref_format *format,\n+static int parse_ref_filter_atom(struct ref_format *format,\n \t\t\t\t const char *atom, const char *ep,\n \t\t\t\t struct strbuf *err)\n {\n@@ -2554,9 +2554,9 @@ static void append_literal(const char *cp, const char *ep, struct ref_formatting\n }\n \n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n-\t\t\t struct strbuf *final_buf,\n-\t\t\t struct strbuf *error_buf)\n+\t\t\t struct ref_format *format,\n+\t\t\t struct strbuf *final_buf,\n+\t\t\t struct strbuf *error_buf)\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n@@ -2601,7 +2601,7 @@ int format_ref_array_item(struct ref_array_item *info,\n }\n \n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format)\n+\t\t struct ref_format *format)\n {\n \tstruct ref_array_item *ref_item;\n \tstruct strbuf output = STRBUF_INIT;\ndiff --git a/ref-filter.h b/ref-filter.h\nindex baf72a71896..74fb423fc89 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);\n void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting, unsigned int mask, int on);\n /* Based on the given format and quote_style, fill the strbuf */\n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n+\t\t\t struct ref_format *format,\n \t\t\t struct strbuf *final_buf,\n \t\t\t struct strbuf *error_buf);\n /* Parse a single sort specifier and add it to the list */\n@@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void);\n * name must be a fully qualified refname.\n */\n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format);\n+\t\t struct ref_format *format);\n \n /*\n * Push a single ref onto the array; this can be used to construct your own\n-- \ngitgitgadget\n\n"},{"id":"429729","messageId":"5bd715ae9f02a580f3594a3207df45707119aeb1.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 04/19] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:44Z","receivedAt":"2021-07-12T11:47:14Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd new formatting option `%(raw)`, which will print the raw\nobject data without any changes. It will help further to migrate\nall cat-file formatting logic from cat-file to ref-filter.\n\nThe raw data of blob, tree objects may contain '\\0', but most of\nthe logic in `ref-filter` depends on the output of the atom being\ntext (specifically, no embedded NULs in it).\n\nE.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()`\nadd the data to the buffer. The raw data of a tree object is\n`100644 one\\0...`, only the `100644 one` will be added to the buffer,\nwhich is incorrect.\n\nTherefore, we need to find a way to record the length of the\natom_value's member `s`. Although strbuf can already record the\nstring and its length, if we want to replace the type of atom_value's\nmember `s` with strbuf, many places in ref-filter that are filled\nwith dynamically allocated mermory in `v->s` are not easy to replace.\nAt the same time, we need to check if `v->s == NULL` in\npopulate_value(), and strbuf cannot easily distinguish NULL and empty\nstrings, but c-style \"const char *\" can do it. So add a new member in\n`struct atom_value`: `s_size`, which can record raw object size, it\ncan help us add raw object data to the buffer or compare two buffers\nwhich contain raw object data.\n\nNote that `--format=%(raw)` cannot be used with `--python`, `--shell`,\n`--tcl`, and `--perl` because if the binary raw data is passed to a\nvariable in such languages, these may not support arbitrary binary data\nin their string variable type.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Bagas Sanjaya \nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Felipe Contreras \nHelped-by: Phillip Wood \nHelped-by: Junio C Hamano \nBased-on-patch-by: Olga Telezhnaya \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 9 ++\n ref-filter.c | 140 +++++++++++++++----\n t/t6300-for-each-ref.sh | 216 +++++++++++++++++++++++++++++\n 3 files changed, 338 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 2ae2478de70..cbb6f87d13f 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`,\n without angle brackets, and `:localpart` to get the part before the `@` symbol\n out of the trimmed email.\n \n+The raw data in an object is `raw`.\n+\n+raw:size::\n+\tThe raw data size of the object.\n+\n+Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n+`--perl` because such language may not support arbitrary binary data in their\n+string variable type.\n+\n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5cee6512fba..506fbc3d691 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -144,6 +144,7 @@ enum atom_type {\n \tATOM_BODY,\n \tATOM_TRAILERS,\n \tATOM_CONTENTS,\n+\tATOM_RAW,\n \tATOM_UPSTREAM,\n \tATOM_PUSH,\n \tATOM_SYMREF,\n@@ -189,6 +190,9 @@ static struct used_atom {\n \t\t\tstruct process_trailer_options trailer_opts;\n \t\t\tunsigned int nlines;\n \t\t} contents;\n+\t\tstruct {\n+\t\t\tenum { RAW_BARE, RAW_LENGTH } option;\n+\t\t} raw_data;\n \t\tstruct {\n \t\t\tcmp_status cmp_status;\n \t\t\tconst char *str;\n@@ -426,6 +430,18 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n+static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\tconst char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\tatom->u.raw_data.option = RAW_BARE;\n+\telse if (!strcmp(arg, \"size\"))\n+\t\tatom->u.raw_data.option = RAW_LENGTH;\n+\telse\n+\t\treturn strbuf_addf_ret(err, -1, _(\"unrecognized %%(raw) argument: %s\"), arg);\n+\treturn 0;\n+}\n+\n static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -586,6 +602,7 @@ static struct {\n \t[ATOM_BODY] = { \"body\", SOURCE_OBJ, FIELD_STR, body_atom_parser },\n \t[ATOM_TRAILERS] = { \"trailers\", SOURCE_OBJ, FIELD_STR, trailers_atom_parser },\n \t[ATOM_CONTENTS] = { \"contents\", SOURCE_OBJ, FIELD_STR, contents_atom_parser },\n+\t[ATOM_RAW] = { \"raw\", SOURCE_OBJ, FIELD_STR, raw_atom_parser },\n \t[ATOM_UPSTREAM] = { \"upstream\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_PUSH] = { \"push\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_SYMREF] = { \"symref\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -620,12 +637,15 @@ struct ref_formatting_state {\n \n struct atom_value {\n \tconst char *s;\n+\tsize_t s_size;\n \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t struct strbuf *err);\n \tuintmax_t value; /* used for sorting when not FIELD_STR */\n \tstruct used_atom *atom;\n };\n \n+#define ATOM_VALUE_S_SIZE_INIT (-1)\n+\n /*\n * Used to parse format string and sort specifiers\n */\n@@ -644,13 +664,6 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \t\treturn strbuf_addf_ret(err, -1, _(\"malformed field name: %.*s\"),\n \t\t\t\t (int)(ep-atom), atom);\n \n-\t/* Do we have the atom already used elsewhere? */\n-\tfor (i = 0; i < used_atom_cnt; i++) {\n-\t\tint len = strlen(used_atom[i].name);\n-\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n-\t\t\treturn i;\n-\t}\n-\n \t/*\n \t * If the atom name has a colon, strip it and everything after\n \t * it off - it specifies the format for this entry, and\n@@ -660,6 +673,13 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \targ = memchr(sp, ':', ep - sp);\n \tatom_len = (arg ? arg : ep) - sp;\n \n+\t/* Do we have the atom already used elsewhere? */\n+\tfor (i = 0; i < used_atom_cnt; i++) {\n+\t\tint len = strlen(used_atom[i].name);\n+\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n+\t\t\treturn i;\n+\t}\n+\n \t/* Is the atom a valid one? */\n \tfor (i = 0; i < ARRAY_SIZE(valid_atom); i++) {\n \t\tint len = strlen(valid_atom[i].name);\n@@ -709,11 +729,14 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \treturn at;\n }\n \n-static void quote_formatting(struct strbuf *s, const char *str, int quote_style)\n+static void quote_formatting(struct strbuf *s, const char *str, size_t len, int quote_style)\n {\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n-\t\tstrbuf_addstr(s, str);\n+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tstrbuf_add(s, str, len);\n+\t\telse\n+\t\t\tstrbuf_addstr(s, str);\n \t\tbreak;\n \tcase QUOTE_SHELL:\n \t\tsq_quote_buf(s, str);\n@@ -740,9 +763,12 @@ static int append_atom(struct atom_value *v, struct ref_formatting_state *state,\n \t * encountered.\n \t */\n \tif (!state->stack->prev)\n-\t\tquote_formatting(&state->stack->output, v->s, state->quote_style);\n+\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n \telse\n-\t\tstrbuf_addstr(&state->stack->output, v->s);\n+\t\tif (v->s_size != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n+\t\telse\n+\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n \treturn 0;\n }\n \n@@ -842,21 +868,23 @@ static int if_atom_handler(struct atom_value *atomv, struct ref_formatting_state\n \treturn 0;\n }\n \n-static int is_empty(const char *s)\n+static int is_empty(struct strbuf *buf)\n {\n-\twhile (*s != '\\0') {\n-\t\tif (!isspace(*s))\n-\t\t\treturn 0;\n-\t\ts++;\n-\t}\n-\treturn 1;\n-}\n+\tconst char *cur = buf->buf;\n+\tconst char *end = buf->buf + buf->len;\n+\n+\twhile (cur != end && (isspace(*cur)))\n+\t\tcur++;\n+\n+\treturn cur == end;\n+ }\n \n static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t\t struct strbuf *err)\n {\n \tstruct ref_formatting_stack *cur = state->stack;\n \tstruct if_then_else *if_then_else = NULL;\n+\tsize_t str_len = 0;\n \n \tif (cur->at_end == if_then_else_handler)\n \t\tif_then_else = (struct if_then_else *)cur->at_end_data;\n@@ -867,18 +895,22 @@ static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_sta\n \tif (if_then_else->else_atom_seen)\n \t\treturn strbuf_addf_ret(err, -1, _(\"format: %%(then) atom used after %%(else)\"));\n \tif_then_else->then_atom_seen = 1;\n+\tif (if_then_else->str)\n+\t\tstr_len = strlen(if_then_else->str);\n \t/*\n \t * If the 'equals' or 'notequals' attribute is used then\n \t * perform the required comparison. If not, only non-empty\n \t * strings satisfy the 'if' condition.\n \t */\n \tif (if_then_else->cmp_status == COMPARE_EQUAL) {\n-\t\tif (!strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len == cur->output.len &&\n+\t\t !memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n \t} else if (if_then_else->cmp_status == COMPARE_UNEQUAL) {\n-\t\tif (strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len != cur->output.len ||\n+\t\t memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n-\t} else if (cur->output.len && !is_empty(cur->output.buf))\n+\t} else if (cur->output.len && !is_empty(&cur->output))\n \t\tif_then_else->condition_satisfied = 1;\n \tstrbuf_reset(&cur->output);\n \treturn 0;\n@@ -924,7 +956,7 @@ static int end_atom_handler(struct atom_value *atomv, struct ref_formatting_stat\n \t * only on the topmost supporting atom.\n \t */\n \tif (!current->prev->prev) {\n-\t\tquote_formatting(&s, current->output.buf, state->quote_style);\n+\t\tquote_formatting(&s, current->output.buf, current->output.len, state->quote_style);\n \t\tstrbuf_swap(¤t->output, &s);\n \t}\n \tstrbuf_release(&s);\n@@ -974,6 +1006,10 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\t\t\tdie(_(\"--format=%.*s cannot be used with\"\n+\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\n@@ -1367,12 +1403,25 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp\n \t\tstruct used_atom *atom = &used_atom[i];\n \t\tconst char *name = atom->name;\n \t\tstruct atom_value *v = &val[i];\n+\t\tenum atom_type atom_type = atom->atom_type;\n \n \t\tif (!!deref != (*name == '*'))\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n \n+\t\tif (atom_type == ATOM_RAW) {\n+\t\t\tunsigned long buf_size = data->size;\n+\n+\t\t\tif (atom->u.raw_data.option == RAW_BARE) {\n+\t\t\t\tv->s = xmemdupz(buf, buf_size);\n+\t\t\t\tv->s_size = buf_size;\n+\t\t\t} else if (atom->u.raw_data.option == RAW_LENGTH) {\n+\t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX, (uintmax_t)buf_size);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif ((data->type != OBJ_TAG &&\n \t\t data->type != OBJ_COMMIT) ||\n \t\t (strcmp(name, \"body\") &&\n@@ -1460,9 +1509,11 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"Eh? Object of type %d?\", obj->type);\n@@ -1766,6 +1817,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n \n+\t\tv->s_size = ATOM_VALUE_S_SIZE_INIT;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n@@ -2369,6 +2421,19 @@ static int compare_detached_head(struct ref_array_item *a, struct ref_array_item\n \treturn 0;\n }\n \n+static int memcasecmp(const void *vs1, const void *vs2, size_t n)\n+{\n+\tconst char *s1 = vs1, *s2 = vs2;\n+\tconst char *end = s1 + n;\n+\n+\tfor (; s1 < end; s1++, s2++) {\n+\t\tint diff = tolower(*s1) - tolower(*s2);\n+\t\tif (diff)\n+\t\t\treturn diff;\n+\t}\n+\treturn 0;\n+}\n+\n static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b)\n {\n \tstruct atom_value *va, *vb;\n@@ -2389,10 +2454,30 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n \t} else if (s->sort_flags & REF_SORTING_VERSION) {\n \t\tcmp = versioncmp(va->s, vb->s);\n \t} else if (cmp_type == FIELD_STR) {\n-\t\tint (*cmp_fn)(const char *, const char *);\n-\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n-\t\t\t? strcasecmp : strcmp;\n-\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\tif (va->s_size == ATOM_VALUE_S_SIZE_INIT &&\n+\t\t vb->s_size == ATOM_VALUE_S_SIZE_INIT) {\n+\t\t\tint (*cmp_fn)(const char *, const char *);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? strcasecmp : strcmp;\n+\t\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\t} else {\n+\t\t\tsize_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT ?\n+\t\t\t\t\tstrlen(va->s) : va->s_size;\n+\t\t\tsize_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT ?\n+\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n+\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? memcasecmp : memcmp;\n+\n+\t\t\tcmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n+\t\t\t\t a_size : b_size);\n+\t\t\tif (!cmp) {\n+\t\t\t\tif (a_size > b_size)\n+\t\t\t\t\tcmp = 1;\n+\t\t\t\telse if (a_size < b_size)\n+\t\t\t\t\tcmp = -1;\n+\t\t\t}\n+\t\t}\n \t} else {\n \t\tif (va->value < vb->value)\n \t\t\tcmp = -1;\n@@ -2492,6 +2577,7 @@ int format_ref_array_item(struct ref_array_item *info,\n \t}\n \tif (format->need_color_reset_at_eol) {\n \t\tstruct atom_value resetv;\n+\t\tresetv.s_size = ATOM_VALUE_S_SIZE_INIT;\n \t\tresetv.s = GIT_COLOR_RESET;\n \t\tif (append_atom(&resetv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 9e0214076b4..18554f62d94 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -130,6 +130,8 @@ test_atom head parent:short=10 ''\n test_atom head numparent 0\n test_atom head object ''\n test_atom head type ''\n+test_atom head raw \"$(git cat-file commit refs/heads/main)\n+\"\n test_atom head '*objectname' ''\n test_atom head '*objecttype' ''\n test_atom head author 'A U Thor 1151968724 +0200'\n@@ -221,6 +223,15 @@ test_atom tag contents 'Tagging at 1151968727\n '\n test_atom tag HEAD ' '\n \n+test_expect_success 'basic atom: refs/tags/testtag *raw' '\n+\tgit cat-file commit refs/tags/testtag^{} >expected &&\n+\tgit for-each-ref --format=\"%(*raw)\" refs/tags/testtag >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'Check invalid atoms names are errors' '\n \ttest_must_fail git for-each-ref --format=\"%(INVALID)\" refs/heads\n '\n@@ -686,6 +697,15 @@ test_atom refs/tags/signed-empty contents:body ''\n test_atom refs/tags/signed-empty contents:signature \"$sig\"\n test_atom refs/tags/signed-empty contents \"$sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-empty raw' '\n+\tgit cat-file tag refs/tags/signed-empty >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-empty >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-short subject 'subject line'\n test_atom refs/tags/signed-short subject:sanitize 'subject-line'\n test_atom refs/tags/signed-short contents:subject 'subject line'\n@@ -695,6 +715,15 @@ test_atom refs/tags/signed-short contents:signature \"$sig\"\n test_atom refs/tags/signed-short contents \"subject line\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-short raw' '\n+\tgit cat-file tag refs/tags/signed-short >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-short >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-long subject 'subject line'\n test_atom refs/tags/signed-long subject:sanitize 'subject-line'\n test_atom refs/tags/signed-long contents:subject 'subject line'\n@@ -708,6 +737,15 @@ test_atom refs/tags/signed-long contents \"subject line\n body contents\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-long raw' '\n+\tgit cat-file tag refs/tags/signed-long >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-long >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'set up refs pointing to tree and blob' '\n \tgit update-ref refs/mytrees/first refs/heads/main^{tree} &&\n \tgit update-ref refs/myblobs/first refs/heads/main:one\n@@ -720,6 +758,16 @@ test_atom refs/mytrees/first contents:body \"\"\n test_atom refs/mytrees/first contents:signature \"\"\n test_atom refs/mytrees/first contents \"\"\n \n+test_expect_success 'basic atom: refs/mytrees/first raw' '\n+\tgit cat-file tree refs/mytrees/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/mytrees/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_atom refs/myblobs/first subject \"\"\n test_atom refs/myblobs/first contents:subject \"\"\n test_atom refs/myblobs/first body \"\"\n@@ -727,6 +775,174 @@ test_atom refs/myblobs/first contents:body \"\"\n test_atom refs/myblobs/first contents:signature \"\"\n test_atom refs/myblobs/first contents \"\"\n \n+test_expect_success 'basic atom: refs/myblobs/first raw' '\n+\tgit cat-file blob refs/myblobs/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/myblobs/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'set up refs pointing to binary blob' '\n+\tprintf \"a\\0b\\0c\" >blob1 &&\n+\tprintf \"a\\0c\\0b\" >blob2 &&\n+\tprintf \"\\0a\\0b\\0c\" >blob3 &&\n+\tprintf \"abc\" >blob4 &&\n+\tprintf \"\\0 \\0 \\0 \" >blob5 &&\n+\tprintf \"\\0 \\0a\\0 \" >blob6 &&\n+\tprintf \" \" >blob7 &&\n+\t>blob8 &&\n+\tobj=$(git hash-object -w blob1) &&\n+\tgit update-ref refs/myblobs/blob1 \"$obj\" &&\n+\tobj=$(git hash-object -w blob2) &&\n+\tgit update-ref refs/myblobs/blob2 \"$obj\" &&\n+\tobj=$(git hash-object -w blob3) &&\n+\tgit update-ref refs/myblobs/blob3 \"$obj\" &&\n+\tobj=$(git hash-object -w blob4) &&\n+\tgit update-ref refs/myblobs/blob4 \"$obj\" &&\n+\tobj=$(git hash-object -w blob5) &&\n+\tgit update-ref refs/myblobs/blob5 \"$obj\" &&\n+\tobj=$(git hash-object -w blob6) &&\n+\tgit update-ref refs/myblobs/blob6 \"$obj\" &&\n+\tobj=$(git hash-object -w blob7) &&\n+\tgit update-ref refs/myblobs/blob7 \"$obj\" &&\n+\tobj=$(git hash-object -w blob8) &&\n+\tgit update-ref refs/myblobs/blob8 \"$obj\"\n+'\n+\n+test_expect_success 'Verify sorts with raw' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob7\n+\trefs/mytrees/first\n+\trefs/myblobs/first\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob4\n+\trefs/heads/main\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'Verify sorts with raw:size' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\trefs/myblobs/blob7\n+\trefs/heads/main\n+\trefs/myblobs/blob4\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/mytrees/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw:size \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:equals)' '\n+\tcat >expected <<-EOF &&\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\trefs/myblobs/blob4\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:equals=abc)%(raw)%(then)%(refname)%(else)not equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:notequals)' '\n+\tcat >expected <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/heads/main\n+\trefs/heads/newtag\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\tequals\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob7\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:notequals=abc)%(raw)%(then)%(refname)%(else)equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'empty raw refs with %(if)' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob1 not empty\n+\trefs/myblobs/blob2 not empty\n+\trefs/myblobs/blob3 not empty\n+\trefs/myblobs/blob4 not empty\n+\trefs/myblobs/blob5 not empty\n+\trefs/myblobs/blob6 not empty\n+\trefs/myblobs/blob7 empty\n+\trefs/myblobs/blob8 empty\n+\trefs/myblobs/first not empty\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname) %(if)%(raw)%(then)not empty%(else)empty%(end)\" \\\n+\t\trefs/myblobs/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success '%(raw) with --python must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --python\n+'\n+\n+test_expect_success '%(raw) with --tcl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n+'\n+\n+test_expect_success '%(raw) with --perl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+'\n+\n+test_expect_success '%(raw) with --shell must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --shell\n+'\n+\n+test_expect_success '%(raw) with --shell and --sort=raw must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --sort=raw --shell\n+'\n+\n+test_expect_success '%(raw:size) with --shell' '\n+\tgit for-each-ref --format=\"%(raw:size)\" | while read line\n+\tdo\n+\t\techo \"'\\''$line'\\''\" >>expect\n+\tdone &&\n+\tgit for-each-ref --format=\"%(raw:size)\" --shell >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for-each-ref --format compare with cat-file --batch' '\n+\tgit rev-parse refs/mytrees/first | git cat-file --batch >expected &&\n+\tgit for-each-ref --format=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'set up multiple-sort tags' '\n \tfor when in 100000 200000\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"429730","messageId":"df2cd9e8e614d8eb1eee05ff7307b88211504116.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 05/19] [GSOC] ref-filter: --format=%(raw) re-support --perl","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:45Z","receivedAt":"2021-07-12T11:47:15Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause the perl language can handle binary data correctly,\nadd the function perl_quote_buf_with_len(), which can specify\nthe length of the data and prevent the data from being truncated\nat '\\0' to help `--format=\"%(raw)\"` re-support `--perl`.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 4 ++--\n quote.c | 17 +++++++++++++++++\n quote.h | 1 +\n ref-filter.c | 15 +++++++++++----\n t/t6300-for-each-ref.sh | 19 +++++++++++++++++--\n 5 files changed, 48 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex cbb6f87d13f..6da899c6296 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -241,8 +241,8 @@ raw:size::\n \tThe raw data size of the object.\n \n Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n-`--perl` because such language may not support arbitrary binary data in their\n-string variable type.\n+because such language may not support arbitrary binary data in their string\n+variable type.\n \n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\ndiff --git a/quote.c b/quote.c\nindex 8a3a5e39eb1..26719d21d1e 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src)\n \tstrbuf_addch(sb, sq);\n }\n \n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len)\n+{\n+\tconst char sq = '\\'';\n+\tconst char bq = '\\\\';\n+\tconst char *c = src;\n+\tconst char *end = src + len;\n+\n+\tstrbuf_addch(sb, sq);\n+\twhile (c != end) {\n+\t\tif (*c == sq || *c == bq)\n+\t\t\tstrbuf_addch(sb, bq);\n+\t\tstrbuf_addch(sb, *c);\n+\t\tc++;\n+\t}\n+\tstrbuf_addch(sb, sq);\n+}\n+\n void python_quote_buf(struct strbuf *sb, const char *src)\n {\n \tconst char sq = '\\'';\ndiff --git a/quote.h b/quote.h\nindex 768cc6338e2..0fe69e264b0 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -94,6 +94,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne\n \n /* quoting as a string literal for other languages */\n void perl_quote_buf(struct strbuf *sb, const char *src);\n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len);\n void python_quote_buf(struct strbuf *sb, const char *src);\n void tcl_quote_buf(struct strbuf *sb, const char *src);\n void basic_regex_quote_buf(struct strbuf *sb, const char *src);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 506fbc3d691..ba9ab35d7ec 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -742,7 +742,10 @@ static void quote_formatting(struct strbuf *s, const char *str, size_t len, int\n \t\tsq_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PERL:\n-\t\tperl_quote_buf(s, str);\n+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tperl_quote_buf_with_len(s, str, len);\n+\t\telse\n+\t\t\tperl_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PYTHON:\n \t\tpython_quote_buf(s, str);\n@@ -1006,10 +1009,14 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n-\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n-\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\n+\t\tif ((format->quote_style == QUOTE_PYTHON ||\n+\t\t format->quote_style == QUOTE_SHELL ||\n+\t\t format->quote_style == QUOTE_TCL) &&\n+\t\t used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n \t\t\tdie(_(\"--format=%.*s cannot be used with\"\n-\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n+\t\t\t \"--python, --shell, --tcl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 18554f62d94..3d15d0a5360 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' '\n \ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n '\n \n-test_expect_success '%(raw) with --perl must fail' '\n-\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+test_expect_success '%(raw) with --perl' '\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob1 --perl | perl >actual &&\n+\tcmp blob1 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob3 --perl | perl >actual &&\n+\tcmp blob3 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob8 --perl | perl >actual &&\n+\tcmp blob8 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/first --perl | perl >actual &&\n+\tcmp one actual &&\n+\tgit cat-file tree refs/mytrees/first > expected &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/mytrees/first --perl | perl >actual &&\n+\tcmp expected actual\n '\n \n test_expect_success '%(raw) with --shell must fail' '\n-- \ngitgitgadget\n\n"},{"id":"429731","messageId":"6ad42c964053c9342554d9b43f8cbbb0e69385ad.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 09/19] [GSOC] ref-filter: introduce free_ref_array_item_value() function","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:49Z","receivedAt":"2021-07-12T11:47:17Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nWhen we use ref_array_item which is not dynamically allocated and\nwant to free the space of its member \"value\" after the end of use,\nfree_array_item() does not meet our needs, because it tries to free\nref_array_item itself and its member \"symref\".\n\nIntroduce free_ref_array_item_value() for freeing ref_array_item value.\nIt will be called internally by free_array_item(), and it will help\n`cat-file --batch` free ref_array_item's value memory later.\n\nHelped-by: Junio C Hamano \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 11 ++++++++---\n ref-filter.h | 2 ++\n 2 files changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 3189872188a..80b09fce1d5 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2292,16 +2292,21 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \treturn 0;\n }\n \n-/* Free memory allocated for a ref_array_item */\n-static void free_array_item(struct ref_array_item *item)\n+void free_ref_array_item_value(struct ref_array_item *item)\n {\n-\tfree((char *)item->symref);\n \tif (item->value) {\n \t\tint i;\n \t\tfor (i = 0; i < used_atom_cnt; i++)\n \t\t\tfree((char *)item->value[i].s);\n \t\tfree(item->value);\n \t}\n+}\n+\n+/* Free memory allocated for a ref_array_item */\n+static void free_array_item(struct ref_array_item *item)\n+{\n+\tfree((char *)item->symref);\n+\tfree_ref_array_item_value(item);\n \tfree(item);\n }\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex c15dee8d6b9..44e6dc05ac2 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -111,6 +111,8 @@ struct ref_format {\n int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type);\n /* Clear all memory allocated to ref_array */\n void ref_array_clear(struct ref_array *array);\n+/* Free ref_array_item's value */\n+void free_ref_array_item_value(struct ref_array_item *item);\n /* Used to verify if the given format is correct and to parse out the used atoms */\n int verify_ref_format(struct ref_format *format);\n /* Sort the given ref_array as per the ref_sorting provided */\n-- \ngitgitgadget\n\n"},{"id":"429732","messageId":"790c558d7cc1f66049c1280e6dddbe194806a0dc.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 11/19] [GSOC] ref-filter: modify the error message and value in get_object","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:51Z","receivedAt":"2021-07-12T11:47:19Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nLet get_object() return 1 and print \" missing\" instead\nof returning -1 and printing \"missing object for \"\nif oid_object_info_extended() unable to find the data corresponding\nto oid. When `cat-file --batch` use ref-filter logic later it can\nhelp `format_ref_array_item()` just report that the object is missing\nwithout letting Git exit.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 4 ++--\n t/t6301-for-each-ref-errors.sh | 2 +-\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 27199ba40f5..b4f41fec871 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1762,8 +1762,8 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t}\n \tif (oid_object_info_extended(the_repository, &oi->oid, &oi->info,\n \t\t\t\t OBJECT_INFO_LOOKUP_REPLACE))\n-\t\treturn strbuf_addf_ret(err, -1, _(\"missing object %s for %s\"),\n-\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n+\t\treturn strbuf_addf_ret(err, 1, _(\"%s missing\"),\n+\t\t\t\t oid_to_hex(&oi->oid));\n \tif (oi->info.disk_sizep && oi->disk_size < 0)\n \t\tBUG(\"Object size is less than zero.\");\n \ndiff --git a/t/t6301-for-each-ref-errors.sh b/t/t6301-for-each-ref-errors.sh\nindex 40edf9dab53..3553f84a00c 100755\n--- a/t/t6301-for-each-ref-errors.sh\n+++ b/t/t6301-for-each-ref-errors.sh\n@@ -41,7 +41,7 @@ test_expect_success 'Missing objects are reported correctly' '\n \tr=refs/heads/missing &&\n \techo $MISSING >.git/$r &&\n \ttest_when_finished \"rm -f .git/$r\" &&\n-\techo \"fatal: missing object $MISSING for $r\" >missing-err &&\n+\techo \"fatal: $MISSING missing\" >missing-err &&\n \ttest_must_fail git for-each-ref 2>err &&\n \ttest_cmp missing-err err &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"429733","messageId":"b61d538d53d7342f503077ecaaec7ea358110049.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 10/19] [GSOC] ref-filter: introduce reject_atom()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:50Z","receivedAt":"2021-07-12T11:47:21Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd `cat_file_mode` member in struct `ref_format` and introduce\nthe function `reject_atom()`, when `cat-file --batch` use ref-filter\nlogic later, it can help us reject atoms in verify_ref_format()\nwhich cat-file cannot use, e.g. `%(refname)`, `%(push)`,\n`%(upstream)`... or the atom `%(rest)` which for-each-ref family\ncannot use.\n\nHelped-by: Eric Sunshine \nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Christian Couder \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 25 ++++++++++++++++++++++---\n ref-filter.h | 1 +\n 2 files changed, 23 insertions(+), 3 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 80b09fce1d5..27199ba40f5 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1000,6 +1000,26 @@ static const char *find_next(const char *cp)\n \treturn NULL;\n }\n \n+static int reject_atom(int cat_file_mode, enum atom_type atom_type)\n+{\n+\tif (!cat_file_mode)\n+\t\treturn atom_type == ATOM_REST;\n+\n+\t/* cat_file_mode */\n+\tswitch (atom_type) {\n+\tcase ATOM_FLAG:\n+\tcase ATOM_HEAD:\n+\tcase ATOM_PUSH:\n+\tcase ATOM_REFNAME:\n+\tcase ATOM_SYMREF:\n+\tcase ATOM_UPSTREAM:\n+\tcase ATOM_WORKTREEPATH:\n+\t\treturn 1;\n+\tdefault:\n+\t\treturn 0;\n+\t}\n+}\n+\n /*\n * Make sure the format string is well formed, and parse out\n * the used atoms.\n@@ -1020,9 +1040,8 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n-\n-\t\tif (used_atom[at].atom_type == ATOM_REST)\n-\t\t\tdie(\"this command reject atom %%(%.*s)\", (int)(ep - sp - 2), sp + 2);\n+\t\tif (reject_atom(format->cat_file_mode, used_atom[at].atom_type))\n+\t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n \t\t format->quote_style == QUOTE_SHELL ||\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 44e6dc05ac2..053980a6a42 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -78,6 +78,7 @@ struct ref_format {\n \t */\n \tconst char *format;\n \tconst char *rest;\n+\tint cat_file_mode;\n \tint quote_style;\n \tint use_rest;\n \tint use_color;\n-- \ngitgitgadget\n\n"},{"id":"429734","messageId":"fca49379025048f3ed458b673483dcdf7f2dd10f.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 12/19] [GSOC] cat-file: add has_object_file() check","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:52Z","receivedAt":"2021-07-12T11:47:21Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse `has_object_file()` in `batch_one_object()` to check\nwhether the input object exists. This can help us reject\nthe missing oid when we let `cat-file --batch` use ref-filter\nlogic later.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 243fe6844bc..59a86412fd0 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -428,6 +428,13 @@ static void batch_one_object(const char *obj_name,\n \t\treturn;\n \t}\n \n+\tif (!has_object_file(&data->oid)) {\n+\t\tprintf(\"%s missing\\n\",\n+\t\t obj_name ? obj_name : oid_to_hex(&data->oid));\n+\t\tfflush(stdout);\n+\t\treturn;\n+\t}\n+\n \tbatch_object_write(obj_name, scratch, opt, data);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"429735","messageId":"43ce6bf26269dc80f87dabb52cbd2af63d24fd69.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 13/19] [GSOC] cat-file: change batch_objects parameter name","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:53Z","receivedAt":"2021-07-12T11:47:23Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause later cat-file reuses ref-filter logic that will add\nparameter \"const struct option *options\" to batch_objects(),\nthe two synonymous parameters of \"opt\" and \"options\" may\nconfuse readers, so change batch_options parameter of\nbatch_objects() from \"opt\" to \"batch\".\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 59a86412fd0..41d407638d5 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -495,7 +495,7 @@ static int batch_unordered_packed(const struct object_id *oid,\n \treturn batch_unordered_object(oid, data);\n }\n \n-static int batch_objects(struct batch_options *opt)\n+static int batch_objects(struct batch_options *batch)\n {\n \tstruct strbuf input = STRBUF_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n@@ -503,8 +503,8 @@ static int batch_objects(struct batch_options *opt)\n \tint save_warning;\n \tint retval = 0;\n \n-\tif (!opt->format)\n-\t\topt->format = \"%(objectname) %(objecttype) %(objectsize)\";\n+\tif (!batch->format)\n+\t\tbatch->format = \"%(objectname) %(objecttype) %(objectsize)\";\n \n \t/*\n \t * Expand once with our special mark_query flag, which will prime the\n@@ -513,20 +513,20 @@ static int batch_objects(struct batch_options *opt)\n \t */\n \tmemset(&data, 0, sizeof(data));\n \tdata.mark_query = 1;\n-\tstrbuf_expand(&output, opt->format, expand_format, &data);\n+\tstrbuf_expand(&output, batch->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (batch->cmdmode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (batch->print_contents)\n \t\tdata.info.typep = &data.type;\n \n-\tif (opt->all_objects) {\n+\tif (batch->all_objects) {\n \t\tstruct object_cb_data cb;\n \t\tstruct object_info empty = OBJECT_INFO_INIT;\n \n@@ -536,11 +536,11 @@ static int batch_objects(struct batch_options *opt)\n \t\tif (has_promisor_remote())\n \t\t\twarning(\"This repository uses promisor remotes. Some objects may not be loaded.\");\n \n-\t\tcb.opt = opt;\n+\t\tcb.opt = batch;\n \t\tcb.expand = &data;\n \t\tcb.scratch = &output;\n \n-\t\tif (opt->unordered) {\n+\t\tif (batch->unordered) {\n \t\t\tstruct oidset seen = OIDSET_INIT;\n \n \t\t\tcb.seen = &seen;\n@@ -590,7 +590,7 @@ static int batch_objects(struct batch_options *opt)\n \t\t\tdata.rest = p;\n \t\t}\n \n-\t\tbatch_one_object(input.buf, &output, opt, &data);\n+\t\tbatch_one_object(input.buf, &output, batch, &data);\n \t}\n \n \tstrbuf_release(&input);\n-- \ngitgitgadget\n\n"},{"id":"429736","messageId":"e20780e9a6cf5a3b76a40c99044575c288949c85.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 15/19] [GSOC] cat-file: reuse err buf in batch_object_write()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:55Z","receivedAt":"2021-07-12T11:47:24Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nReuse the `err` buffer in batch_object_write(), as the\nbuffer `scratch` does. This will reduce the overhead\nof multiple allocations of memory of the err buffer.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 22 ++++++++++++++--------\n 1 file changed, 14 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5b163551fc6..dc604a9879d 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -212,35 +212,36 @@ static void batch_write(struct batch_options *opt, const void *data, int len)\n \n static void batch_object_write(const char *obj_name,\n \t\t\t struct strbuf *scratch,\n+\t\t\t struct strbuf *err,\n \t\t\t struct batch_options *opt,\n \t\t\t struct expand_data *data)\n {\n \tint ret;\n-\tstruct strbuf err = STRBUF_INIT;\n \tstruct ref_array_item item = { data->oid, data->rest };\n \n \tstrbuf_reset(scratch);\n+\tstrbuf_reset(err);\n \n-\tret = format_ref_array_item(&item, &opt->format, scratch, &err);\n+\tret = format_ref_array_item(&item, &opt->format, scratch, err);\n \tif (ret < 0)\n-\t\tdie(\"%s\\n\", err.buf);\n+\t\tdie(\"%s\\n\", err->buf);\n \tif (ret) {\n \t\t/* ret > 0 means when the object corresponding to oid\n \t\t * cannot be found in format_ref_array_item(), we only print\n \t\t * the error message.\n \t\t */\n-\t\tprintf(\"%s\\n\", err.buf);\n+\t\tprintf(\"%s\\n\", err->buf);\n \t\tfflush(stdout);\n \t} else {\n \t\tstrbuf_addch(scratch, '\\n');\n \t\tbatch_write(opt, scratch->buf, scratch->len);\n \t}\n \tfree_ref_array_item_value(&item);\n-\tstrbuf_release(&err);\n }\n \n static void batch_one_object(const char *obj_name,\n \t\t\t struct strbuf *scratch,\n+\t\t\t struct strbuf *err,\n \t\t\t struct batch_options *opt,\n \t\t\t struct expand_data *data)\n {\n@@ -294,7 +295,7 @@ static void batch_one_object(const char *obj_name,\n \t\treturn;\n \t}\n \n-\tbatch_object_write(obj_name, scratch, opt, data);\n+\tbatch_object_write(obj_name, scratch, err, opt, data);\n }\n \n struct object_cb_data {\n@@ -302,13 +303,14 @@ struct object_cb_data {\n \tstruct expand_data *expand;\n \tstruct oidset *seen;\n \tstruct strbuf *scratch;\n+\tstruct strbuf *err;\n };\n \n static int batch_object_cb(const struct object_id *oid, void *vdata)\n {\n \tstruct object_cb_data *data = vdata;\n \toidcpy(&data->expand->oid, oid);\n-\tbatch_object_write(NULL, data->scratch, data->opt, data->expand);\n+\tbatch_object_write(NULL, data->scratch, data->err, data->opt, data->expand);\n \treturn 0;\n }\n \n@@ -364,6 +366,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n {\n \tstruct strbuf input = STRBUF_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n+\tstruct strbuf err = STRBUF_INIT;\n \tstruct strbuf format = STRBUF_INIT;\n \tstruct expand_data data;\n \tint save_warning;\n@@ -392,6 +395,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n \t\tcb.opt = batch;\n \t\tcb.expand = &data;\n \t\tcb.scratch = &output;\n+\t\tcb.err = &err;\n \n \t\tif (batch->unordered) {\n \t\t\tstruct oidset seen = OIDSET_INIT;\n@@ -416,6 +420,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n \n \t\tstrbuf_release(&format);\n \t\tstrbuf_release(&output);\n+\t\tstrbuf_release(&err);\n \t\treturn 0;\n \t}\n \n@@ -444,12 +449,13 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n \t\t\tdata.rest = p;\n \t\t}\n \n-\t\tbatch_one_object(input.buf, &output, batch, &data);\n+\t\tbatch_one_object(input.buf, &output, &err, batch, &data);\n \t}\n \n \tstrbuf_release(&format);\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n+\tstrbuf_release(&err);\n \twarn_on_object_refname_ambiguity = save_warning;\n \treturn retval;\n }\n-- \ngitgitgadget\n\n"},{"id":"429737","messageId":"fa74bf9451cb452ced0e49a6ad5accb984449c06.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 16/19] [GSOC] cat-file: re-implement --textconv, --filters options","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:56Z","receivedAt":"2021-07-12T11:47:26Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAfter cat-file reuses the ref-filter logic, we re-implement the\nfunctions of --textconv and --filters options.\n\nAdd members `cat_file_cmdmode` to struct `ref_array_item`,\nso that struct `batch_option` member `cmdmode` will be passed\nto ref-filter, and then ref-filter will take use of it to filter\nthe content of the object in get_object().\n\nUse `actual_oi` to record the real expand_data: it may point to the\noriginal `oi` or the `act_oi` processed by `textconv_object()` or\n`convert_to_working_tree()`. `grab_values()` will grab the contents\nof `actual_oi` and `grab_common_values()` to grab the contents of origin\n`oi`, this ensures that `%(objectsize)` still uses the size of the\nunfiltered data.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 2 +-\n ref-filter.c | 35 +++++++++++++++++++++++++++++++++--\n ref-filter.h | 1 +\n 3 files changed, 35 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex dc604a9879d..3a6153e778f 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -217,7 +217,7 @@ static void batch_object_write(const char *obj_name,\n \t\t\t struct expand_data *data)\n {\n \tint ret;\n-\tstruct ref_array_item item = { data->oid, data->rest };\n+\tstruct ref_array_item item = { data->oid, data->rest, opt->cmdmode };\n \n \tstrbuf_reset(scratch);\n \tstrbuf_reset(err);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex b4f41fec871..91e26c9aba3 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1,3 +1,4 @@\n+#define USE_THE_INDEX_COMPATIBILITY_MACROS\n #include \"builtin.h\"\n #include \"cache.h\"\n #include \"parse-options.h\"\n@@ -1755,6 +1756,9 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n {\n \t/* parse_object_buffer() will set eaten to 0 if free() will be needed */\n \tint eaten = 1;\n+\tstruct expand_data *actual_oi = oi;\n+\tstruct expand_data act_oi = {0};\n+\n \tif (oi->info.contentp) {\n \t\t/* We need to know that to use parse_object_buffer properly */\n \t\toi->info.sizep = &oi->size;\n@@ -1768,19 +1772,45 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\tBUG(\"Object size is less than zero.\");\n \n \tif (oi->info.contentp) {\n-\t\t*obj = parse_object_buffer(the_repository, &oi->oid, oi->type, oi->size, oi->content, &eaten);\n+\t\tif ((ref->cat_file_cmdmode == 'c' || ref->cat_file_cmdmode == 'w') && !ref->rest)\n+\t\t\treturn strbuf_addf_ret(err, -1, _(\"missing path for '%s'\"),\n+\t\t\t\t\t oid_to_hex(&act_oi.oid));\n+\t\tif (oi->type == OBJ_BLOB) {\n+\t\t\tif (ref->cat_file_cmdmode == 'c') {\n+\t\t\t\tact_oi = *oi;\n+\t\t\t\tif (textconv_object(the_repository,\n+\t\t\t\t\t\t ref->rest, 0100644, &act_oi.oid,\n+\t\t\t\t\t\t 1, (char **)(&act_oi.content), &act_oi.size))\n+\t\t\t\t\tactual_oi = &act_oi;\n+\t\t\t} else if (ref->cat_file_cmdmode == 'w') {\n+\t\t\t\tstruct strbuf strbuf = STRBUF_INIT;\n+\t\t\t\tstruct checkout_metadata meta;\n+\t\t\t\tact_oi = *oi;\n+\n+\t\t\t\tinit_checkout_metadata(&meta, NULL, NULL, &act_oi.oid);\n+\t\t\t\tif (!convert_to_working_tree(&the_index, ref->rest, act_oi.content, act_oi.size, &strbuf, &meta))\n+\t\t\t\t\tdie(\"could not convert '%s' %s\",\n+\t\t\t\t\t oid_to_hex(&oi->oid), ref->rest);\n+\t\t\t\tact_oi.size = strbuf.len;\n+\t\t\t\tact_oi.content = strbuf_detach(&strbuf, NULL);\n+\t\t\t\tactual_oi = &act_oi;\n+\t\t\t}\n+\t\t}\n+\t\t*obj = parse_object_buffer(the_repository, &actual_oi->oid, actual_oi->type, actual_oi->size, actual_oi->content, &eaten);\n \t\tif (!*obj) {\n \t\t\tif (!eaten)\n \t\t\t\tfree(oi->content);\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi);\n+\t\tgrab_values(ref->value, deref, *obj, actual_oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n \tif (!eaten)\n \t\tfree(oi->content);\n+\tif (actual_oi != oi)\n+\t\tfree(actual_oi->content);\n \treturn 0;\n }\n \n@@ -2189,6 +2219,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n \tref->rest = NULL;\n+\tref->cat_file_cmdmode = 0;\n \n \treturn ref;\n }\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 053980a6a42..a93d5e4dd61 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -39,6 +39,7 @@ struct ref_sorting {\n struct ref_array_item {\n \tstruct object_id objectname;\n \tconst char *rest;\n+\tint cat_file_cmdmode;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n-- \ngitgitgadget\n\n"},{"id":"429739","messageId":"70e83e4ba3cc8a55bb8d90cb2c581cd04ac116d4.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:54Z","receivedAt":"2021-07-12T11:47:27Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nIn order to let cat-file use ref-filter logic, let's do the\nfollowing:\n\n1. Change the type of member `format` in struct `batch_options`\nto `ref_format`, we will pass it to ref-filter later.\n2. Let `batch_objects()` add atoms to format, and use\n`verify_ref_format()` to check atoms.\n3. Use `format_ref_array_item()` in `batch_object_write()` to\nget the formatted data corresponding to the object. If the\nreturn value of `format_ref_array_item()` is equals to zero,\nuse `batch_write()` to print object data; else if the return\nvalue is less than zero, use `die()` to print the error message\nand exit; else if return value is greater than zero, only print\nthe error message, but don't exit.\n4. Use free_ref_array_item_value() to free ref_array_item's\nvalue.\n\nMost of the atoms in `for-each-ref --format` are now supported,\nsuch as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`,\n`%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected:\n`%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`,\n`%(flag)`, `%(HEAD)`, because these atoms are unique to those objects\nthat pointed to by a ref, \"for-each-ref\"'s family can naturally use\nthese atoms, but not all objects are pointed to be a ref, so \"cat-file\"\nwill not be able to use them.\n\nThe performance for `git cat-file --batch-all-objects\n--batch-check` on the Git repository itself with performance\ntesting tool `hyperfine` changes from 669.4 ms ± 31.1 ms to\n1.134 s ± 0.063 s.\n\nThe performance for `git cat-file --batch-all-objects --batch\n>/dev/null` on the Git repository itself with performance testing\ntool `time` change from \"27.37s user 0.29s system 98% cpu 28.089\ntotal\" to \"33.69s user 1.54s system 87% cpu 40.258 total\".\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-cat-file.txt | 6 +\n builtin/cat-file.c | 242 ++++++-------------------------\n t/t1006-cat-file.sh | 251 +++++++++++++++++++++++++++++++++\n 3 files changed, 304 insertions(+), 195 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex 4eb0421b3fd..ef8ab952b2f 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -226,6 +226,12 @@ newline. The available atoms are:\n \tafter that first run of whitespace (i.e., the \"rest\" of the\n \tline) are output in place of the `%(rest)` atom.\n \n+Note that most of the atoms in `for-each-ref --format` are now supported,\n+such as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`,\n+`%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected:\n+`%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`,\n+`%(flag)`, `%(HEAD)`. See linkgit:git-for-each-ref[1].\n+\n If no format is specified, the default format is `%(objectname)\n %(objecttype) %(objectsize)`.\n \ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 41d407638d5..5b163551fc6 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -16,6 +16,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n+#include \"ref-filter.h\"\n \n struct batch_options {\n \tint enabled;\n@@ -25,7 +26,7 @@ struct batch_options {\n \tint all_objects;\n \tint unordered;\n \tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n-\tconst char *format;\n+\tstruct ref_format format;\n };\n \n static const char *force_path;\n@@ -195,99 +196,10 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \n struct expand_data {\n \tstruct object_id oid;\n-\tenum object_type type;\n-\tunsigned long size;\n-\toff_t disk_size;\n \tconst char *rest;\n-\tstruct object_id delta_base_oid;\n-\n-\t/*\n-\t * If mark_query is true, we do not expand anything, but rather\n-\t * just mark the object_info with items we wish to query.\n-\t */\n-\tint mark_query;\n-\n-\t/*\n-\t * Whether to split the input on whitespace before feeding it to\n-\t * get_sha1; this is decided during the mark_query phase based on\n-\t * whether we have a %(rest) token in our format.\n-\t */\n \tint split_on_whitespace;\n-\n-\t/*\n-\t * After a mark_query run, this object_info is set up to be\n-\t * passed to oid_object_info_extended. It will point to the data\n-\t * elements above, so you can retrieve the response from there.\n-\t */\n-\tstruct object_info info;\n-\n-\t/*\n-\t * This flag will be true if the requested batch format and options\n-\t * don't require us to call oid_object_info, which can then be\n-\t * optimized out.\n-\t */\n-\tunsigned skip_object_info : 1;\n };\n \n-static int is_atom(const char *atom, const char *s, int slen)\n-{\n-\tint alen = strlen(atom);\n-\treturn alen == slen && !memcmp(atom, s, alen);\n-}\n-\n-static void expand_atom(struct strbuf *sb, const char *atom, int len,\n-\t\t\tvoid *vdata)\n-{\n-\tstruct expand_data *data = vdata;\n-\n-\tif (is_atom(\"objectname\", atom, len)) {\n-\t\tif (!data->mark_query)\n-\t\t\tstrbuf_addstr(sb, oid_to_hex(&data->oid));\n-\t} else if (is_atom(\"objecttype\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->info.typep = &data->type;\n-\t\telse\n-\t\t\tstrbuf_addstr(sb, type_name(data->type));\n-\t} else if (is_atom(\"objectsize\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->info.sizep = &data->size;\n-\t\telse\n-\t\t\tstrbuf_addf(sb, \"%\"PRIuMAX , (uintmax_t)data->size);\n-\t} else if (is_atom(\"objectsize:disk\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->info.disk_sizep = &data->disk_size;\n-\t\telse\n-\t\t\tstrbuf_addf(sb, \"%\"PRIuMAX, (uintmax_t)data->disk_size);\n-\t} else if (is_atom(\"rest\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->split_on_whitespace = 1;\n-\t\telse if (data->rest)\n-\t\t\tstrbuf_addstr(sb, data->rest);\n-\t} else if (is_atom(\"deltabase\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->info.delta_base_oid = &data->delta_base_oid;\n-\t\telse\n-\t\t\tstrbuf_addstr(sb,\n-\t\t\t\t oid_to_hex(&data->delta_base_oid));\n-\t} else\n-\t\tdie(\"unknown format element: %.*s\", len, atom);\n-}\n-\n-static size_t expand_format(struct strbuf *sb, const char *start, void *data)\n-{\n-\tconst char *end;\n-\n-\tif (*start != '(')\n-\t\treturn 0;\n-\tend = strchr(start + 1, ')');\n-\tif (!end)\n-\t\tdie(\"format element '%s' does not end in ')'\", start);\n-\n-\texpand_atom(sb, start + 1, end - start - 1, data);\n-\n-\treturn end - start + 1;\n-}\n-\n static void batch_write(struct batch_options *opt, const void *data, int len)\n {\n \tif (opt->buffer_output) {\n@@ -297,87 +209,34 @@ static void batch_write(struct batch_options *opt, const void *data, int len)\n \t\twrite_or_die(1, data, len);\n }\n \n-static void print_object_or_die(struct batch_options *opt, struct expand_data *data)\n-{\n-\tconst struct object_id *oid = &data->oid;\n-\n-\tassert(data->info.typep);\n-\n-\tif (data->type == OBJ_BLOB) {\n-\t\tif (opt->buffer_output)\n-\t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n-\t\t\tchar *contents;\n-\t\t\tunsigned long size;\n-\n-\t\t\tif (!data->rest)\n-\t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n-\n-\t\t\tif (opt->cmdmode == 'w') {\n-\t\t\t\tif (filter_object(data->rest, 0100644, oid,\n-\t\t\t\t\t\t &contents, &size))\n-\t\t\t\t\tdie(\"could not convert '%s' %s\",\n-\t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n-\t\t\t\tenum object_type type;\n-\t\t\t\tif (!textconv_object(the_repository,\n-\t\t\t\t\t\t data->rest, 0100644, oid,\n-\t\t\t\t\t\t 1, &contents, &size))\n-\t\t\t\t\tcontents = read_object_file(oid,\n-\t\t\t\t\t\t\t\t &type,\n-\t\t\t\t\t\t\t\t &size);\n-\t\t\t\tif (!contents)\n-\t\t\t\t\tdie(\"could not convert '%s' %s\",\n-\t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n-\t\t\tbatch_write(opt, contents, size);\n-\t\t\tfree(contents);\n-\t\t} else {\n-\t\t\tstream_blob(oid);\n-\t\t}\n-\t}\n-\telse {\n-\t\tenum object_type type;\n-\t\tunsigned long size;\n-\t\tvoid *contents;\n-\n-\t\tcontents = read_object_file(oid, &type, &size);\n-\t\tif (!contents)\n-\t\t\tdie(\"object %s disappeared\", oid_to_hex(oid));\n-\t\tif (type != data->type)\n-\t\t\tdie(\"object %s changed type!?\", oid_to_hex(oid));\n-\t\tif (data->info.sizep && size != data->size)\n-\t\t\tdie(\"object %s changed size!?\", oid_to_hex(oid));\n-\n-\t\tbatch_write(opt, contents, size);\n-\t\tfree(contents);\n-\t}\n-}\n \n static void batch_object_write(const char *obj_name,\n \t\t\t struct strbuf *scratch,\n \t\t\t struct batch_options *opt,\n \t\t\t struct expand_data *data)\n {\n-\tif (!data->skip_object_info &&\n-\t oid_object_info_extended(the_repository, &data->oid, &data->info,\n-\t\t\t\t OBJECT_INFO_LOOKUP_REPLACE) < 0) {\n-\t\tprintf(\"%s missing\\n\",\n-\t\t obj_name ? obj_name : oid_to_hex(&data->oid));\n-\t\tfflush(stdout);\n-\t\treturn;\n-\t}\n+\tint ret;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_array_item item = { data->oid, data->rest };\n \n \tstrbuf_reset(scratch);\n-\tstrbuf_expand(scratch, opt->format, expand_format, data);\n-\tstrbuf_addch(scratch, '\\n');\n-\tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n-\t\tprint_object_or_die(opt, data);\n-\t\tbatch_write(opt, \"\\n\", 1);\n+\tret = format_ref_array_item(&item, &opt->format, scratch, &err);\n+\tif (ret < 0)\n+\t\tdie(\"%s\\n\", err.buf);\n+\tif (ret) {\n+\t\t/* ret > 0 means when the object corresponding to oid\n+\t\t * cannot be found in format_ref_array_item(), we only print\n+\t\t * the error message.\n+\t\t */\n+\t\tprintf(\"%s\\n\", err.buf);\n+\t\tfflush(stdout);\n+\t} else {\n+\t\tstrbuf_addch(scratch, '\\n');\n+\t\tbatch_write(opt, scratch->buf, scratch->len);\n \t}\n+\tfree_ref_array_item_value(&item);\n+\tstrbuf_release(&err);\n }\n \n static void batch_one_object(const char *obj_name,\n@@ -495,43 +354,37 @@ static int batch_unordered_packed(const struct object_id *oid,\n \treturn batch_unordered_object(oid, data);\n }\n \n-static int batch_objects(struct batch_options *batch)\n+static const char * const cat_file_usage[] = {\n+\tN_(\"git cat-file (-t [--allow-unknown-type] | -s [--allow-unknown-type] | -e | -p | | --textconv | --filters) [--path=] \"),\n+\tN_(\"git cat-file (--batch[=] | --batch-check[=]) [--follow-symlinks] [--textconv | --filters]\"),\n+\tNULL\n+};\n+\n+static int batch_objects(struct batch_options *batch, const struct option *options)\n {\n \tstruct strbuf input = STRBUF_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n+\tstruct strbuf format = STRBUF_INIT;\n \tstruct expand_data data;\n \tint save_warning;\n \tint retval = 0;\n \n-\tif (!batch->format)\n-\t\tbatch->format = \"%(objectname) %(objecttype) %(objectsize)\";\n-\n-\t/*\n-\t * Expand once with our special mark_query flag, which will prime the\n-\t * object_info to be handed to oid_object_info_extended for each\n-\t * object.\n-\t */\n \tmemset(&data, 0, sizeof(data));\n-\tdata.mark_query = 1;\n-\tstrbuf_expand(&output, batch->format, expand_format, &data);\n-\tdata.mark_query = 0;\n-\tstrbuf_release(&output);\n-\tif (batch->cmdmode)\n-\t\tdata.split_on_whitespace = 1;\n-\n-\t/*\n-\t * If we are printing out the object, then always fill in the type,\n-\t * since we will want to decide whether or not to stream.\n-\t */\n+\tif (batch->format.format)\n+\t\tstrbuf_addstr(&format, batch->format.format);\n+\telse\n+\t\tstrbuf_addstr(&format, \"%(objectname) %(objecttype) %(objectsize)\");\n \tif (batch->print_contents)\n-\t\tdata.info.typep = &data.type;\n+\t\tstrbuf_addstr(&format, \"\\n%(raw)\");\n+\tbatch->format.format = format.buf;\n+\tif (verify_ref_format(&batch->format))\n+\t\tusage_with_options(cat_file_usage, options);\n+\n+\tif (batch->cmdmode || batch->format.use_rest)\n+\t\tdata.split_on_whitespace = 1;\n \n \tif (batch->all_objects) {\n \t\tstruct object_cb_data cb;\n-\t\tstruct object_info empty = OBJECT_INFO_INIT;\n-\n-\t\tif (!memcmp(&data.info, &empty, sizeof(empty)))\n-\t\t\tdata.skip_object_info = 1;\n \n \t\tif (has_promisor_remote())\n \t\t\twarning(\"This repository uses promisor remotes. Some objects may not be loaded.\");\n@@ -561,6 +414,7 @@ static int batch_objects(struct batch_options *batch)\n \t\t\toid_array_clear(&sa);\n \t\t}\n \n+\t\tstrbuf_release(&format);\n \t\tstrbuf_release(&output);\n \t\treturn 0;\n \t}\n@@ -593,18 +447,13 @@ static int batch_objects(struct batch_options *batch)\n \t\tbatch_one_object(input.buf, &output, batch, &data);\n \t}\n \n+\tstrbuf_release(&format);\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n \treturn retval;\n }\n \n-static const char * const cat_file_usage[] = {\n-\tN_(\"git cat-file (-t [--allow-unknown-type] | -s [--allow-unknown-type] | -e | -p | | --textconv | --filters) [--path=] \"),\n-\tN_(\"git cat-file (--batch[=] | --batch-check[=]) [--follow-symlinks] [--textconv | --filters]\"),\n-\tNULL\n-};\n-\n static int git_cat_file_config(const char *var, const char *value, void *cb)\n {\n \tif (userdiff_config(var, value) < 0)\n@@ -627,7 +476,7 @@ static int batch_option_callback(const struct option *opt,\n \n \tbo->enabled = 1;\n \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n-\tbo->format = arg;\n+\tbo->format.format = arg;\n \n \treturn 0;\n }\n@@ -636,7 +485,9 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n {\n \tint opt = 0;\n \tconst char *exp_type = NULL, *obj_name = NULL;\n-\tstruct batch_options batch = {0};\n+\tstruct batch_options batch = {\n+\t\t.format = REF_FORMAT_INIT\n+\t};\n \tint unknown_type = 0;\n \n \tconst struct option options[] = {\n@@ -675,6 +526,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \tgit_config(git_cat_file_config, NULL);\n \n \tbatch.buffer_output = -1;\n+\tbatch.format.cat_file_mode = 1;\n \targc = parse_options(argc, argv, prefix, options, cat_file_usage, 0);\n \n \tif (opt) {\n@@ -718,7 +570,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\tbatch.buffer_output = batch.all_objects;\n \n \tif (batch.enabled)\n-\t\treturn batch_objects(&batch);\n+\t\treturn batch_objects(&batch, options);\n \n \tif (unknown_type && opt != 't' && opt != 's')\n \t\tdie(\"git cat-file --allow-unknown-type: use with -s or -t\");\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 18b3779ccb6..7452404f24a 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -607,5 +607,256 @@ test_expect_success 'cat-file --batch=\"batman\" with --batch-all-objects will wor\n \tgit -C all-two cat-file --batch-all-objects --batch=\"batman\" >actual &&\n \tcmp expect actual\n '\n+. \"$TEST_DIRECTORY\"/lib-gpg.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n+\n+test_expect_success 'cat-file --batch|--batch-check setup' '\n+\techo 1>blob1 &&\n+\tprintf \"a\\0b\\0\\c\" >blob2 &&\n+\tgit add blob1 blob2 &&\n+\tgit commit -m \"Commit Message\" &&\n+\tgit branch -M main &&\n+\tgit tag -a -m \"v0.0.0\" testtag &&\n+\tgit update-ref refs/myblobs/blob1 HEAD:blob1 &&\n+\tgit update-ref refs/myblobs/blob2 HEAD:blob2 &&\n+\tgit update-ref refs/mytrees/tree1 HEAD^{tree}\n+'\n+\n+batch_test_atom() {\n+\tif test \"$3\" = \"fail\"\n+\tthen\n+\t\ttest_expect_${4:-success} $PREREQ \"basic atom: $1 $2 must fail\" \"\n+\t\t\ttest_must_fail git cat-file --batch-check='$2' >bad <<-EOF\n+\t\t\t$1\n+\t\t\tEOF\n+\t\t\"\n+\telse\n+\t\ttest_expect_${4:-success} $PREREQ \"basic atom: $1 $2\" \"\n+\t\t\tgit for-each-ref --format='$2' $1 >expected &&\n+\t\t\tgit cat-file --batch-check='$2' >actual <<-EOF &&\n+\t\t\t$1\n+\t\t\tEOF\n+\t\t\tsanitize_pgp actual.clean &&\n+\t\t\tcmp expected actual.clean\n+\t\t\"\n+\tfi\n+}\n+\n+batch_test_atom refs/heads/main '%(refname)' fail\n+batch_test_atom refs/heads/main '%(refname:)' fail\n+batch_test_atom refs/heads/main '%(refname:short)' fail\n+batch_test_atom refs/heads/main '%(refname:lstrip=1)' fail\n+batch_test_atom refs/heads/main '%(refname:lstrip=2)' fail\n+batch_test_atom refs/heads/main '%(refname:lstrip=-1)' fail\n+batch_test_atom refs/heads/main '%(refname:lstrip=-2)' fail\n+batch_test_atom refs/heads/main '%(refname:rstrip=1)' fail\n+batch_test_atom refs/heads/main '%(refname:rstrip=2)' fail\n+batch_test_atom refs/heads/main '%(refname:rstrip=-1)' fail\n+batch_test_atom refs/heads/main '%(refname:rstrip=-2)' fail\n+batch_test_atom refs/heads/main '%(refname:strip=1)' fail\n+batch_test_atom refs/heads/main '%(refname:strip=2)' fail\n+batch_test_atom refs/heads/main '%(refname:strip=-1)' fail\n+batch_test_atom refs/heads/main '%(refname:strip=-2)' fail\n+batch_test_atom refs/heads/main '%(upstream)' fail\n+batch_test_atom refs/heads/main '%(upstream:short)' fail\n+batch_test_atom refs/heads/main '%(upstream:lstrip=2)' fail\n+batch_test_atom refs/heads/main '%(upstream:lstrip=-2)' fail\n+batch_test_atom refs/heads/main '%(upstream:rstrip=2)' fail\n+batch_test_atom refs/heads/main '%(upstream:rstrip=-2)' fail\n+batch_test_atom refs/heads/main '%(upstream:strip=2)' fail\n+batch_test_atom refs/heads/main '%(upstream:strip=-2)' fail\n+batch_test_atom refs/heads/main '%(push)' fail\n+batch_test_atom refs/heads/main '%(push:short)' fail\n+batch_test_atom refs/heads/main '%(push:lstrip=1)' fail\n+batch_test_atom refs/heads/main '%(push:lstrip=-1)' fail\n+batch_test_atom refs/heads/main '%(push:rstrip=1)' fail\n+batch_test_atom refs/heads/main '%(push:rstrip=-1)' fail\n+batch_test_atom refs/heads/main '%(push:strip=1)' fail\n+batch_test_atom refs/heads/main '%(push:strip=-1)' fail\n+batch_test_atom refs/heads/main '%(objecttype)'\n+batch_test_atom refs/heads/main '%(objectsize)'\n+batch_test_atom refs/heads/main '%(objectsize:disk)'\n+batch_test_atom refs/heads/main '%(deltabase)'\n+batch_test_atom refs/heads/main '%(objectname)'\n+batch_test_atom refs/heads/main '%(objectname:short)'\n+batch_test_atom refs/heads/main '%(objectname:short=1)'\n+batch_test_atom refs/heads/main '%(objectname:short=10)'\n+batch_test_atom refs/heads/main '%(tree)'\n+batch_test_atom refs/heads/main '%(tree:short)'\n+batch_test_atom refs/heads/main '%(tree:short=1)'\n+batch_test_atom refs/heads/main '%(tree:short=10)'\n+batch_test_atom refs/heads/main '%(parent)'\n+batch_test_atom refs/heads/main '%(parent:short)'\n+batch_test_atom refs/heads/main '%(parent:short=1)'\n+batch_test_atom refs/heads/main '%(parent:short=10)'\n+batch_test_atom refs/heads/main '%(numparent)'\n+batch_test_atom refs/heads/main '%(object)'\n+batch_test_atom refs/heads/main '%(type)'\n+batch_test_atom refs/heads/main '%(raw)'\n+batch_test_atom refs/heads/main '%(*objectname)'\n+batch_test_atom refs/heads/main '%(*objecttype)'\n+batch_test_atom refs/heads/main '%(author)'\n+batch_test_atom refs/heads/main '%(authorname)'\n+batch_test_atom refs/heads/main '%(authoremail)'\n+batch_test_atom refs/heads/main '%(authoremail:trim)'\n+batch_test_atom refs/heads/main '%(authoremail:localpart)'\n+batch_test_atom refs/heads/main '%(authordate)'\n+batch_test_atom refs/heads/main '%(committer)'\n+batch_test_atom refs/heads/main '%(committername)'\n+batch_test_atom refs/heads/main '%(committeremail)'\n+batch_test_atom refs/heads/main '%(committeremail:trim)'\n+batch_test_atom refs/heads/main '%(committeremail:localpart)'\n+batch_test_atom refs/heads/main '%(committerdate)'\n+batch_test_atom refs/heads/main '%(tag)'\n+batch_test_atom refs/heads/main '%(tagger)'\n+batch_test_atom refs/heads/main '%(taggername)'\n+batch_test_atom refs/heads/main '%(taggeremail)'\n+batch_test_atom refs/heads/main '%(taggeremail:trim)'\n+batch_test_atom refs/heads/main '%(taggeremail:localpart)'\n+batch_test_atom refs/heads/main '%(taggerdate)'\n+batch_test_atom refs/heads/main '%(creator)'\n+batch_test_atom refs/heads/main '%(creatordate)'\n+batch_test_atom refs/heads/main '%(subject)'\n+batch_test_atom refs/heads/main '%(subject:sanitize)'\n+batch_test_atom refs/heads/main '%(contents:subject)'\n+batch_test_atom refs/heads/main '%(body)'\n+batch_test_atom refs/heads/main '%(contents:body)'\n+batch_test_atom refs/heads/main '%(contents:signature)'\n+batch_test_atom refs/heads/main '%(contents)'\n+batch_test_atom refs/heads/main '%(HEAD)' fail\n+batch_test_atom refs/heads/main '%(upstream:track)' fail\n+batch_test_atom refs/heads/main '%(upstream:trackshort)' fail\n+batch_test_atom refs/heads/main '%(upstream:track,nobracket)' fail\n+batch_test_atom refs/heads/main '%(upstream:nobracket,track)' fail\n+batch_test_atom refs/heads/main '%(push:track)' fail\n+batch_test_atom refs/heads/main '%(push:trackshort)' fail\n+batch_test_atom refs/heads/main '%(worktreepath)' fail\n+batch_test_atom refs/heads/main '%(symref)' fail\n+batch_test_atom refs/heads/main '%(flag)' fail\n+\n+batch_test_atom refs/tags/testtag '%(refname)' fail\n+batch_test_atom refs/tags/testtag '%(refname:short)' fail\n+batch_test_atom refs/tags/testtag '%(upstream)' fail\n+batch_test_atom refs/tags/testtag '%(push)' fail\n+batch_test_atom refs/tags/testtag '%(objecttype)'\n+batch_test_atom refs/tags/testtag '%(objectsize)'\n+batch_test_atom refs/tags/testtag '%(objectsize:disk)'\n+batch_test_atom refs/tags/testtag '%(*objectsize:disk)'\n+batch_test_atom refs/tags/testtag '%(deltabase)'\n+batch_test_atom refs/tags/testtag '%(*deltabase)'\n+batch_test_atom refs/tags/testtag '%(objectname)'\n+batch_test_atom refs/tags/testtag '%(objectname:short)'\n+batch_test_atom refs/tags/testtag '%(tree)'\n+batch_test_atom refs/tags/testtag '%(tree:short)'\n+batch_test_atom refs/tags/testtag '%(tree:short=1)'\n+batch_test_atom refs/tags/testtag '%(tree:short=10)'\n+batch_test_atom refs/tags/testtag '%(parent)'\n+batch_test_atom refs/tags/testtag '%(parent:short)'\n+batch_test_atom refs/tags/testtag '%(parent:short=1)'\n+batch_test_atom refs/tags/testtag '%(parent:short=10)'\n+batch_test_atom refs/tags/testtag '%(numparent)'\n+batch_test_atom refs/tags/testtag '%(object)'\n+batch_test_atom refs/tags/testtag '%(type)'\n+batch_test_atom refs/tags/testtag '%(*objectname)'\n+batch_test_atom refs/tags/testtag '%(*objecttype)'\n+batch_test_atom refs/tags/testtag '%(author)'\n+batch_test_atom refs/tags/testtag '%(authorname)'\n+batch_test_atom refs/tags/testtag '%(authoremail)'\n+batch_test_atom refs/tags/testtag '%(authoremail:trim)'\n+batch_test_atom refs/tags/testtag '%(authoremail:localpart)'\n+batch_test_atom refs/tags/testtag '%(authordate)'\n+batch_test_atom refs/tags/testtag '%(committer)'\n+batch_test_atom refs/tags/testtag '%(committername)'\n+batch_test_atom refs/tags/testtag '%(committeremail)'\n+batch_test_atom refs/tags/testtag '%(committeremail:trim)'\n+batch_test_atom refs/tags/testtag '%(committeremail:localpart)'\n+batch_test_atom refs/tags/testtag '%(committerdate)'\n+batch_test_atom refs/tags/testtag '%(tag)'\n+batch_test_atom refs/tags/testtag '%(tagger)'\n+batch_test_atom refs/tags/testtag '%(taggername)'\n+batch_test_atom refs/tags/testtag '%(taggeremail)'\n+batch_test_atom refs/tags/testtag '%(taggeremail:trim)'\n+batch_test_atom refs/tags/testtag '%(taggeremail:localpart)'\n+batch_test_atom refs/tags/testtag '%(taggerdate)'\n+batch_test_atom refs/tags/testtag '%(creator)'\n+batch_test_atom refs/tags/testtag '%(creatordate)'\n+batch_test_atom refs/tags/testtag '%(subject)'\n+batch_test_atom refs/tags/testtag '%(subject:sanitize)'\n+batch_test_atom refs/tags/testtag '%(contents:subject)'\n+batch_test_atom refs/tags/testtag '%(body)'\n+batch_test_atom refs/tags/testtag '%(contents:body)'\n+batch_test_atom refs/tags/testtag '%(contents:signature)'\n+batch_test_atom refs/tags/testtag '%(contents)'\n+batch_test_atom refs/tags/testtag '%(HEAD)' fail\n+\n+batch_test_atom refs/myblobs/blob1 '%(refname)' fail\n+batch_test_atom refs/myblobs/blob1 '%(upstream)' fail\n+batch_test_atom refs/myblobs/blob1 '%(push)' fail\n+batch_test_atom refs/myblobs/blob1 '%(HEAD)' fail\n+\n+batch_test_atom refs/myblobs/blob1 '%(objectname)'\n+batch_test_atom refs/myblobs/blob1 '%(objecttype)'\n+batch_test_atom refs/myblobs/blob1 '%(objectsize)'\n+batch_test_atom refs/myblobs/blob1 '%(objectsize:disk)'\n+batch_test_atom refs/myblobs/blob1 '%(deltabase)'\n+\n+batch_test_atom refs/myblobs/blob1 '%(contents)'\n+batch_test_atom refs/myblobs/blob2 '%(contents)'\n+\n+batch_test_atom refs/myblobs/blob1 '%(raw)'\n+batch_test_atom refs/myblobs/blob2 '%(raw)'\n+batch_test_atom refs/mytrees/tree1 '%(raw)'\n+\n+batch_test_atom refs/myblobs/blob1 '%(raw:size)'\n+batch_test_atom refs/myblobs/blob2 '%(raw:size)'\n+batch_test_atom refs/mytrees/tree1 '%(raw:size)'\n+\n+batch_test_atom refs/myblobs/blob1 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)'\n+batch_test_atom refs/myblobs/blob2 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)'\n+batch_test_atom refs/mytrees/tree1 '%(if:equals=tree)%(objecttype)%(then)tree%(else)not tree%(end)'\n+\n+batch_test_atom refs/heads/main '%(align:60) objectname is %(objectname)%(end)|%(objectname)'\n+batch_test_atom refs/heads/main '%(align:left,60) objectname is %(objectname)%(end)|%(objectname)'\n+batch_test_atom refs/heads/main '%(align:middle,60) objectname is %(objectname)%(end)|%(objectname)'\n+batch_test_atom refs/heads/main '%(align:60,right) objectname is %(objectname)%(end)|%(objectname)'\n+\n+batch_test_atom refs/heads/main 'VALID'\n+batch_test_atom refs/heads/main '%(INVALID)' fail\n+batch_test_atom refs/heads/main '%(authordate:INVALID)' fail\n+\n+test_expect_success '%(rest) works with both a branch and a tag' '\n+\tcat >expected <<-EOF &&\n+\t123 commit 123\n+\t456 tag 456\n+\tEOF\n+\tgit cat-file --batch-check=\"%(rest) %(objecttype) %(rest)\" >actual <<-EOF &&\n+\trefs/heads/main 123\n+\trefs/tags/testtag 456\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+batch_test_atom refs/heads/main '%(objectname) %(objecttype) %(objectsize)\n+%(raw)'\n+batch_test_atom refs/tags/testtag '%(objectname) %(objecttype) %(objectsize)\n+%(raw)'\n+batch_test_atom refs/myblobs/blob1 '%(objectname) %(objecttype) %(objectsize)\n+%(raw)'\n+batch_test_atom refs/myblobs/blob2 '%(objectname) %(objecttype) %(objectsize)\n+%(raw)'\n+\n+\n+test_expect_success 'cat-file --batch equals to --batch-check with atoms' '\n+\tgit cat-file --batch-check=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" >expected <<-EOF &&\n+\trefs/heads/main\n+\trefs/tags/testtag\n+\tEOF\n+\tgit cat-file --batch >actual <<-EOF &&\n+\trefs/heads/main\n+\trefs/tags/testtag\n+\tEOF\n+\tcmp expected actual\n+'\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"429738","messageId":"ff74fa9f2f2fb1ea77782254da74c2d0ce6c6bb8.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 17/19] [GSOC] ref-filter: remove grab_oid() function","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:57Z","receivedAt":"2021-07-12T11:47:28Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause \"atom_type == ATOM_OBJECTNAME\" implies the condition\nof `starts_with(name, \"objectname\")`, \"atom_type == ATOM_TREE\"\nimplies the condition of `starts_with(name, \"tree\")`, so the\ncheck for `starts_with(name, field)` in grab_oid() is redundant.\n\nSo Remove the grab_oid() from ref-filter, to reduce repeated check.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 26 +++++++++-----------------\n 1 file changed, 9 insertions(+), 17 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 91e26c9aba3..1c7287f1061 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1077,16 +1077,6 @@ static const char *do_grab_oid(const char *field, const struct object_id *oid,\n \t}\n }\n \n-static int grab_oid(const char *name, const char *field, const struct object_id *oid,\n-\t\t struct atom_value *v, struct used_atom *atom)\n-{\n-\tif (starts_with(name, field)) {\n-\t\tv->s = xstrdup(do_grab_oid(field, oid, atom));\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n /* See grab_values */\n static void grab_common_values(struct atom_value *val, int deref, struct expand_data *oi)\n {\n@@ -1112,8 +1102,9 @@ static void grab_common_values(struct atom_value *val, int deref, struct expand_\n \t\t\t}\n \t\t} else if (atom_type == ATOM_DELTABASE)\n \t\t\tv->s = xstrdup(oid_to_hex(&oi->delta_base_oid));\n-\t\telse if (atom_type == ATOM_OBJECTNAME && deref)\n-\t\t\tgrab_oid(name, \"objectname\", &oi->oid, v, &used_atom[i]);\n+\t\telse if (atom_type == ATOM_OBJECTNAME && deref) {\n+\t\t\tv->s = xstrdup(do_grab_oid(\"objectname\", &oi->oid, &used_atom[i]));\n+\t\t}\n \t}\n }\n \n@@ -1154,9 +1145,10 @@ static void grab_commit_values(struct atom_value *val, int deref, struct object\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (atom_type == ATOM_TREE &&\n-\t\t grab_oid(name, \"tree\", get_commit_tree_oid(commit), v, &used_atom[i]))\n+\t\tif (atom_type == ATOM_TREE) {\n+\t\t\tv->s = xstrdup(do_grab_oid(\"tree\", get_commit_tree_oid(commit), &used_atom[i]));\n \t\t\tcontinue;\n+\t\t}\n \t\tif (atom_type == ATOM_NUMPARENT) {\n \t\t\tv->value = commit_list_count(commit->parents);\n \t\t\tv->s = xstrfmt(\"%lu\", (unsigned long)v->value);\n@@ -1959,9 +1951,9 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\t\tv->s = xstrdup(buf + 1);\n \t\t\t}\n \t\t\tcontinue;\n-\t\t} else if (!deref && atom_type == ATOM_OBJECTNAME &&\n-\t\t\t grab_oid(name, \"objectname\", &ref->objectname, v, atom)) {\n-\t\t\t\tcontinue;\n+\t\t} else if (!deref && atom_type == ATOM_OBJECTNAME) {\n+\t\t\t v->s = xstrdup(do_grab_oid(\"objectname\", &ref->objectname, atom));\n+\t\t\t continue;\n \t\t} else if (atom_type == ATOM_HEAD) {\n \t\t\tif (atom->u.head && !strcmp(ref->refname, atom->u.head))\n \t\t\t\tv->s = xstrdup(\"*\");\n-- \ngitgitgadget\n\n"},{"id":"429740","messageId":"2edca9dc46538d85bc32a206e71c3336e384df41.1626090419.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH 18/19] [GSOC] cat-file: create p1006-cat-file.sh","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T11:46:58Z","receivedAt":"2021-07-12T11:47:28Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nCreate p1006-cat-file.sh to provide performance testing for\n`git cat-file --batch` and `git cat-file --batch-check`. This\nwill help us compare the performance changes after we let\ncat-file reuse the ref-filter logic.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n t/perf/p1006-cat-file.sh | 28 ++++++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n create mode 100755 t/perf/p1006-cat-file.sh\n\ndiff --git a/t/perf/p1006-cat-file.sh b/t/perf/p1006-cat-file.sh\nnew file mode 100755\nindex 00000000000..b84ac31f9cc\n--- /dev/null\n+++ b/t/perf/p1006-cat-file.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='Basic sort performance tests'\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+test_expect_success 'setup' '\n+\tgit rev-list --all >rla\n+'\n+\n+test_perf 'cat-file --batch-check' '\n+\tgit cat-file --batch-check \n\nAdd the member `default_format` to struct `batch_options`,\nwhen we are using the default format on `git cat-file --batch`,\nor `git cat-file --batch-check`, `default_format` will be set,\nif we don't use `--textconv` or `--filter`, then we will not call\nverify_ref_format(), has_object_file() and format_ref_array_item().\nInstead, we get the object data directly through\noid_object_info_extended() and then output the data directly.\n\nBy using this fast path, we can reduce some of the extra overhead\nwhen `cat-file --batch` using ref-filter. The running time of\n`git cat-file --batch-check` will be similar to before, and the\nrunning time of `git cat-file --batch` will be 9.1% less than before.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 79 +++++++++++++++++++++++++++++++++-------------\n 1 file changed, 57 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 3a6153e778f..8edc19f2d5a 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -26,6 +26,7 @@ struct batch_options {\n \tint all_objects;\n \tint unordered;\n \tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint default_format;\n \tstruct ref_format format;\n };\n \n@@ -196,6 +197,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \n struct expand_data {\n \tstruct object_id oid;\n+\tstruct object_info info;\n \tconst char *rest;\n \tint split_on_whitespace;\n };\n@@ -216,27 +218,58 @@ static void batch_object_write(const char *obj_name,\n \t\t\t struct batch_options *opt,\n \t\t\t struct expand_data *data)\n {\n-\tint ret;\n-\tstruct ref_array_item item = { data->oid, data->rest, opt->cmdmode };\n-\n-\tstrbuf_reset(scratch);\n-\tstrbuf_reset(err);\n-\n-\tret = format_ref_array_item(&item, &opt->format, scratch, err);\n-\tif (ret < 0)\n-\t\tdie(\"%s\\n\", err->buf);\n-\tif (ret) {\n-\t\t/* ret > 0 means when the object corresponding to oid\n-\t\t * cannot be found in format_ref_array_item(), we only print\n-\t\t * the error message.\n-\t\t */\n-\t\tprintf(\"%s\\n\", err->buf);\n+\tif (opt->default_format && !opt->cmdmode) {\n+\t\tstruct strbuf type_name = STRBUF_INIT;\n+\t\tunsigned long size;\n+\t\tvoid *content;\n+\n+\t\tif (opt->print_contents)\n+\t\t\tdata->info.contentp = &content;\n+\n+\t\tdata->info.type_name = &type_name;\n+\t\tdata->info.sizep = &size;\n+\n+\t\tif (oid_object_info_extended(the_repository, &data->oid, &data->info,\n+\t\t\t\t\t OBJECT_INFO_LOOKUP_REPLACE) < 0) {\n+\t\t\tprintf(\"%s missing\\n\",\n+\t\t\t obj_name ? obj_name : oid_to_hex(&data->oid));\n+\t\t\tfflush(stdout);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tfprintf(stdout, \"%s %s %\"PRIuMAX\"\\n\", oid_to_hex(&data->oid),\n+\t\t\tdata->info.type_name->buf,\n+\t\t\t(uintmax_t)*data->info.sizep);\n \t\tfflush(stdout);\n+\t\tstrbuf_release(&type_name);\n+\t\tif (opt->print_contents) {\n+\t\t\tbatch_write(opt, content, *data->info.sizep);\n+\t\t\tbatch_write(opt, \"\\n\", 1);\n+\t\t\tfree(content);\n+\t\t}\n \t} else {\n-\t\tstrbuf_addch(scratch, '\\n');\n-\t\tbatch_write(opt, scratch->buf, scratch->len);\n+\t\tint ret;\n+\t\tstruct ref_array_item item = { data->oid, data->rest, opt->cmdmode };\n+\n+\t\tstrbuf_reset(scratch);\n+\t\tstrbuf_reset(err);\n+\n+\t\tret = format_ref_array_item(&item, &opt->format, scratch, err);\n+\t\tif (ret < 0)\n+\t\t\tdie(\"%s\\n\", err->buf);\n+\t\tif (ret) {\n+\t\t\t/* ret > 0 means when the object corresponding to oid\n+\t\t\t * cannot be found in format_ref_array_item(), we only print\n+\t\t\t * the error message.\n+\t\t\t */\n+\t\t\tprintf(\"%s\\n\", err->buf);\n+\t\t\tfflush(stdout);\n+\t\t} else {\n+\t\t\tstrbuf_addch(scratch, '\\n');\n+\t\t\tbatch_write(opt, scratch->buf, scratch->len);\n+\t\t}\n+\t\tfree_ref_array_item_value(&item);\n \t}\n-\tfree_ref_array_item_value(&item);\n }\n \n static void batch_one_object(const char *obj_name,\n@@ -288,7 +321,7 @@ static void batch_one_object(const char *obj_name,\n \t\treturn;\n \t}\n \n-\tif (!has_object_file(&data->oid)) {\n+\tif ((!opt->default_format || opt->cmdmode) && !has_object_file(&data->oid)) {\n \t\tprintf(\"%s missing\\n\",\n \t\t obj_name ? obj_name : oid_to_hex(&data->oid));\n \t\tfflush(stdout);\n@@ -380,7 +413,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n \tif (batch->print_contents)\n \t\tstrbuf_addstr(&format, \"\\n%(raw)\");\n \tbatch->format.format = format.buf;\n-\tif (verify_ref_format(&batch->format))\n+\tif ((!batch->default_format || batch->cmdmode) && verify_ref_format(&batch->format))\n \t\tusage_with_options(cat_file_usage, options);\n \n \tif (batch->cmdmode || batch->format.use_rest)\n@@ -483,7 +516,8 @@ static int batch_option_callback(const struct option *opt,\n \tbo->enabled = 1;\n \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n \tbo->format.format = arg;\n-\n+\tif (arg)\n+\t\tbo->default_format = 0;\n \treturn 0;\n }\n \n@@ -492,7 +526,8 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \tint opt = 0;\n \tconst char *exp_type = NULL, *obj_name = NULL;\n \tstruct batch_options batch = {\n-\t\t.format = REF_FORMAT_INIT\n+\t\t.format = REF_FORMAT_INIT,\n+\t\t.default_format = 1\n \t};\n \tint unknown_type = 0;\n \n-- \ngitgitgadget\n"},{"id":"429743","messageId":"CAP8UFD2gfwB-bEgj6pd__2zkSU20rxr3qN8HOYuLaQYmcuCynw@mail.gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-12T12:36:11Z","receivedAt":"2021-07-12T12:36:28Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Jul 12, 2021 at 1:47 PM ZheNing Hu via GitGitGadget\n wrote:\n>\n> This patch series make cat-file reuse ref-filter logic.\n\ns/make/makes/\n\nBy the way if you have already sent some of the patches in this series\n(and if they haven't changed much since when you sent them), it's a\ngood idea to use V2 or V3, V4, etc, so we can easily refer to each of\nthe versions you sent. (See the `-v` option of `git format-patch`.)\n\n> Change from last version:\n>\n> 1. Declare buf_size in if (atom_type == ATOM_RAW) block.\n> 2. Modify the code style of the test.\n> 3. Delete \"use_textconv\" and \"use_filter\" flag. Instead, add member\n\ns/flag/flags/\n\n> cat_file_cmdmode to struct ref_array_item.\n> 4. Add function reject_atom() to enhance the readability of the code.\n> 5. Create p1006-cat-file.sh for performance regression testing.\n> 6. Use a \"fast path\" to output object data to reduce the performance\n> degradation of cat-file --batch with the suggest of Ævar Arnfjörð\n> Bjarmason.\n\nMaybe:\n\ns/with the suggest of Ævar Arnfjörð Bjarmason/as suggested by Ævar\nArnfjörð Bjarmason/\n\nor:\n\ns/with the suggest of Ævar Arnfjörð Bjarmason/according to Ævar\nArnfjörð Bjarmason's suggestion/\n\n\n> ZheNing Hu (19):\n> cat-file: handle trivial --batch format with --batch-all-objects\n> cat-file: merge two block into one\n\nIt's a bit strange that the above 2 don't have [GSOC] while the others\nbelow have it.\n\n> [GSOC] ref-filter: add obj-type check in grab contents\n> [GSOC] ref-filter: add %(raw) atom\n> [GSOC] ref-filter: --format=%(raw) re-support --perl\n> [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n> [GSOC] ref-filter: add %(rest) atom\n> [GSOC] ref-filter: pass get_object() return value to their callers\n> [GSOC] ref-filter: introduce free_ref_array_item_value() function\n> [GSOC] ref-filter: introduce reject_atom()\n> [GSOC] ref-filter: modify the error message and value in get_object\n> [GSOC] cat-file: add has_object_file() check\n> [GSOC] cat-file: change batch_objects parameter name\n> [GSOC] cat-file: reuse ref-filter logic\n> [GSOC] cat-file: reuse err buf in batch_object_write()\n> [GSOC] cat-file: re-implement --textconv, --filters options\n> [GSOC] ref-filter: remove grab_oid() function\n> [GSOC] cat-file: create p1006-cat-file.sh\n\nMaybe you could add the new perf test earlier in the series so that we\ncould see how performance changes when ref-filter logic is reused in\ncat-file earlier in the series.\n\n> [GSOC] cat-file: use fast path when using default_format\n"},{"id":"429744","messageId":"CAOLTT8TwwBQ+OXAgVvGSOQiM19GWy+Y=m1QxQ8nc6SaE=1x=9w@mail.gmail.com","threadId":"56095","inReplyTo":"CAP8UFD2gfwB-bEgj6pd__2zkSU20rxr3qN8HOYuLaQYmcuCynw@mail.gmail.com","subject":"Re: [PATCH 00/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-12T13:01:19Z","receivedAt":"2021-07-12T13:01:12Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年7月12日周一 下午8:36写道：\n>\n> On Mon, Jul 12, 2021 at 1:47 PM ZheNing Hu via GitGitGadget\n> wrote:\n> >\n> > This patch series make cat-file reuse ref-filter logic.\n>\n> s/make/makes/\n>\n> By the way if you have already sent some of the patches in this series\n> (and if they haven't changed much since when you sent them), it's a\n> good idea to use V2 or V3, V4, etc, so we can easily refer to each of\n> the versions you sent. (See the `-v` option of `git format-patch`.)\n>\n\nYes, but my patch was generated by gitgitgadget, when I want to send\na new patch which is totally different with before, I will send a new PR.\nGGG will not know the connection between these two PRs. So the\nversion number of the patch will be reset.\n\n> > Change from last version:\n> >\n> > 1. Declare buf_size in if (atom_type == ATOM_RAW) block.\n> > 2. Modify the code style of the test.\n> > 3. Delete \"use_textconv\" and \"use_filter\" flag. Instead, add member\n>\n> s/flag/flags/\n>\n> > cat_file_cmdmode to struct ref_array_item.\n> > 4. Add function reject_atom() to enhance the readability of the code.\n> > 5. Create p1006-cat-file.sh for performance regression testing.\n> > 6. Use a \"fast path\" to output object data to reduce the performance\n> > degradation of cat-file --batch with the suggest of Ævar Arnfjörð\n> > Bjarmason.\n>\n> Maybe:\n>\n> s/with the suggest of Ævar Arnfjörð Bjarmason/as suggested by Ævar\n> Arnfjörð Bjarmason/\n>\n> or:\n>\n> s/with the suggest of Ævar Arnfjörð Bjarmason/according to Ævar\n> Arnfjörð Bjarmason's suggestion/\n>\n>\n> > ZheNing Hu (19):\n> > cat-file: handle trivial --batch format with --batch-all-objects\n> > cat-file: merge two block into one\n>\n> It's a bit strange that the above 2 don't have [GSOC] while the others\n> below have it.\n>\n\nThat's because it's belong to the branch zh/cat-file-batch-fix. I\nshould mention it\nin the cover-letter.\n\n> > [GSOC] ref-filter: add obj-type check in grab contents\n> > [GSOC] ref-filter: add %(raw) atom\n> > [GSOC] ref-filter: --format=%(raw) re-support --perl\n> > [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n> > [GSOC] ref-filter: add %(rest) atom\n> > [GSOC] ref-filter: pass get_object() return value to their callers\n> > [GSOC] ref-filter: introduce free_ref_array_item_value() function\n> > [GSOC] ref-filter: introduce reject_atom()\n> > [GSOC] ref-filter: modify the error message and value in get_object\n> > [GSOC] cat-file: add has_object_file() check\n> > [GSOC] cat-file: change batch_objects parameter name\n> > [GSOC] cat-file: reuse ref-filter logic\n> > [GSOC] cat-file: reuse err buf in batch_object_write()\n> > [GSOC] cat-file: re-implement --textconv, --filters options\n> > [GSOC] ref-filter: remove grab_oid() function\n> > [GSOC] cat-file: create p1006-cat-file.sh\n>\n> Maybe you could add the new perf test earlier in the series so that we\n> could see how performance changes when ref-filter logic is reused in\n> cat-file earlier in the series.\n\nMake sence.\n\n>\n> > [GSOC] cat-file: use fast path when using default_format\n\nThanks.\n--\nZheNing Hu\n"},{"id":"429745","messageId":"b6f661d6-d18f-9b26-da64-acad31b07188@iee.email","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2021-07-12T13:02:06Z","receivedAt":"2021-07-12T13:02:13Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 12/07/2021 12:46, ZheNing Hu via GitGitGadget wrote:\n> This patch series make cat-file reuse ref-filter logic.\n>\n> Change from last version:\nminor nit..\nNot sure if this is a gitgitgadget feature, but would it be possible\nthat a version indication be included in future versions of the patch,\ne.g. [PATCH vN 00/19] [GSOC] ?\n--\nPhilip\n"},{"id":"429746","messageId":"CAP8UFD0vHw8cK90RbOUcrAcxNKiHVTMX0VEiK8+MNyHH0CNm4A@mail.gmail.com","threadId":"56095","inReplyTo":"70e83e4ba3cc8a55bb8d90cb2c581cd04ac116d4.1626090419.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-12T13:17:33Z","receivedAt":"2021-07-12T13:17:49Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Jul 12, 2021 at 1:47 PM ZheNing Hu via GitGitGadget\n wrote:\n>\n> From: ZheNing Hu \n>\n> In order to let cat-file use ref-filter logic, let's do the\n> following:\n>\n> 1. Change the type of member `format` in struct `batch_options`\n> to `ref_format`, we will pass it to ref-filter later.\n> 2. Let `batch_objects()` add atoms to format, and use\n> `verify_ref_format()` to check atoms.\n> 3. Use `format_ref_array_item()` in `batch_object_write()` to\n> get the formatted data corresponding to the object. If the\n> return value of `format_ref_array_item()` is equals to zero,\n> use `batch_write()` to print object data; else if the return\n> value is less than zero, use `die()` to print the error message\n> and exit; else if return value is greater than zero, only print\n> the error message, but don't exit.\n> 4. Use free_ref_array_item_value() to free ref_array_item's\n> value.\n>\n> Most of the atoms in `for-each-ref --format` are now supported,\n> such as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`,\n> `%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected:\n> `%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`,\n> `%(flag)`, `%(HEAD)`, because these atoms are unique to those objects\n> that pointed to by a ref, \"for-each-ref\"'s family can naturally use\n> these atoms, but not all objects are pointed to be a ref, so \"cat-file\"\n> will not be able to use them.\n>\n> The performance for `git cat-file --batch-all-objects\n> --batch-check` on the Git repository itself with performance\n> testing tool `hyperfine` changes from 669.4 ms ± 31.1 ms to\n> 1.134 s ± 0.063 s.\n>\n> The performance for `git cat-file --batch-all-objects --batch\n> >/dev/null` on the Git repository itself with performance testing\n> tool `time` change from \"27.37s user 0.29s system 98% cpu 28.089\n> total\" to \"33.69s user 1.54s system 87% cpu 40.258 total\".\n\nSaying that a later patch will add a fast path which will mitigate the\nperformance regression introduced by this patch might help reassure\nreviewers.\n\nBy the way it is not clear if adding the fast path fully mitigates\nthis performance regression or not. You might want to discuss that in\nthe cover letter, or maybe in the patch adding the fast path.\n\n> Mentored-by: Christian Couder \n> Mentored-by: Hariom Verma \n> Signed-off-by: ZheNing Hu \n> ---\n> Documentation/git-cat-file.txt | 6 +\n> builtin/cat-file.c | 242 ++++++-------------------------\n> t/t1006-cat-file.sh | 251 +++++++++++++++++++++++++++++++++\n> 3 files changed, 304 insertions(+), 195 deletions(-)\n>\n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> index 4eb0421b3fd..ef8ab952b2f 100644\n> --- a/Documentation/git-cat-file.txt\n> +++ b/Documentation/git-cat-file.txt\n> @@ -226,6 +226,12 @@ newline. The available atoms are:\n> after that first run of whitespace (i.e., the \"rest\" of the\n> line) are output in place of the `%(rest)` atom.\n>\n> +Note that most of the atoms in `for-each-ref --format` are now supported,\n> +such as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`,\n> +`%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected:\n> +`%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`,\n> +`%(flag)`, `%(HEAD)`. See linkgit:git-for-each-ref[1].\n> +\n> If no format is specified, the default format is `%(objectname)\n> %(objecttype) %(objectsize)`.\n>\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 41d407638d5..5b163551fc6 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -16,6 +16,7 @@\n> #include \"packfile.h\"\n> #include \"object-store.h\"\n> #include \"promisor-remote.h\"\n> +#include \"ref-filter.h\"\n>\n> struct batch_options {\n> int enabled;\n> @@ -25,7 +26,7 @@ struct batch_options {\n> int all_objects;\n> int unordered;\n> int cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n> - const char *format;\n> + struct ref_format format;\n> };\n>\n> static const char *force_path;\n> @@ -195,99 +196,10 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n>\n> struct expand_data {\n> struct object_id oid;\n> - enum object_type type;\n> - unsigned long size;\n> - off_t disk_size;\n> const char *rest;\n> - struct object_id delta_base_oid;\n> -\n> - /*\n> - * If mark_query is true, we do not expand anything, but rather\n> - * just mark the object_info with items we wish to query.\n> - */\n> - int mark_query;\n> -\n> - /*\n> - * Whether to split the input on whitespace before feeding it to\n> - * get_sha1; this is decided during the mark_query phase based on\n> - * whether we have a %(rest) token in our format.\n> - */\n> int split_on_whitespace;\n> -\n> - /*\n> - * After a mark_query run, this object_info is set up to be\n> - * passed to oid_object_info_extended. It will point to the data\n> - * elements above, so you can retrieve the response from there.\n> - */\n> - struct object_info info;\n> -\n> - /*\n> - * This flag will be true if the requested batch format and options\n> - * don't require us to call oid_object_info, which can then be\n> - * optimized out.\n> - */\n> - unsigned skip_object_info : 1;\n> };\n>\n> -static int is_atom(const char *atom, const char *s, int slen)\n> -{\n> - int alen = strlen(atom);\n> - return alen == slen && !memcmp(atom, s, alen);\n> -}\n> -\n> -static void expand_atom(struct strbuf *sb, const char *atom, int len,\n> - void *vdata)\n> -{\n> - struct expand_data *data = vdata;\n> -\n> - if (is_atom(\"objectname\", atom, len)) {\n> - if (!data->mark_query)\n> - strbuf_addstr(sb, oid_to_hex(&data->oid));\n> - } else if (is_atom(\"objecttype\", atom, len)) {\n> - if (data->mark_query)\n> - data->info.typep = &data->type;\n> - else\n> - strbuf_addstr(sb, type_name(data->type));\n> - } else if (is_atom(\"objectsize\", atom, len)) {\n> - if (data->mark_query)\n> - data->info.sizep = &data->size;\n> - else\n> - strbuf_addf(sb, \"%\"PRIuMAX , (uintmax_t)data->size);\n> - } else if (is_atom(\"objectsize:disk\", atom, len)) {\n> - if (data->mark_query)\n> - data->info.disk_sizep = &data->disk_size;\n> - else\n> - strbuf_addf(sb, \"%\"PRIuMAX, (uintmax_t)data->disk_size);\n> - } else if (is_atom(\"rest\", atom, len)) {\n> - if (data->mark_query)\n> - data->split_on_whitespace = 1;\n> - else if (data->rest)\n> - strbuf_addstr(sb, data->rest);\n> - } else if (is_atom(\"deltabase\", atom, len)) {\n> - if (data->mark_query)\n> - data->info.delta_base_oid = &data->delta_base_oid;\n> - else\n> - strbuf_addstr(sb,\n> - oid_to_hex(&data->delta_base_oid));\n> - } else\n> - die(\"unknown format element: %.*s\", len, atom);\n> -}\n> -\n> -static size_t expand_format(struct strbuf *sb, const char *start, void *data)\n> -{\n> - const char *end;\n> -\n> - if (*start != '(')\n> - return 0;\n> - end = strchr(start + 1, ')');\n> - if (!end)\n> - die(\"format element '%s' does not end in ')'\", start);\n> -\n> - expand_atom(sb, start + 1, end - start - 1, data);\n> -\n> - return end - start + 1;\n> -}\n> -\n> static void batch_write(struct batch_options *opt, const void *data, int len)\n> {\n> if (opt->buffer_output) {\n> @@ -297,87 +209,34 @@ static void batch_write(struct batch_options *opt, const void *data, int len)\n> write_or_die(1, data, len);\n> }\n>\n> -static void print_object_or_die(struct batch_options *opt, struct expand_data *data)\n> -{\n> - const struct object_id *oid = &data->oid;\n> -\n> - assert(data->info.typep);\n> -\n> - if (data->type == OBJ_BLOB) {\n> - if (opt->buffer_output)\n> - fflush(stdout);\n> - if (opt->cmdmode) {\n> - char *contents;\n> - unsigned long size;\n> -\n> - if (!data->rest)\n> - die(\"missing path for '%s'\", oid_to_hex(oid));\n> -\n> - if (opt->cmdmode == 'w') {\n> - if (filter_object(data->rest, 0100644, oid,\n> - &contents, &size))\n> - die(\"could not convert '%s' %s\",\n> - oid_to_hex(oid), data->rest);\n> - } else if (opt->cmdmode == 'c') {\n> - enum object_type type;\n> - if (!textconv_object(the_repository,\n> - data->rest, 0100644, oid,\n> - 1, &contents, &size))\n> - contents = read_object_file(oid,\n> - &type,\n> - &size);\n> - if (!contents)\n> - die(\"could not convert '%s' %s\",\n> - oid_to_hex(oid), data->rest);\n> - } else\n> - BUG(\"invalid cmdmode: %c\", opt->cmdmode);\n> - batch_write(opt, contents, size);\n> - free(contents);\n> - } else {\n> - stream_blob(oid);\n> - }\n> - }\n> - else {\n> - enum object_type type;\n> - unsigned long size;\n> - void *contents;\n> -\n> - contents = read_object_file(oid, &type, &size);\n> - if (!contents)\n> - die(\"object %s disappeared\", oid_to_hex(oid));\n> - if (type != data->type)\n> - die(\"object %s changed type!?\", oid_to_hex(oid));\n> - if (data->info.sizep && size != data->size)\n> - die(\"object %s changed size!?\", oid_to_hex(oid));\n> -\n> - batch_write(opt, contents, size);\n> - free(contents);\n> - }\n> -}\n>\n> static void batch_object_write(const char *obj_name,\n> struct strbuf *scratch,\n> struct batch_options *opt,\n> struct expand_data *data)\n> {\n> - if (!data->skip_object_info &&\n> - oid_object_info_extended(the_repository, &data->oid, &data->info,\n> - OBJECT_INFO_LOOKUP_REPLACE) < 0) {\n> - printf(\"%s missing\\n\",\n> - obj_name ? obj_name : oid_to_hex(&data->oid));\n> - fflush(stdout);\n> - return;\n> - }\n> + int ret;\n> + struct strbuf err = STRBUF_INIT;\n> + struct ref_array_item item = { data->oid, data->rest };\n>\n> strbuf_reset(scratch);\n> - strbuf_expand(scratch, opt->format, expand_format, data);\n> - strbuf_addch(scratch, '\\n');\n> - batch_write(opt, scratch->buf, scratch->len);\n>\n> - if (opt->print_contents) {\n> - print_object_or_die(opt, data);\n> - batch_write(opt, \"\\n\", 1);\n> + ret = format_ref_array_item(&item, &opt->format, scratch, &err);\n> + if (ret < 0)\n> + die(\"%s\\n\", err.buf);\n> + if (ret) {\n> + /* ret > 0 means when the object corresponding to oid\n> + * cannot be found in format_ref_array_item(), we only print\n> + * the error message.\n> + */\n> + printf(\"%s\\n\", err.buf);\n> + fflush(stdout);\n> + } else {\n> + strbuf_addch(scratch, '\\n');\n> + batch_write(opt, scratch->buf, scratch->len);\n> }\n> + free_ref_array_item_value(&item);\n> + strbuf_release(&err);\n> }\n>\n> static void batch_one_object(const char *obj_name,\n> @@ -495,43 +354,37 @@ static int batch_unordered_packed(const struct object_id *oid,\n> return batch_unordered_object(oid, data);\n> }\n>\n> -static int batch_objects(struct batch_options *batch)\n> +static const char * const cat_file_usage[] = {\n> + N_(\"git cat-file (-t [--allow-unknown-type] | -s [--allow-unknown-type] | -e | -p | | --textconv | --filters) [--path=] \"),\n> + N_(\"git cat-file (--batch[=] | --batch-check[=]) [--follow-symlinks] [--textconv | --filters]\"),\n> + NULL\n> +};\n> +\n> +static int batch_objects(struct batch_options *batch, const struct option *options)\n> {\n> struct strbuf input = STRBUF_INIT;\n> struct strbuf output = STRBUF_INIT;\n> + struct strbuf format = STRBUF_INIT;\n> struct expand_data data;\n> int save_warning;\n> int retval = 0;\n>\n> - if (!batch->format)\n> - batch->format = \"%(objectname) %(objecttype) %(objectsize)\";\n> -\n> - /*\n> - * Expand once with our special mark_query flag, which will prime the\n> - * object_info to be handed to oid_object_info_extended for each\n> - * object.\n> - */\n> memset(&data, 0, sizeof(data));\n> - data.mark_query = 1;\n> - strbuf_expand(&output, batch->format, expand_format, &data);\n> - data.mark_query = 0;\n> - strbuf_release(&output);\n> - if (batch->cmdmode)\n> - data.split_on_whitespace = 1;\n> -\n> - /*\n> - * If we are printing out the object, then always fill in the type,\n> - * since we will want to decide whether or not to stream.\n> - */\n> + if (batch->format.format)\n> + strbuf_addstr(&format, batch->format.format);\n> + else\n> + strbuf_addstr(&format, \"%(objectname) %(objecttype) %(objectsize)\");\n> if (batch->print_contents)\n> - data.info.typep = &data.type;\n> + strbuf_addstr(&format, \"\\n%(raw)\");\n> + batch->format.format = format.buf;\n> + if (verify_ref_format(&batch->format))\n> + usage_with_options(cat_file_usage, options);\n> +\n> + if (batch->cmdmode || batch->format.use_rest)\n> + data.split_on_whitespace = 1;\n>\n> if (batch->all_objects) {\n> struct object_cb_data cb;\n> - struct object_info empty = OBJECT_INFO_INIT;\n> -\n> - if (!memcmp(&data.info, &empty, sizeof(empty)))\n> - data.skip_object_info = 1;\n>\n> if (has_promisor_remote())\n> warning(\"This repository uses promisor remotes. Some objects may not be loaded.\");\n> @@ -561,6 +414,7 @@ static int batch_objects(struct batch_options *batch)\n> oid_array_clear(&sa);\n> }\n>\n> + strbuf_release(&format);\n> strbuf_release(&output);\n> return 0;\n> }\n> @@ -593,18 +447,13 @@ static int batch_objects(struct batch_options *batch)\n> batch_one_object(input.buf, &output, batch, &data);\n> }\n>\n> + strbuf_release(&format);\n> strbuf_release(&input);\n> strbuf_release(&output);\n> warn_on_object_refname_ambiguity = save_warning;\n> return retval;\n> }\n>\n> -static const char * const cat_file_usage[] = {\n> - N_(\"git cat-file (-t [--allow-unknown-type] | -s [--allow-unknown-type] | -e | -p | | --textconv | --filters) [--path=] \"),\n> - N_(\"git cat-file (--batch[=] | --batch-check[=]) [--follow-symlinks] [--textconv | --filters]\"),\n> - NULL\n> -};\n> -\n> static int git_cat_file_config(const char *var, const char *value, void *cb)\n> {\n> if (userdiff_config(var, value) < 0)\n> @@ -627,7 +476,7 @@ static int batch_option_callback(const struct option *opt,\n>\n> bo->enabled = 1;\n> bo->print_contents = !strcmp(opt->long_name, \"batch\");\n> - bo->format = arg;\n> + bo->format.format = arg;\n>\n> return 0;\n> }\n> @@ -636,7 +485,9 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n> {\n> int opt = 0;\n> const char *exp_type = NULL, *obj_name = NULL;\n> - struct batch_options batch = {0};\n> + struct batch_options batch = {\n> + .format = REF_FORMAT_INIT\n> + };\n> int unknown_type = 0;\n>\n> const struct option options[] = {\n> @@ -675,6 +526,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n> git_config(git_cat_file_config, NULL);\n>\n> batch.buffer_output = -1;\n> + batch.format.cat_file_mode = 1;\n> argc = parse_options(argc, argv, prefix, options, cat_file_usage, 0);\n>\n> if (opt) {\n> @@ -718,7 +570,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n> batch.buffer_output = batch.all_objects;\n>\n> if (batch.enabled)\n> - return batch_objects(&batch);\n> + return batch_objects(&batch, options);\n>\n> if (unknown_type && opt != 't' && opt != 's')\n> die(\"git cat-file --allow-unknown-type: use with -s or -t\");\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 18b3779ccb6..7452404f24a 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -607,5 +607,256 @@ test_expect_success 'cat-file --batch=\"batman\" with --batch-all-objects will wor\n> git -C all-two cat-file --batch-all-objects --batch=\"batman\" >actual &&\n> cmp expect actual\n> '\n> +. \"$TEST_DIRECTORY\"/lib-gpg.sh\n> +. \"$TEST_DIRECTORY\"/lib-terminal.sh\n> +\n> +test_expect_success 'cat-file --batch|--batch-check setup' '\n> + echo 1>blob1 &&\n> + printf \"a\\0b\\0\\c\" >blob2 &&\n> + git add blob1 blob2 &&\n> + git commit -m \"Commit Message\" &&\n> + git branch -M main &&\n> + git tag -a -m \"v0.0.0\" testtag &&\n> + git update-ref refs/myblobs/blob1 HEAD:blob1 &&\n> + git update-ref refs/myblobs/blob2 HEAD:blob2 &&\n> + git update-ref refs/mytrees/tree1 HEAD^{tree}\n> +'\n> +\n> +batch_test_atom() {\n> + if test \"$3\" = \"fail\"\n> + then\n> + test_expect_${4:-success} $PREREQ \"basic atom: $1 $2 must fail\" \"\n> + test_must_fail git cat-file --batch-check='$2' >bad <<-EOF\n> + $1\n> + EOF\n> + \"\n> + else\n> + test_expect_${4:-success} $PREREQ \"basic atom: $1 $2\" \"\n> + git for-each-ref --format='$2' $1 >expected &&\n> + git cat-file --batch-check='$2' >actual <<-EOF &&\n> + $1\n> + EOF\n> + sanitize_pgp actual.clean &&\n> + cmp expected actual.clean\n> + \"\n> + fi\n> +}\n\nI wonder if the above function and some of the tests below could be\nintroduced in a preparatory patch before this one. It could help check\nthat reusing ref-filter doesn't change the behavior with some atoms\nthat were previously supported or rejected. Of course if some atoms\nare now failing or are now supported, then it's ok to add new tests\nfor these atoms in this patch.\n\n> +batch_test_atom refs/heads/main '%(refname)' fail\n> +batch_test_atom refs/heads/main '%(refname:)' fail\n\n[...]\n\n> +batch_test_atom refs/heads/main 'VALID'\n> +batch_test_atom refs/heads/main '%(INVALID)' fail\n> +batch_test_atom refs/heads/main '%(authordate:INVALID)' fail\n> +\n> +test_expect_success '%(rest) works with both a branch and a tag' '\n> + cat >expected <<-EOF &&\n> + 123 commit 123\n> + 456 tag 456\n> + EOF\n> + git cat-file --batch-check=\"%(rest) %(objecttype) %(rest)\" >actual <<-EOF &&\n> + refs/heads/main 123\n> + refs/tags/testtag 456\n> + EOF\n> + test_cmp expected actual\n> +'\n\nIt's a bit strange that this test is added in this patch while the\ncommit message doesn't talk about %(rest). So I wonder if this new\ntest could move to another previous commit.\n\n> +batch_test_atom refs/heads/main '%(objectname) %(objecttype) %(objectsize)\n> +%(raw)'\n> +batch_test_atom refs/tags/testtag '%(objectname) %(objecttype) %(objectsize)\n> +%(raw)'\n> +batch_test_atom refs/myblobs/blob1 '%(objectname) %(objecttype) %(objectsize)\n> +%(raw)'\n> +batch_test_atom refs/myblobs/blob2 '%(objectname) %(objecttype) %(objectsize)\n> +%(raw)'\n> +\n> +\n\nIt looks like there are two empty lines instead of one above.\n\n> +test_expect_success 'cat-file --batch equals to --batch-check with atoms' '\n> + git cat-file --batch-check=\"%(objectname) %(objecttype) %(objectsize)\n> +%(raw)\" >expected <<-EOF &&\n> + refs/heads/main\n> + refs/tags/testtag\n> + EOF\n> + git cat-file --batch >actual <<-EOF &&\n> + refs/heads/main\n> + refs/tags/testtag\n> + EOF\n> + cmp expected actual\n> +'\n\nI also wonder if the above new test belong to this commit or if it\ncould be moved to a previous commit.\n"},{"id":"429747","messageId":"CAP8UFD1NfCXDVwaVWKvTWofFHbLee=JRzr4sQBecMMaqi0eWBw@mail.gmail.com","threadId":"56095","inReplyTo":"CAP8UFD0vHw8cK90RbOUcrAcxNKiHVTMX0VEiK8+MNyHH0CNm4A@mail.gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-12T13:26:11Z","receivedAt":"2021-07-12T13:26:31Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Jul 12, 2021 at 3:17 PM Christian Couder\n wrote:\n>\n> On Mon, Jul 12, 2021 at 1:47 PM ZheNing Hu via GitGitGadget\n> wrote:\n\n> > +batch_test_atom() {\n> > + if test \"$3\" = \"fail\"\n> > + then\n> > + test_expect_${4:-success} $PREREQ \"basic atom: $1 $2 must fail\" \"\n> > + test_must_fail git cat-file --batch-check='$2' >bad <<-EOF\n> > + $1\n> > + EOF\n> > + \"\n> > + else\n> > + test_expect_${4:-success} $PREREQ \"basic atom: $1 $2\" \"\n> > + git for-each-ref --format='$2' $1 >expected &&\n> > + git cat-file --batch-check='$2' >actual <<-EOF &&\n> > + $1\n> > + EOF\n> > + sanitize_pgp actual.clean &&\n> > + cmp expected actual.clean\n> > + \"\n> > + fi\n> > +}\n>\n> I wonder if the above function and some of the tests below could be\n> introduced in a preparatory patch before this one. It could help check\n> that reusing ref-filter doesn't change the behavior with some atoms\n> that were previously supported or rejected. Of course if some atoms\n> are now failing or are now supported, then it's ok to add new tests\n> for these atoms in this patch.\n\nFor example maybe some of the tests could be introduced earlier when\nthe reject_atom() function is introduced.\n"},{"id":"429748","messageId":"CAOLTT8TEq3Zrms7MTj9onTsqHgqK4NypkzPpRv2Zaze5SMbBsg@mail.gmail.com","threadId":"56095","inReplyTo":"b6f661d6-d18f-9b26-da64-acad31b07188@iee.email","subject":"Re: [PATCH 00/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-12T13:27:12Z","receivedAt":"2021-07-12T13:27:14Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Philip Oakley 于2021年7月12日周一 下午9:02写道：\n>\n> On 12/07/2021 12:46, ZheNing Hu via GitGitGadget wrote:\n> > This patch series make cat-file reuse ref-filter logic.\n> >\n> > Change from last version:\n> minor nit..\n> Not sure if this is a gitgitgadget feature, but would it be possible\n> that a version indication be included in future versions of the patch,\n> e.g. [PATCH vN 00/19] [GSOC] ?\n> --\n\nI think ggg have this [1] , but it is not for the user to control.\n\n[1] https://github.com/gitgitgadget/gitgitgadget/blob/1df3c008552abd1fb788d4988e6d3b92f5765369/lib/patch-series.ts#L654\n\n> Philip\n\nThanks.\n--\nZheNing Hu\n"},{"id":"429750","messageId":"CAOLTT8SKp2Zfd+R31+vJbW_QJG5vJArB2yPn-O_34y1ezaDBcw@mail.gmail.com","threadId":"56095","inReplyTo":"CAP8UFD0vHw8cK90RbOUcrAcxNKiHVTMX0VEiK8+MNyHH0CNm4A@mail.gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-12T13:49:41Z","receivedAt":"2021-07-12T13:49:36Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年7月12日周一 下午9:17写道：\n>\n> On Mon, Jul 12, 2021 at 1:47 PM ZheNing Hu via GitGitGadget\n> wrote:\n> >\n> > From: ZheNing Hu \n> >\n> > In order to let cat-file use ref-filter logic, let's do the\n> > following:\n> >\n> > 1. Change the type of member `format` in struct `batch_options`\n> > to `ref_format`, we will pass it to ref-filter later.\n> > 2. Let `batch_objects()` add atoms to format, and use\n> > `verify_ref_format()` to check atoms.\n> > 3. Use `format_ref_array_item()` in `batch_object_write()` to\n> > get the formatted data corresponding to the object. If the\n> > return value of `format_ref_array_item()` is equals to zero,\n> > use `batch_write()` to print object data; else if the return\n> > value is less than zero, use `die()` to print the error message\n> > and exit; else if return value is greater than zero, only print\n> > the error message, but don't exit.\n> > 4. Use free_ref_array_item_value() to free ref_array_item's\n> > value.\n> >\n> > Most of the atoms in `for-each-ref --format` are now supported,\n> > such as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`,\n> > `%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected:\n> > `%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`,\n> > `%(flag)`, `%(HEAD)`, because these atoms are unique to those objects\n> > that pointed to by a ref, \"for-each-ref\"'s family can naturally use\n> > these atoms, but not all objects are pointed to be a ref, so \"cat-file\"\n> > will not be able to use them.\n> >\n> > The performance for `git cat-file --batch-all-objects\n> > --batch-check` on the Git repository itself with performance\n> > testing tool `hyperfine` changes from 669.4 ms ± 31.1 ms to\n> > 1.134 s ± 0.063 s.\n> >\n> > The performance for `git cat-file --batch-all-objects --batch\n> > >/dev/null` on the Git repository itself with performance testing\n> > tool `time` change from \"27.37s user 0.29s system 98% cpu 28.089\n> > total\" to \"33.69s user 1.54s system 87% cpu 40.258 total\".\n>\n> Saying that a later patch will add a fast path which will mitigate the\n> performance regression introduced by this patch might help reassure\n> reviewers.\n>\n\nOK.\n\n> By the way it is not clear if adding the fast path fully mitigates\n> this performance regression or not. You might want to discuss that in\n> the cover letter, or maybe in the patch adding the fast path.\n>\n\nI mentioned it: \"By using this fast path, we can reduce some of the\nextra overhead\nwhen cat-file --batch using ref-filter. The running time of\ngit cat-file --batch-check will be similar to before, and the\nrunning time of git cat-file --batch will be 9.1% less than before.\"\nwhich is using the result of t/perf/p1006-cat-file.sh.\n\n>\n> I wonder if the above function and some of the tests below could be\n> introduced in a preparatory patch before this one. It could help check\n> that reusing ref-filter doesn't change the behavior with some atoms\n> that were previously supported or rejected. Of course if some atoms\n> are now failing or are now supported, then it's ok to add new tests\n> for these atoms in this patch.\n>\n\nYes, it might be worth splitting into two commits.\n\n> > +batch_test_atom refs/heads/main '%(refname)' fail\n> > +batch_test_atom refs/heads/main '%(refname:)' fail\n>\n> [...]\n>\n> > +batch_test_atom refs/heads/main 'VALID'\n> > +batch_test_atom refs/heads/main '%(INVALID)' fail\n> > +batch_test_atom refs/heads/main '%(authordate:INVALID)' fail\n> > +\n> > +test_expect_success '%(rest) works with both a branch and a tag' '\n> > + cat >expected <<-EOF &&\n> > + 123 commit 123\n> > + 456 tag 456\n> > + EOF\n> > + git cat-file --batch-check=\"%(rest) %(objecttype) %(rest)\" >actual <<-EOF &&\n> > + refs/heads/main 123\n> > + refs/tags/testtag 456\n> > + EOF\n> > + test_cmp expected actual\n> > +'\n>\n> It's a bit strange that this test is added in this patch while the\n> commit message doesn't talk about %(rest). So I wonder if this new\n> test could move to another previous commit.\n>\n\nIt's just used for checking the uncommonly atoms \"%(rest)\".\nBut as you said, we can move it to a split commit.\n\n> > +batch_test_atom refs/heads/main '%(objectname) %(objecttype) %(objectsize)\n> > +%(raw)'\n> > +batch_test_atom refs/tags/testtag '%(objectname) %(objecttype) %(objectsize)\n> > +%(raw)'\n> > +batch_test_atom refs/myblobs/blob1 '%(objectname) %(objecttype) %(objectsize)\n> > +%(raw)'\n> > +batch_test_atom refs/myblobs/blob2 '%(objectname) %(objecttype) %(objectsize)\n> > +%(raw)'\n> > +\n> > +\n>\n> It looks like there are two empty lines instead of one above.\n>\n> > +test_expect_success 'cat-file --batch equals to --batch-check with atoms' '\n> > + git cat-file --batch-check=\"%(objectname) %(objecttype) %(objectsize)\n> > +%(raw)\" >expected <<-EOF &&\n> > + refs/heads/main\n> > + refs/tags/testtag\n> > + EOF\n> > + git cat-file --batch >actual <<-EOF &&\n> > + refs/heads/main\n> > + refs/tags/testtag\n> > + EOF\n> > + cmp expected actual\n> > +'\n>\n> I also wonder if the above new test belong to this commit or if it\n> could be moved to a previous commit.\n\nIt's same.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"429751","messageId":"CAOLTT8QzBQOJahy2_MX_ShwErfxgwfGByfsjdZBy2C=qStCswA@mail.gmail.com","threadId":"56095","inReplyTo":"CAP8UFD1NfCXDVwaVWKvTWofFHbLee=JRzr4sQBecMMaqi0eWBw@mail.gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-12T13:51:47Z","receivedAt":"2021-07-12T13:51:39Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年7月12日周一 下午9:26写道：\n>\n> >\n> > I wonder if the above function and some of the tests below could be\n> > introduced in a preparatory patch before this one. It could help check\n> > that reusing ref-filter doesn't change the behavior with some atoms\n> > that were previously supported or rejected. Of course if some atoms\n> > are now failing or are now supported, then it's ok to add new tests\n> > for these atoms in this patch.\n>\n> For example maybe some of the tests could be introduced earlier when\n> the reject_atom() function is introduced.\n\nYeah, move it to \"[GSOC] ref-filter: add %(rest) atom\" is better.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"429831","messageId":"xmqqtukz2rp4.fsf@gitster.g","threadId":"56095","inReplyTo":"CAP8UFD0vHw8cK90RbOUcrAcxNKiHVTMX0VEiK8+MNyHH0CNm4A@mail.gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-12T20:38:31Z","receivedAt":"2021-07-12T20:38:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> The performance for `git cat-file --batch-all-objects --batch\n>> >/dev/null` on the Git repository itself with performance testing\n>> tool `time` change from \"27.37s user 0.29s system 98% cpu 28.089\n>> total\" to \"33.69s user 1.54s system 87% cpu 40.258 total\".\n>\n> Saying that a later patch will add a fast path which will mitigate the\n> performance regression introduced by this patch might help reassure\n> reviewers.\n\nMore importantly, why is such a fast-path even needed? Isn't it a\nsign that the ref-filter implementation is eating more cycles than\nit should for given set of placeholders? Do we know where the extra\ncycles goes?\n\nI find it somewhat alarming if we are talking about \"fast-path\"\nworkaround before understanding why we are seeing slowdown in the\nfirst place.\n"},{"id":"430068","messageId":"CAOLTT8RR4+tUuT2yc2PDL9NwCburW8bM_Sht6nhKJ_fYV8fGsQ@mail.gmail.com","threadId":"56095","inReplyTo":"xmqqtukz2rp4.fsf@gitster.g","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-14T16:24:14Z","receivedAt":"2021-07-14T16:24:02Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年7月13日周二 上午4:38写道：\n\n>\n> Christian Couder writes:\n>\n> More importantly, why is such a fast-path even needed? Isn't it a\n> sign that the ref-filter implementation is eating more cycles than\n> it should for given set of placeholders? Do we know where the extra\n> cycles goes?\n>\n> I find it somewhat alarming if we are talking about \"fast-path\"\n> workaround before understanding why we are seeing slowdown in the\n> first place.\n\nThere is no complete conclusion yet, but I try to use time and hyperfine test\nfor these commits (t/perf/* is not accurate enough):\n\n----------------------------------------------------------------------------------------------------------------------------\n| subject |\n--batch-check (using hyperfine) | --batch(using time) |\n----------------------------------------------------------------------------------------------------------------------------\n|[GSOC] cat-file: use fast path when using default_format |\n 700ms | 25.450s |\n----------------------------------------------------------------------------------------------------------------------------\n|[GSOC] cat-file: re-implement --textconv, --filters options |\n 790ms | 29.933s |\n----------------------------------------------------------------------------------------------------------------------------\n|[GSOC] cat-file: reuse err buf in batch_object_write() |\n 770ms | 29.153s |\n----------------------------------------------------------------------------------------------------------------------------\n|[GSOC] cat-file: reuse ref-filter logic |\n 780ms | 29.412s |\n----------------------------------------------------------------------------------------------------------------------------\n|The third batch (upstream/master) |\n 640ms | 26.025s |\n----------------------------------------------------------------------------------------------------------------------------\n\nI think we their cost is indeed from \"[GSOC] cat-file: reuse ref-filter logic\".\nBut what causes the loss of performance needs further analysis.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"430167","messageId":"CAOLTT8Sa984Eo18QMBeGnMCX3_7sr+9qUYoAR4FS3UF6+CDtGw@mail.gmail.com","threadId":"56095","inReplyTo":"CAOLTT8RR4+tUuT2yc2PDL9NwCburW8bM_Sht6nhKJ_fYV8fGsQ@mail.gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-15T01:53:23Z","receivedAt":"2021-07-15T01:53:13Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"ZheNing Hu 于2021年7月15日周四 上午12:24写道：\n>\n> Junio C Hamano 于2021年7月13日周二 上午4:38写道：\n>\n> >\n> > Christian Couder writes:\n> >\n> > More importantly, why is such a fast-path even needed? Isn't it a\n> > sign that the ref-filter implementation is eating more cycles than\n> > it should for given set of placeholders? Do we know where the extra\n> > cycles goes?\n> >\n> > I find it somewhat alarming if we are talking about \"fast-path\"\n> > workaround before understanding why we are seeing slowdown in the\n> > first place.\n>\n> There is no complete conclusion yet, but I try to use time and hyperfine test\n> for these commits (t/perf/* is not accurate enough):\n>\n> ----------------------------------------------------------------------------------------------------------------------------\n> | subject |\n> --batch-check (using hyperfine) | --batch(using time) |\n> ----------------------------------------------------------------------------------------------------------------------------\n> |[GSOC] cat-file: use fast path when using default_format |\n> 700ms | 25.450s |\n> ----------------------------------------------------------------------------------------------------------------------------\n> |[GSOC] cat-file: re-implement --textconv, --filters options |\n> 790ms | 29.933s |\n> ----------------------------------------------------------------------------------------------------------------------------\n> |[GSOC] cat-file: reuse err buf in batch_object_write() |\n> 770ms | 29.153s |\n> ----------------------------------------------------------------------------------------------------------------------------\n> |[GSOC] cat-file: reuse ref-filter logic |\n> 780ms | 29.412s |\n> ----------------------------------------------------------------------------------------------------------------------------\n> |The third batch (upstream/master) |\n> 640ms | 26.025s |\n> ----------------------------------------------------------------------------------------------------------------------------\n>\n> I think we their cost is indeed from \"[GSOC] cat-file: reuse ref-filter logic\".\n> But what causes the loss of performance needs further analysis.\n>\n\nNow I think:\nThere are three main reasons why the performance of cat-file --batch\ndeteriorates after refactor.\n\n1. Too many copies are used in ref-filter and we cannot avoid these copies\neasily because ref-filter needs these copied data to implement atoms %(if),\n%(else), %(end)... and the --sort option. The original cat-file\n--batch only needs\nto output the data to the final string. Its copy times are relatively small.\n\n2. More complex data structure and parsing process are used in ref-filter.\nThis is why it can provide more and more useful atoms. Therefore, I think the\nperformance degradation that occurs here is normal.\n\n3. As Ævar Arnfjörð Bjarmason mentioned, oid_object_info_extend() was used\ntwice in get_object() before. oid_object_info_extend() is the hot\npath, we should\ntry to avoid calling it, So in last version of \"[GSOC] cat-file:\nre-implement --textconv,\n--filters options\", I make the unified processing of --textconv and\n--filter avoid calling\noid_object_info_extend() twice.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"430192","messageId":"CAP8UFD24X7UjXGKsRWr+f_xmX0x4EVDJHLBs2c1KhECb8-BnBw@mail.gmail.com","threadId":"56095","inReplyTo":"CAOLTT8Sa984Eo18QMBeGnMCX3_7sr+9qUYoAR4FS3UF6+CDtGw@mail.gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-15T09:45:42Z","receivedAt":"2021-07-15T09:45:57Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Jul 15, 2021 at 3:53 AM ZheNing Hu wrote:\n>\n> ZheNing Hu 于2021年7月15日周四 上午12:24写道：\n> >\n> > Junio C Hamano 于2021年7月13日周二 上午4:38写道：\n\n> > > I find it somewhat alarming if we are talking about \"fast-path\"\n> > > workaround before understanding why we are seeing slowdown in the\n> > > first place.\n> >\n> > There is no complete conclusion yet, but I try to use time and hyperfine test\n> > for these commits (t/perf/* is not accurate enough):\n> >\n> > ----------------------------------------------------------------------------------------------------------------------------\n> > | subject |\n> > --batch-check (using hyperfine) | --batch(using time) |\n> > ----------------------------------------------------------------------------------------------------------------------------\n> > |[GSOC] cat-file: use fast path when using default_format |\n> > 700ms | 25.450s |\n> > ----------------------------------------------------------------------------------------------------------------------------\n> > |[GSOC] cat-file: re-implement --textconv, --filters options |\n> > 790ms | 29.933s |\n> > ----------------------------------------------------------------------------------------------------------------------------\n> > |[GSOC] cat-file: reuse err buf in batch_object_write() |\n> > 770ms | 29.153s |\n> > ----------------------------------------------------------------------------------------------------------------------------\n> > |[GSOC] cat-file: reuse ref-filter logic |\n> > 780ms | 29.412s |\n> > ----------------------------------------------------------------------------------------------------------------------------\n> > |The third batch (upstream/master) |\n> > 640ms | 26.025s |\n> > ----------------------------------------------------------------------------------------------------------------------------\n> >\n> > I think we their cost is indeed from \"[GSOC] cat-file: reuse ref-filter logic\".\n> > But what causes the loss of performance needs further analysis.\n>\n> Now I think:\n> There are three main reasons why the performance of cat-file --batch\n> deteriorates after refactor.\n>\n> 1. Too many copies are used in ref-filter and we cannot avoid these copies\n> easily because ref-filter needs these copied data to implement atoms %(if),\n> %(else), %(end)... and the --sort option. The original cat-file\n> --batch only needs\n> to output the data to the final string. Its copy times are relatively small.\n\nIs it possible to check early if any of the atoms that needs these\ncopied data is specified, and if none of them is specified then to\navoid the copies?\n\n> 2. More complex data structure and parsing process are used in ref-filter.\n> This is why it can provide more and more useful atoms. Therefore, I think the\n> performance degradation that occurs here is normal.\n\nAre there way the more complex parsing could be avoided if it's not\nneeded by the atoms that are actually used?\n\n> 3. As Ævar Arnfjörð Bjarmason mentioned, oid_object_info_extend() was used\n> twice in get_object() before. oid_object_info_extend() is the hot\n> path, we should\n> try to avoid calling it, So in last version of \"[GSOC] cat-file:\n> re-implement --textconv,\n> --filters options\", I make the unified processing of --textconv and\n> --filter avoid calling\n> oid_object_info_extend() twice.\n\nOk, thanks for the details and your work on this performance issue!\n\nI wonder if your patch series could be split, so that the early parts\nthat add new atoms to ref-filter could be merged sooner?\n\nBest,\nChristian.\n"},{"id":"430200","messageId":"CAOLTT8Qj_zAhPEzKxJ2dGg8O3R_b8Sn05G29VE5WJhymR4EQSg@mail.gmail.com","threadId":"56095","inReplyTo":"CAP8UFD24X7UjXGKsRWr+f_xmX0x4EVDJHLBs2c1KhECb8-BnBw@mail.gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-15T13:53:04Z","receivedAt":"2021-07-15T13:52:51Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年7月15日周四 下午5:45写道：\n>\n> On Thu, Jul 15, 2021 at 3:53 AM ZheNing Hu wrote:\n> >\n> > ZheNing Hu 于2021年7月15日周四 上午12:24写道：\n> > >\n> > > Junio C Hamano 于2021年7月13日周二 上午4:38写道：\n>\n> > > > I find it somewhat alarming if we are talking about \"fast-path\"\n> > > > workaround before understanding why we are seeing slowdown in the\n> > > > first place.\n> > >\n> > > There is no complete conclusion yet, but I try to use time and hyperfine test\n> > > for these commits (t/perf/* is not accurate enough):\n> > >\n> > > ----------------------------------------------------------------------------------------------------------------------------\n> > > | subject |\n> > > --batch-check (using hyperfine) | --batch(using time) |\n> > > ----------------------------------------------------------------------------------------------------------------------------\n> > > |[GSOC] cat-file: use fast path when using default_format |\n> > > 700ms | 25.450s |\n> > > ----------------------------------------------------------------------------------------------------------------------------\n> > > |[GSOC] cat-file: re-implement --textconv, --filters options |\n> > > 790ms | 29.933s |\n> > > ----------------------------------------------------------------------------------------------------------------------------\n> > > |[GSOC] cat-file: reuse err buf in batch_object_write() |\n> > > 770ms | 29.153s |\n> > > ----------------------------------------------------------------------------------------------------------------------------\n> > > |[GSOC] cat-file: reuse ref-filter logic |\n> > > 780ms | 29.412s |\n> > > ----------------------------------------------------------------------------------------------------------------------------\n> > > |The third batch (upstream/master) |\n> > > 640ms | 26.025s |\n> > > ----------------------------------------------------------------------------------------------------------------------------\n> > >\n> > > I think we their cost is indeed from \"[GSOC] cat-file: reuse ref-filter logic\".\n> > > But what causes the loss of performance needs further analysis.\n> >\n> > Now I think:\n> > There are three main reasons why the performance of cat-file --batch\n> > deteriorates after refactor.\n> >\n> > 1. Too many copies are used in ref-filter and we cannot avoid these copies\n> > easily because ref-filter needs these copied data to implement atoms %(if),\n> > %(else), %(end)... and the --sort option. The original cat-file\n> > --batch only needs\n> > to output the data to the final string. Its copy times are relatively small.\n>\n> Is it possible to check early if any of the atoms that needs these\n> copied data is specified, and if none of them is specified then to\n> avoid the copies?\n>\n\nWell, The copy I'm talking about here refers to something like \"v->s =\nxstrdup(xxx)\";\nbut v->s is need by --sort, so it is very difficult to remove. At the\nmoment I think the\nonly solution is the fast path mentioned by Ævar Arnfjörð Bjarmason.\n\n> > 2. More complex data structure and parsing process are used in ref-filter.\n> > This is why it can provide more and more useful atoms. Therefore, I think the\n> > performance degradation that occurs here is normal.\n>\n> Are there way the more complex parsing could be avoided if it's not\n> needed by the atoms that are actually used?\n\nNo. For example, we can only support \"objectsize\" before and now we can\nsupport \"objectsize:short\", so we need to pay more parsing process here.\n(It's necessary)\n\n>\n> > 3. As Ævar Arnfjörð Bjarmason mentioned, oid_object_info_extend() was used\n> > twice in get_object() before. oid_object_info_extend() is the hot\n> > path, we should\n> > try to avoid calling it, So in last version of \"[GSOC] cat-file:\n> > re-implement --textconv,\n> > --filters options\", I make the unified processing of --textconv and\n> > --filter avoid calling\n> > oid_object_info_extend() twice.\n>\n> Ok, thanks for the details and your work on this performance issue!\n>\n> I wonder if your patch series could be split, so that the early parts\n> that add new atoms to ref-filter could be merged sooner?\n>\n\nShould this part of the work be handed over to Junio?\nThe implementation of %(rest) and %(raw) may be worth merging,\nthey are truly \"zh/ref-filter-raw-data\".\nThe other part may be called \"cat-file-reuse-ref-filter-logic\".\n\n> Best,\n> Christian.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"430206","messageId":"CAOLTT8SLMptcGf1mKtumVTAXdD5_qygeohtXvbCUuj39MOJ0yA@mail.gmail.com","threadId":"56095","inReplyTo":"CAP8UFD0vHw8cK90RbOUcrAcxNKiHVTMX0VEiK8+MNyHH0CNm4A@mail.gmail.com","subject":"Re: [PATCH 14/19] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-15T14:55:01Z","receivedAt":"2021-07-15T14:54:50Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年7月12日周一 下午9:17写道：\n>\n> > +test_expect_success 'cat-file --batch equals to --batch-check with atoms' '\n> > + git cat-file --batch-check=\"%(objectname) %(objecttype) %(objectsize)\n> > +%(raw)\" >expected <<-EOF &&\n> > + refs/heads/main\n> > + refs/tags/testtag\n> > + EOF\n> > + git cat-file --batch >actual <<-EOF &&\n> > + refs/heads/main\n> > + refs/tags/testtag\n> > + EOF\n> > + cmp expected actual\n> > +'\n>\n> I also wonder if the above new test belong to this commit or if it\n> could be moved to a previous commit.\n\nIt's relate to %(raw), but cat-file don't realize %(raw) before. So the location\nof this test should remain unchanged.\n\n--\nZheNing Hu\n"},{"id":"430209","messageId":"45c0cbe44d56b7f6dbe08db5fc1d06da529f6bd5.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 01/17] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:09Z","receivedAt":"2021-07-15T15:40:31Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nOnly tag and commit objects use `grab_sub_body_contents()` to grab\nobject contents in the current codebase. We want to teach the\nfunction to also handle blobs and trees to get their raw data,\nwithout parsing a blob (whose contents looks like a commit or a tag)\nincorrectly as a commit or a tag.\n\nSkip the block of code that is specific to handling commits and tags\nearly when the given object is of a wrong type to help later\naddition to handle other types of objects in this function.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Junio C Hamano \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4db0e40ff4c..5cee6512fba 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n+static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n {\n \tint i;\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n+\tvoid *buf = data->content;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n@@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (strcmp(name, \"body\") &&\n-\t\t !starts_with(name, \"subject\") &&\n-\t\t !starts_with(name, \"trailers\") &&\n-\t\t !starts_with(name, \"contents\"))\n+\n+\t\tif ((data->type != OBJ_TAG &&\n+\t\t data->type != OBJ_COMMIT) ||\n+\t\t (strcmp(name, \"body\") &&\n+\t\t !starts_with(name, \"subject\") &&\n+\t\t !starts_with(name, \"trailers\") &&\n+\t\t !starts_with(name, \"contents\")))\n \t\t\tcontinue;\n \t\tif (!subpos)\n \t\t\tfind_subpos(buf,\n@@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n+static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n {\n+\tvoid *buf = data->content;\n+\n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n@@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi->content);\n+\t\tgrab_values(ref->value, deref, *obj, oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n-- \ngitgitgadget\n\n"},{"id":"430210","messageId":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.git.1626090419.gitgitgadget@gmail.com","subject":"[PATCH v2 00/17] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:08Z","receivedAt":"2021-07-15T15:40:32Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"This patch series makes cat-file reuse ref-filter logic.\n\nChange from last version:\n\n 1. Move some atoms' test to the commit: [GSOC] ref-filter: add\n cat_file_mode to ref_format.\n 2. Advance the commit of performance tests.\n 3. Modified some commit messages related to cat-file performance.\n\nBy the way, \"[GSOC] ref-filter: add %(rest) atom\" and its previous commits\nshould belong to zh/ref-filter-raw-data and the rest should belong to\nzh/cat-file-batch-refactor.\n\nZheNing Hu (17):\n [GSOC] ref-filter: add obj-type check in grab contents\n [GSOC] ref-filter: add %(raw) atom\n [GSOC] ref-filter: --format=%(raw) re-support --perl\n [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n [GSOC] ref-filter: add %(rest) atom\n [GSOC] ref-filter: pass get_object() return value to their callers\n [GSOC] ref-filter: introduce free_ref_array_item_value() function\n [GSOC] ref-filter: add cat_file_mode to ref_format\n [GSOC] ref-filter: modify the error message and value in get_object\n [GSOC] cat-file: add has_object_file() check\n [GSOC] cat-file: change batch_objects parameter name\n [GSOC] cat-file: create p1006-cat-file.sh\n [GSOC] cat-file: reuse ref-filter logic\n [GSOC] cat-file: reuse err buf in batch_object_write()\n [GSOC] cat-file: re-implement --textconv, --filters options\n [GSOC] ref-filter: remove grab_oid() function\n [GSOC] cat-file: use fast path when using default_format\n\n Documentation/git-cat-file.txt | 6 +\n Documentation/git-for-each-ref.txt | 9 +\n builtin/cat-file.c | 306 +++++++++----------------\n builtin/tag.c | 2 +-\n quote.c | 17 ++\n quote.h | 1 +\n ref-filter.c | 346 +++++++++++++++++++++--------\n ref-filter.h | 13 +-\n t/perf/p1006-cat-file.sh | 28 +++\n t/t1006-cat-file.sh | 239 ++++++++++++++++++++\n t/t3203-branch-output.sh | 4 +\n t/t6300-for-each-ref.sh | 235 ++++++++++++++++++++\n t/t6301-for-each-ref-errors.sh | 2 +-\n t/t7004-tag.sh | 4 +\n t/t7030-verify-tag.sh | 4 +\n 15 files changed, 921 insertions(+), 295 deletions(-)\n create mode 100755 t/perf/p1006-cat-file.sh\n\n\nbase-commit: 75ae10bc75336db031ee58d13c5037b929235912\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-993%2Fadlternative%2Fcat-file-batch-refactor-2-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-993/adlternative/cat-file-batch-refactor-2-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/993\n\nRange-diff vs v1:\n\n 1: 9aef8882bd1 < -: ----------- cat-file: handle trivial --batch format with --batch-all-objects\n 2: 1332006006f < -: ----------- cat-file: merge two block into one\n 3: 0dacc60bbcc = 1: 45c0cbe44d5 [GSOC] ref-filter: add obj-type check in grab contents\n 4: 5bd715ae9f0 = 2: 554d7653ee7 [GSOC] ref-filter: add %(raw) atom\n 5: df2cd9e8e61 = 3: 94addd4676a [GSOC] ref-filter: --format=%(raw) re-support --perl\n 6: 2d34ad3bc77 = 4: 45984f94bf3 [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n 7: 460c807c6ab ! 5: 6fb9cfdeab1 [GSOC] ref-filter: add %(rest) atom\n @@ Commit message\n [GSOC] ref-filter: add %(rest) atom\n \n In order to let \"cat-file --batch=%(rest)\" use the ref-filter\n - interface, add %(rest) atom for ref-filter. \"git for-each-ref\",\n - \"git branch\", \"git tag\" and \"git verify-tag\" will reject %(rest)\n - by default.\n + interface, add %(rest) atom for ref-filter. Introduce the\n + reject_atom() to reject the atom %(rest) for \"git for-each-ref\",\n + \"git branch\", \"git tag\" and \"git verify-tag\".\n \n Mentored-by: Christian Couder \n Mentored-by: Hariom Verma \n @@ ref-filter.c: static struct {\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n +@@ ref-filter.c: static const char *find_next(const char *cp)\n + \treturn NULL;\n + }\n + \n ++static int reject_atom(enum atom_type atom_type)\n ++{\n ++\treturn atom_type == ATOM_REST;\n ++}\n ++\n + /*\n + * Make sure the format string is well formed, and parse out\n + * the used atoms.\n @@ ref-filter.c: int verify_ref_format(struct ref_format *format)\n + \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n ++\t\tif (reject_atom(used_atom[at].atom_type))\n ++\t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n -+\t\tif (used_atom[at].atom_type == ATOM_REST)\n -+\t\t\tdie(\"this command reject atom %%(%.*s)\", (int)(ep - sp - 2), sp + 2);\n -+\n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n \t\t format->quote_style == QUOTE_SHELL ||\n - \t\t format->quote_style == QUOTE_TCL) &&\n @@ ref-filter.c: static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n 8: e1aca51d500 = 6: c3378dbfaed [GSOC] ref-filter: pass get_object() return value to their callers\n 9: 6ad42c96405 = 7: e5cf5541024 [GSOC] ref-filter: introduce free_ref_array_item_value() function\n 10: b61d538d53d ! 8: bf052cc5d3f [GSOC] ref-filter: introduce reject_atom()\n @@ Metadata\n Author: ZheNing Hu \n \n ## Commit message ##\n - [GSOC] ref-filter: introduce reject_atom()\n + [GSOC] ref-filter: add cat_file_mode to ref_format\n \n - Add `cat_file_mode` member in struct `ref_format` and introduce\n - the function `reject_atom()`, when `cat-file --batch` use ref-filter\n - logic later, it can help us reject atoms in verify_ref_format()\n - which cat-file cannot use, e.g. `%(refname)`, `%(push)`,\n - `%(upstream)`... or the atom `%(rest)` which for-each-ref family\n - cannot use.\n + Add `cat_file_mode` member to struct `ref_format`. When\n + `cat-file --batch` use ref-filter logic later, it can help us\n + reject atoms in verify_ref_format() which cat-file cannot use,\n + e.g. `%(refname)`, `%(push)`, `%(upstream)\"...\n +\n + Add batch_test_atom() to t/t1006-cat-file.sh and add check\n + for cat-file --batch, this can help us clearly show which\n + atoms cat-file accepts and which atoms it rejects.\n \n Helped-by: Eric Sunshine \n Helped-by: Ævar Arnfjörð Bjarmason \n @@ ref-filter.c: static const char *find_next(const char *cp)\n \treturn NULL;\n }\n \n +-static int reject_atom(enum atom_type atom_type)\n +-{\n +-\treturn atom_type == ATOM_REST;\n +static int reject_atom(int cat_file_mode, enum atom_type atom_type)\n +{\n +\tif (!cat_file_mode)\n @@ ref-filter.c: static const char *find_next(const char *cp)\n +\tdefault:\n +\t\treturn 0;\n +\t}\n -+}\n -+\n + }\n + \n /*\n - * Make sure the format string is well formed, and parse out\n - * the used atoms.\n @@ ref-filter.c: int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n --\n --\t\tif (used_atom[at].atom_type == ATOM_REST)\n --\t\t\tdie(\"this command reject atom %%(%.*s)\", (int)(ep - sp - 2), sp + 2);\n +-\t\tif (reject_atom(used_atom[at].atom_type))\n +\t\tif (reject_atom(format->cat_file_mode, used_atom[at].atom_type))\n -+\t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n + \t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n - \t\t format->quote_style == QUOTE_SHELL ||\n \n ## ref-filter.h ##\n @@ ref-filter.h: struct ref_format {\n @@ ref-filter.h: struct ref_format {\n \tint quote_style;\n \tint use_rest;\n \tint use_color;\n +\n + ## t/t1006-cat-file.sh ##\n +@@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch=\"batman\" with --batch-all-objects will wor\n + \tcmp expect actual\n + '\n + \n ++. \"$TEST_DIRECTORY\"/lib-gpg.sh\n ++. \"$TEST_DIRECTORY\"/lib-terminal.sh\n ++\n ++test_expect_success 'cat-file --batch|--batch-check setup' '\n ++\techo 1>blob1 &&\n ++\tprintf \"a\\0b\\0\\c\" >blob2 &&\n ++\tgit add blob1 blob2 &&\n ++\tgit commit -m \"Commit Message\" &&\n ++\tgit branch -M main &&\n ++\tgit tag -a -m \"v0.0.0\" testtag &&\n ++\tgit update-ref refs/myblobs/blob1 HEAD:blob1 &&\n ++\tgit update-ref refs/myblobs/blob2 HEAD:blob2 &&\n ++\tgit update-ref refs/mytrees/tree1 HEAD^{tree}\n ++'\n ++\n ++batch_test_atom() {\n ++\tif test \"$3\" = \"fail\"\n ++\tthen\n ++\t\ttest_expect_${4:-success} $PREREQ \"basic atom: $1 $2 must fail\" \"\n ++\t\t\ttest_must_fail git cat-file --batch-check='$2' >bad <<-EOF\n ++\t\t\t$1\n ++\t\t\tEOF\n ++\t\t\"\n ++\telse\n ++\t\ttest_expect_${4:-success} $PREREQ \"basic atom: $1 $2\" \"\n ++\t\t\tgit for-each-ref --format='$2' $1 >expected &&\n ++\t\t\tgit cat-file --batch-check='$2' >actual <<-EOF &&\n ++\t\t\t$1\n ++\t\t\tEOF\n ++\t\t\tsanitize_pgp actual.clean &&\n ++\t\t\tcmp expected actual.clean\n ++\t\t\"\n ++\tfi\n ++}\n ++\n ++batch_test_atom refs/heads/main '%(refname)' fail\n ++batch_test_atom refs/heads/main '%(refname:)' fail\n ++batch_test_atom refs/heads/main '%(refname:short)' fail\n ++batch_test_atom refs/heads/main '%(refname:lstrip=1)' fail\n ++batch_test_atom refs/heads/main '%(refname:lstrip=2)' fail\n ++batch_test_atom refs/heads/main '%(refname:lstrip=-1)' fail\n ++batch_test_atom refs/heads/main '%(refname:lstrip=-2)' fail\n ++batch_test_atom refs/heads/main '%(refname:rstrip=1)' fail\n ++batch_test_atom refs/heads/main '%(refname:rstrip=2)' fail\n ++batch_test_atom refs/heads/main '%(refname:rstrip=-1)' fail\n ++batch_test_atom refs/heads/main '%(refname:rstrip=-2)' fail\n ++batch_test_atom refs/heads/main '%(refname:strip=1)' fail\n ++batch_test_atom refs/heads/main '%(refname:strip=2)' fail\n ++batch_test_atom refs/heads/main '%(refname:strip=-1)' fail\n ++batch_test_atom refs/heads/main '%(refname:strip=-2)' fail\n ++batch_test_atom refs/heads/main '%(upstream)' fail\n ++batch_test_atom refs/heads/main '%(upstream:short)' fail\n ++batch_test_atom refs/heads/main '%(upstream:lstrip=2)' fail\n ++batch_test_atom refs/heads/main '%(upstream:lstrip=-2)' fail\n ++batch_test_atom refs/heads/main '%(upstream:rstrip=2)' fail\n ++batch_test_atom refs/heads/main '%(upstream:rstrip=-2)' fail\n ++batch_test_atom refs/heads/main '%(upstream:strip=2)' fail\n ++batch_test_atom refs/heads/main '%(upstream:strip=-2)' fail\n ++batch_test_atom refs/heads/main '%(push)' fail\n ++batch_test_atom refs/heads/main '%(push:short)' fail\n ++batch_test_atom refs/heads/main '%(push:lstrip=1)' fail\n ++batch_test_atom refs/heads/main '%(push:lstrip=-1)' fail\n ++batch_test_atom refs/heads/main '%(push:rstrip=1)' fail\n ++batch_test_atom refs/heads/main '%(push:rstrip=-1)' fail\n ++batch_test_atom refs/heads/main '%(push:strip=1)' fail\n ++batch_test_atom refs/heads/main '%(push:strip=-1)' fail\n ++batch_test_atom refs/heads/main '%(objecttype)'\n ++batch_test_atom refs/heads/main '%(objectsize)'\n ++batch_test_atom refs/heads/main '%(objectsize:disk)'\n ++batch_test_atom refs/heads/main '%(deltabase)'\n ++batch_test_atom refs/heads/main '%(objectname)'\n ++batch_test_atom refs/heads/main '%(objectname:short)' fail\n ++batch_test_atom refs/heads/main '%(objectname:short=1)' fail\n ++batch_test_atom refs/heads/main '%(objectname:short=10)' fail\n ++batch_test_atom refs/heads/main '%(tree)' fail\n ++batch_test_atom refs/heads/main '%(tree:short)' fail\n ++batch_test_atom refs/heads/main '%(tree:short=1)' fail\n ++batch_test_atom refs/heads/main '%(tree:short=10)' fail\n ++batch_test_atom refs/heads/main '%(parent)' fail\n ++batch_test_atom refs/heads/main '%(parent:short)' fail\n ++batch_test_atom refs/heads/main '%(parent:short=1)' fail\n ++batch_test_atom refs/heads/main '%(parent:short=10)' fail\n ++batch_test_atom refs/heads/main '%(numparent)' fail\n ++batch_test_atom refs/heads/main '%(object)' fail\n ++batch_test_atom refs/heads/main '%(type)' fail\n ++batch_test_atom refs/heads/main '%(raw)' fail\n ++batch_test_atom refs/heads/main '%(*objectname)' fail\n ++batch_test_atom refs/heads/main '%(*objecttype)' fail\n ++batch_test_atom refs/heads/main '%(author)' fail\n ++batch_test_atom refs/heads/main '%(authorname)' fail\n ++batch_test_atom refs/heads/main '%(authoremail)' fail\n ++batch_test_atom refs/heads/main '%(authoremail:trim)' fail\n ++batch_test_atom refs/heads/main '%(authoremail:localpart)' fail\n ++batch_test_atom refs/heads/main '%(authordate)' fail\n ++batch_test_atom refs/heads/main '%(committer)' fail\n ++batch_test_atom refs/heads/main '%(committername)' fail\n ++batch_test_atom refs/heads/main '%(committeremail)' fail\n ++batch_test_atom refs/heads/main '%(committeremail:trim)' fail\n ++batch_test_atom refs/heads/main '%(committeremail:localpart)' fail\n ++batch_test_atom refs/heads/main '%(committerdate)' fail\n ++batch_test_atom refs/heads/main '%(tag)' fail\n ++batch_test_atom refs/heads/main '%(tagger)' fail\n ++batch_test_atom refs/heads/main '%(taggername)' fail\n ++batch_test_atom refs/heads/main '%(taggeremail)' fail\n ++batch_test_atom refs/heads/main '%(taggeremail:trim)' fail\n ++batch_test_atom refs/heads/main '%(taggeremail:localpart)' fail\n ++batch_test_atom refs/heads/main '%(taggerdate)' fail\n ++batch_test_atom refs/heads/main '%(creator)' fail\n ++batch_test_atom refs/heads/main '%(creatordate)' fail\n ++batch_test_atom refs/heads/main '%(subject)' fail\n ++batch_test_atom refs/heads/main '%(subject:sanitize)' fail\n ++batch_test_atom refs/heads/main '%(contents:subject)' fail\n ++batch_test_atom refs/heads/main '%(body)' fail\n ++batch_test_atom refs/heads/main '%(contents:body)' fail\n ++batch_test_atom refs/heads/main '%(contents:signature)' fail\n ++batch_test_atom refs/heads/main '%(contents)' fail\n ++batch_test_atom refs/heads/main '%(HEAD)' fail\n ++batch_test_atom refs/heads/main '%(upstream:track)' fail\n ++batch_test_atom refs/heads/main '%(upstream:trackshort)' fail\n ++batch_test_atom refs/heads/main '%(upstream:track,nobracket)' fail\n ++batch_test_atom refs/heads/main '%(upstream:nobracket,track)' fail\n ++batch_test_atom refs/heads/main '%(push:track)' fail\n ++batch_test_atom refs/heads/main '%(push:trackshort)' fail\n ++batch_test_atom refs/heads/main '%(worktreepath)' fail\n ++batch_test_atom refs/heads/main '%(symref)' fail\n ++batch_test_atom refs/heads/main '%(flag)' fail\n ++\n ++batch_test_atom refs/tags/testtag '%(refname)' fail\n ++batch_test_atom refs/tags/testtag '%(refname:short)' fail\n ++batch_test_atom refs/tags/testtag '%(upstream)' fail\n ++batch_test_atom refs/tags/testtag '%(push)' fail\n ++batch_test_atom refs/tags/testtag '%(objecttype)'\n ++batch_test_atom refs/tags/testtag '%(objectsize)'\n ++batch_test_atom refs/tags/testtag '%(objectsize:disk)'\n ++batch_test_atom refs/tags/testtag '%(*objectsize:disk)' fail\n ++batch_test_atom refs/tags/testtag '%(deltabase)'\n ++batch_test_atom refs/tags/testtag '%(*deltabase)' fail\n ++batch_test_atom refs/tags/testtag '%(objectname)'\n ++batch_test_atom refs/tags/testtag '%(objectname:short)' fail\n ++batch_test_atom refs/tags/testtag '%(tree)' fail\n ++batch_test_atom refs/tags/testtag '%(tree:short)' fail\n ++batch_test_atom refs/tags/testtag '%(tree:short=1)' fail\n ++batch_test_atom refs/tags/testtag '%(tree:short=10)' fail\n ++batch_test_atom refs/tags/testtag '%(parent)' fail\n ++batch_test_atom refs/tags/testtag '%(parent:short)' fail\n ++batch_test_atom refs/tags/testtag '%(parent:short=1)' fail\n ++batch_test_atom refs/tags/testtag '%(parent:short=10)' fail\n ++batch_test_atom refs/tags/testtag '%(numparent)' fail\n ++batch_test_atom refs/tags/testtag '%(object)' fail\n ++batch_test_atom refs/tags/testtag '%(type)' fail\n ++batch_test_atom refs/tags/testtag '%(*objectname)' fail\n ++batch_test_atom refs/tags/testtag '%(*objecttype)' fail\n ++batch_test_atom refs/tags/testtag '%(author)' fail\n ++batch_test_atom refs/tags/testtag '%(authorname)' fail\n ++batch_test_atom refs/tags/testtag '%(authoremail)' fail\n ++batch_test_atom refs/tags/testtag '%(authoremail:trim)' fail\n ++batch_test_atom refs/tags/testtag '%(authoremail:localpart)' fail\n ++batch_test_atom refs/tags/testtag '%(authordate)' fail\n ++batch_test_atom refs/tags/testtag '%(committer)' fail\n ++batch_test_atom refs/tags/testtag '%(committername)' fail\n ++batch_test_atom refs/tags/testtag '%(committeremail)' fail\n ++batch_test_atom refs/tags/testtag '%(committeremail:trim)' fail\n ++batch_test_atom refs/tags/testtag '%(committeremail:localpart)' fail\n ++batch_test_atom refs/tags/testtag '%(committerdate)' fail\n ++batch_test_atom refs/tags/testtag '%(tag)' fail\n ++batch_test_atom refs/tags/testtag '%(tagger)' fail\n ++batch_test_atom refs/tags/testtag '%(taggername)' fail\n ++batch_test_atom refs/tags/testtag '%(taggeremail)' fail\n ++batch_test_atom refs/tags/testtag '%(taggeremail:trim)' fail\n ++batch_test_atom refs/tags/testtag '%(taggeremail:localpart)' fail\n ++batch_test_atom refs/tags/testtag '%(taggerdate)' fail\n ++batch_test_atom refs/tags/testtag '%(creator)' fail\n ++batch_test_atom refs/tags/testtag '%(creatordate)' fail\n ++batch_test_atom refs/tags/testtag '%(subject)' fail\n ++batch_test_atom refs/tags/testtag '%(subject:sanitize)' fail\n ++batch_test_atom refs/tags/testtag '%(contents:subject)' fail\n ++batch_test_atom refs/tags/testtag '%(body)' fail\n ++batch_test_atom refs/tags/testtag '%(contents:body)' fail\n ++batch_test_atom refs/tags/testtag '%(contents:signature)' fail\n ++batch_test_atom refs/tags/testtag '%(contents)' fail\n ++batch_test_atom refs/tags/testtag '%(HEAD)' fail\n ++\n ++batch_test_atom refs/myblobs/blob1 '%(refname)' fail\n ++batch_test_atom refs/myblobs/blob1 '%(upstream)' fail\n ++batch_test_atom refs/myblobs/blob1 '%(push)' fail\n ++batch_test_atom refs/myblobs/blob1 '%(HEAD)' fail\n ++\n ++batch_test_atom refs/myblobs/blob1 '%(objectname)'\n ++batch_test_atom refs/myblobs/blob1 '%(objecttype)'\n ++batch_test_atom refs/myblobs/blob1 '%(objectsize)'\n ++batch_test_atom refs/myblobs/blob1 '%(objectsize:disk)'\n ++batch_test_atom refs/myblobs/blob1 '%(deltabase)'\n ++\n ++batch_test_atom refs/myblobs/blob1 '%(contents)' fail\n ++batch_test_atom refs/myblobs/blob2 '%(contents)' fail\n ++\n ++batch_test_atom refs/myblobs/blob1 '%(raw)' fail\n ++batch_test_atom refs/myblobs/blob2 '%(raw)' fail\n ++batch_test_atom refs/mytrees/tree1 '%(raw)' fail\n ++\n ++batch_test_atom refs/myblobs/blob1 '%(raw:size)' fail\n ++batch_test_atom refs/myblobs/blob2 '%(raw:size)' fail\n ++batch_test_atom refs/mytrees/tree1 '%(raw:size)' fail\n ++\n ++batch_test_atom refs/myblobs/blob1 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)' fail\n ++batch_test_atom refs/myblobs/blob2 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)' fail\n ++batch_test_atom refs/mytrees/tree1 '%(if:equals=tree)%(objecttype)%(then)tree%(else)not tree%(end)' fail\n ++\n ++batch_test_atom refs/heads/main '%(align:60) objectname is %(objectname)%(end)|%(objectname)' fail\n ++batch_test_atom refs/heads/main '%(align:left,60) objectname is %(objectname)%(end)|%(objectname)' fail\n ++batch_test_atom refs/heads/main '%(align:middle,60) objectname is %(objectname)%(end)|%(objectname)' fail\n ++batch_test_atom refs/heads/main '%(align:60,right) objectname is %(objectname)%(end)|%(objectname)' fail\n ++\n ++batch_test_atom refs/heads/main 'VALID'\n ++batch_test_atom refs/heads/main '%(INVALID)' fail\n ++batch_test_atom refs/heads/main '%(authordate:INVALID)' fail\n ++\n ++batch_test_atom refs/heads/main '%(objectname) %(objecttype) %(objectsize)\n ++%(raw)' fail\n ++batch_test_atom refs/tags/testtag '%(objectname) %(objecttype) %(objectsize)\n ++%(raw)' fail\n ++batch_test_atom refs/myblobs/blob1 '%(objectname) %(objecttype) %(objectsize)\n ++%(raw)' fail\n ++batch_test_atom refs/myblobs/blob2 '%(objectname) %(objecttype) %(objectsize)\n ++%(raw)' fail\n ++\n + test_done\n 11: 790c558d7cc = 9: 7058f47d41d [GSOC] ref-filter: modify the error message and value in get_object\n 12: fca49379025 = 10: 3af0def894c [GSOC] cat-file: add has_object_file() check\n 13: 43ce6bf2626 = 11: c445fa6520b [GSOC] cat-file: change batch_objects parameter name\n 18: 2edca9dc465 = 12: ae3a7816013 [GSOC] cat-file: create p1006-cat-file.sh\n 14: 70e83e4ba3c ! 13: 8b26c9e7455 [GSOC] cat-file: reuse ref-filter logic\n @@ Commit message\n \n The performance for `git cat-file --batch-all-objects\n --batch-check` on the Git repository itself with performance\n - testing tool `hyperfine` changes from 669.4 ms ± 31.1 ms to\n - 1.134 s ± 0.063 s.\n + testing tool `hyperfine` changes from 616.7 ms ± 8.9 ms to\n + 758.7 ms ± 16.4 ms.\n \n The performance for `git cat-file --batch-all-objects --batch\n >/dev/null` on the Git repository itself with performance testing\n - tool `time` change from \"27.37s user 0.29s system 98% cpu 28.089\n - total\" to \"33.69s user 1.54s system 87% cpu 40.258 total\".\n + tool `time` change from \"25.26s user 0.30s system 98% cpu 25.840 total\"\n + to \"28.79s user 0.83s system 99% cpu 29.829 total\".\n +\n + The reasons for the performance degradation are as follows:\n + 1. There are a lot of data copies in the logic of ref-filter.\n + 2, In order to be able to support more useful formats, complex\n + data structure and parsing process are used in ref-filter.\n +\n + A later patch will add a fast path which will mitigate the\n + performance regression introduced by this patch.\n \n Mentored-by: Christian Couder \n Mentored-by: Hariom Verma \n @@ builtin/cat-file.c: int cmd_cat_file(int argc, const char **argv, const char *pr\n \t\tdie(\"git cat-file --allow-unknown-type: use with -s or -t\");\n \n ## t/t1006-cat-file.sh ##\n -@@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch=\"batman\" with --batch-all-objects will wor\n - \tgit -C all-two cat-file --batch-all-objects --batch=\"batman\" >actual &&\n - \tcmp expect actual\n - '\n -+. \"$TEST_DIRECTORY\"/lib-gpg.sh\n -+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n -+\n -+test_expect_success 'cat-file --batch|--batch-check setup' '\n -+\techo 1>blob1 &&\n -+\tprintf \"a\\0b\\0\\c\" >blob2 &&\n -+\tgit add blob1 blob2 &&\n -+\tgit commit -m \"Commit Message\" &&\n -+\tgit branch -M main &&\n -+\tgit tag -a -m \"v0.0.0\" testtag &&\n -+\tgit update-ref refs/myblobs/blob1 HEAD:blob1 &&\n -+\tgit update-ref refs/myblobs/blob2 HEAD:blob2 &&\n -+\tgit update-ref refs/mytrees/tree1 HEAD^{tree}\n -+'\n -+\n -+batch_test_atom() {\n -+\tif test \"$3\" = \"fail\"\n -+\tthen\n -+\t\ttest_expect_${4:-success} $PREREQ \"basic atom: $1 $2 must fail\" \"\n -+\t\t\ttest_must_fail git cat-file --batch-check='$2' >bad <<-EOF\n -+\t\t\t$1\n -+\t\t\tEOF\n -+\t\t\"\n -+\telse\n -+\t\ttest_expect_${4:-success} $PREREQ \"basic atom: $1 $2\" \"\n -+\t\t\tgit for-each-ref --format='$2' $1 >expected &&\n -+\t\t\tgit cat-file --batch-check='$2' >actual <<-EOF &&\n -+\t\t\t$1\n -+\t\t\tEOF\n -+\t\t\tsanitize_pgp actual.clean &&\n -+\t\t\tcmp expected actual.clean\n -+\t\t\"\n -+\tfi\n -+}\n -+\n -+batch_test_atom refs/heads/main '%(refname)' fail\n -+batch_test_atom refs/heads/main '%(refname:)' fail\n -+batch_test_atom refs/heads/main '%(refname:short)' fail\n -+batch_test_atom refs/heads/main '%(refname:lstrip=1)' fail\n -+batch_test_atom refs/heads/main '%(refname:lstrip=2)' fail\n -+batch_test_atom refs/heads/main '%(refname:lstrip=-1)' fail\n -+batch_test_atom refs/heads/main '%(refname:lstrip=-2)' fail\n -+batch_test_atom refs/heads/main '%(refname:rstrip=1)' fail\n -+batch_test_atom refs/heads/main '%(refname:rstrip=2)' fail\n -+batch_test_atom refs/heads/main '%(refname:rstrip=-1)' fail\n -+batch_test_atom refs/heads/main '%(refname:rstrip=-2)' fail\n -+batch_test_atom refs/heads/main '%(refname:strip=1)' fail\n -+batch_test_atom refs/heads/main '%(refname:strip=2)' fail\n -+batch_test_atom refs/heads/main '%(refname:strip=-1)' fail\n -+batch_test_atom refs/heads/main '%(refname:strip=-2)' fail\n -+batch_test_atom refs/heads/main '%(upstream)' fail\n -+batch_test_atom refs/heads/main '%(upstream:short)' fail\n -+batch_test_atom refs/heads/main '%(upstream:lstrip=2)' fail\n -+batch_test_atom refs/heads/main '%(upstream:lstrip=-2)' fail\n -+batch_test_atom refs/heads/main '%(upstream:rstrip=2)' fail\n -+batch_test_atom refs/heads/main '%(upstream:rstrip=-2)' fail\n -+batch_test_atom refs/heads/main '%(upstream:strip=2)' fail\n -+batch_test_atom refs/heads/main '%(upstream:strip=-2)' fail\n -+batch_test_atom refs/heads/main '%(push)' fail\n -+batch_test_atom refs/heads/main '%(push:short)' fail\n -+batch_test_atom refs/heads/main '%(push:lstrip=1)' fail\n -+batch_test_atom refs/heads/main '%(push:lstrip=-1)' fail\n -+batch_test_atom refs/heads/main '%(push:rstrip=1)' fail\n -+batch_test_atom refs/heads/main '%(push:rstrip=-1)' fail\n -+batch_test_atom refs/heads/main '%(push:strip=1)' fail\n -+batch_test_atom refs/heads/main '%(push:strip=-1)' fail\n -+batch_test_atom refs/heads/main '%(objecttype)'\n -+batch_test_atom refs/heads/main '%(objectsize)'\n -+batch_test_atom refs/heads/main '%(objectsize:disk)'\n -+batch_test_atom refs/heads/main '%(deltabase)'\n -+batch_test_atom refs/heads/main '%(objectname)'\n +@@ t/t1006-cat-file.sh: batch_test_atom refs/heads/main '%(objectsize)'\n + batch_test_atom refs/heads/main '%(objectsize:disk)'\n + batch_test_atom refs/heads/main '%(deltabase)'\n + batch_test_atom refs/heads/main '%(objectname)'\n +-batch_test_atom refs/heads/main '%(objectname:short)' fail\n +-batch_test_atom refs/heads/main '%(objectname:short=1)' fail\n +-batch_test_atom refs/heads/main '%(objectname:short=10)' fail\n +-batch_test_atom refs/heads/main '%(tree)' fail\n +-batch_test_atom refs/heads/main '%(tree:short)' fail\n +-batch_test_atom refs/heads/main '%(tree:short=1)' fail\n +-batch_test_atom refs/heads/main '%(tree:short=10)' fail\n +-batch_test_atom refs/heads/main '%(parent)' fail\n +-batch_test_atom refs/heads/main '%(parent:short)' fail\n +-batch_test_atom refs/heads/main '%(parent:short=1)' fail\n +-batch_test_atom refs/heads/main '%(parent:short=10)' fail\n +-batch_test_atom refs/heads/main '%(numparent)' fail\n +-batch_test_atom refs/heads/main '%(object)' fail\n +-batch_test_atom refs/heads/main '%(type)' fail\n +-batch_test_atom refs/heads/main '%(raw)' fail\n +-batch_test_atom refs/heads/main '%(*objectname)' fail\n +-batch_test_atom refs/heads/main '%(*objecttype)' fail\n +-batch_test_atom refs/heads/main '%(author)' fail\n +-batch_test_atom refs/heads/main '%(authorname)' fail\n +-batch_test_atom refs/heads/main '%(authoremail)' fail\n +-batch_test_atom refs/heads/main '%(authoremail:trim)' fail\n +-batch_test_atom refs/heads/main '%(authoremail:localpart)' fail\n +-batch_test_atom refs/heads/main '%(authordate)' fail\n +-batch_test_atom refs/heads/main '%(committer)' fail\n +-batch_test_atom refs/heads/main '%(committername)' fail\n +-batch_test_atom refs/heads/main '%(committeremail)' fail\n +-batch_test_atom refs/heads/main '%(committeremail:trim)' fail\n +-batch_test_atom refs/heads/main '%(committeremail:localpart)' fail\n +-batch_test_atom refs/heads/main '%(committerdate)' fail\n +-batch_test_atom refs/heads/main '%(tag)' fail\n +-batch_test_atom refs/heads/main '%(tagger)' fail\n +-batch_test_atom refs/heads/main '%(taggername)' fail\n +-batch_test_atom refs/heads/main '%(taggeremail)' fail\n +-batch_test_atom refs/heads/main '%(taggeremail:trim)' fail\n +-batch_test_atom refs/heads/main '%(taggeremail:localpart)' fail\n +-batch_test_atom refs/heads/main '%(taggerdate)' fail\n +-batch_test_atom refs/heads/main '%(creator)' fail\n +-batch_test_atom refs/heads/main '%(creatordate)' fail\n +-batch_test_atom refs/heads/main '%(subject)' fail\n +-batch_test_atom refs/heads/main '%(subject:sanitize)' fail\n +-batch_test_atom refs/heads/main '%(contents:subject)' fail\n +-batch_test_atom refs/heads/main '%(body)' fail\n +-batch_test_atom refs/heads/main '%(contents:body)' fail\n +-batch_test_atom refs/heads/main '%(contents:signature)' fail\n +-batch_test_atom refs/heads/main '%(contents)' fail\n +batch_test_atom refs/heads/main '%(objectname:short)'\n +batch_test_atom refs/heads/main '%(objectname:short=1)'\n +batch_test_atom refs/heads/main '%(objectname:short=10)'\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch=\"batman\" with --batch\n +batch_test_atom refs/heads/main '%(contents:body)'\n +batch_test_atom refs/heads/main '%(contents:signature)'\n +batch_test_atom refs/heads/main '%(contents)'\n -+batch_test_atom refs/heads/main '%(HEAD)' fail\n -+batch_test_atom refs/heads/main '%(upstream:track)' fail\n -+batch_test_atom refs/heads/main '%(upstream:trackshort)' fail\n -+batch_test_atom refs/heads/main '%(upstream:track,nobracket)' fail\n -+batch_test_atom refs/heads/main '%(upstream:nobracket,track)' fail\n -+batch_test_atom refs/heads/main '%(push:track)' fail\n -+batch_test_atom refs/heads/main '%(push:trackshort)' fail\n -+batch_test_atom refs/heads/main '%(worktreepath)' fail\n -+batch_test_atom refs/heads/main '%(symref)' fail\n -+batch_test_atom refs/heads/main '%(flag)' fail\n -+\n -+batch_test_atom refs/tags/testtag '%(refname)' fail\n -+batch_test_atom refs/tags/testtag '%(refname:short)' fail\n -+batch_test_atom refs/tags/testtag '%(upstream)' fail\n -+batch_test_atom refs/tags/testtag '%(push)' fail\n -+batch_test_atom refs/tags/testtag '%(objecttype)'\n -+batch_test_atom refs/tags/testtag '%(objectsize)'\n -+batch_test_atom refs/tags/testtag '%(objectsize:disk)'\n + batch_test_atom refs/heads/main '%(HEAD)' fail\n + batch_test_atom refs/heads/main '%(upstream:track)' fail\n + batch_test_atom refs/heads/main '%(upstream:trackshort)' fail\n +@@ t/t1006-cat-file.sh: batch_test_atom refs/tags/testtag '%(push)' fail\n + batch_test_atom refs/tags/testtag '%(objecttype)'\n + batch_test_atom refs/tags/testtag '%(objectsize)'\n + batch_test_atom refs/tags/testtag '%(objectsize:disk)'\n +-batch_test_atom refs/tags/testtag '%(*objectsize:disk)' fail\n +batch_test_atom refs/tags/testtag '%(*objectsize:disk)'\n -+batch_test_atom refs/tags/testtag '%(deltabase)'\n + batch_test_atom refs/tags/testtag '%(deltabase)'\n +-batch_test_atom refs/tags/testtag '%(*deltabase)' fail\n +batch_test_atom refs/tags/testtag '%(*deltabase)'\n -+batch_test_atom refs/tags/testtag '%(objectname)'\n + batch_test_atom refs/tags/testtag '%(objectname)'\n +-batch_test_atom refs/tags/testtag '%(objectname:short)' fail\n +-batch_test_atom refs/tags/testtag '%(tree)' fail\n +-batch_test_atom refs/tags/testtag '%(tree:short)' fail\n +-batch_test_atom refs/tags/testtag '%(tree:short=1)' fail\n +-batch_test_atom refs/tags/testtag '%(tree:short=10)' fail\n +-batch_test_atom refs/tags/testtag '%(parent)' fail\n +-batch_test_atom refs/tags/testtag '%(parent:short)' fail\n +-batch_test_atom refs/tags/testtag '%(parent:short=1)' fail\n +-batch_test_atom refs/tags/testtag '%(parent:short=10)' fail\n +-batch_test_atom refs/tags/testtag '%(numparent)' fail\n +-batch_test_atom refs/tags/testtag '%(object)' fail\n +-batch_test_atom refs/tags/testtag '%(type)' fail\n +-batch_test_atom refs/tags/testtag '%(*objectname)' fail\n +-batch_test_atom refs/tags/testtag '%(*objecttype)' fail\n +-batch_test_atom refs/tags/testtag '%(author)' fail\n +-batch_test_atom refs/tags/testtag '%(authorname)' fail\n +-batch_test_atom refs/tags/testtag '%(authoremail)' fail\n +-batch_test_atom refs/tags/testtag '%(authoremail:trim)' fail\n +-batch_test_atom refs/tags/testtag '%(authoremail:localpart)' fail\n +-batch_test_atom refs/tags/testtag '%(authordate)' fail\n +-batch_test_atom refs/tags/testtag '%(committer)' fail\n +-batch_test_atom refs/tags/testtag '%(committername)' fail\n +-batch_test_atom refs/tags/testtag '%(committeremail)' fail\n +-batch_test_atom refs/tags/testtag '%(committeremail:trim)' fail\n +-batch_test_atom refs/tags/testtag '%(committeremail:localpart)' fail\n +-batch_test_atom refs/tags/testtag '%(committerdate)' fail\n +-batch_test_atom refs/tags/testtag '%(tag)' fail\n +-batch_test_atom refs/tags/testtag '%(tagger)' fail\n +-batch_test_atom refs/tags/testtag '%(taggername)' fail\n +-batch_test_atom refs/tags/testtag '%(taggeremail)' fail\n +-batch_test_atom refs/tags/testtag '%(taggeremail:trim)' fail\n +-batch_test_atom refs/tags/testtag '%(taggeremail:localpart)' fail\n +-batch_test_atom refs/tags/testtag '%(taggerdate)' fail\n +-batch_test_atom refs/tags/testtag '%(creator)' fail\n +-batch_test_atom refs/tags/testtag '%(creatordate)' fail\n +-batch_test_atom refs/tags/testtag '%(subject)' fail\n +-batch_test_atom refs/tags/testtag '%(subject:sanitize)' fail\n +-batch_test_atom refs/tags/testtag '%(contents:subject)' fail\n +-batch_test_atom refs/tags/testtag '%(body)' fail\n +-batch_test_atom refs/tags/testtag '%(contents:body)' fail\n +-batch_test_atom refs/tags/testtag '%(contents:signature)' fail\n +-batch_test_atom refs/tags/testtag '%(contents)' fail\n +batch_test_atom refs/tags/testtag '%(objectname:short)'\n +batch_test_atom refs/tags/testtag '%(tree)'\n +batch_test_atom refs/tags/testtag '%(tree:short)'\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file --batch=\"batman\" with --batch\n +batch_test_atom refs/tags/testtag '%(contents:body)'\n +batch_test_atom refs/tags/testtag '%(contents:signature)'\n +batch_test_atom refs/tags/testtag '%(contents)'\n -+batch_test_atom refs/tags/testtag '%(HEAD)' fail\n -+\n -+batch_test_atom refs/myblobs/blob1 '%(refname)' fail\n -+batch_test_atom refs/myblobs/blob1 '%(upstream)' fail\n -+batch_test_atom refs/myblobs/blob1 '%(push)' fail\n -+batch_test_atom refs/myblobs/blob1 '%(HEAD)' fail\n -+\n -+batch_test_atom refs/myblobs/blob1 '%(objectname)'\n -+batch_test_atom refs/myblobs/blob1 '%(objecttype)'\n -+batch_test_atom refs/myblobs/blob1 '%(objectsize)'\n -+batch_test_atom refs/myblobs/blob1 '%(objectsize:disk)'\n -+batch_test_atom refs/myblobs/blob1 '%(deltabase)'\n -+\n + batch_test_atom refs/tags/testtag '%(HEAD)' fail\n + \n + batch_test_atom refs/myblobs/blob1 '%(refname)' fail\n +@@ t/t1006-cat-file.sh: batch_test_atom refs/myblobs/blob1 '%(objectsize)'\n + batch_test_atom refs/myblobs/blob1 '%(objectsize:disk)'\n + batch_test_atom refs/myblobs/blob1 '%(deltabase)'\n + \n +-batch_test_atom refs/myblobs/blob1 '%(contents)' fail\n +-batch_test_atom refs/myblobs/blob2 '%(contents)' fail\n +batch_test_atom refs/myblobs/blob1 '%(contents)'\n +batch_test_atom refs/myblobs/blob2 '%(contents)'\n -+\n + \n +-batch_test_atom refs/myblobs/blob1 '%(raw)' fail\n +-batch_test_atom refs/myblobs/blob2 '%(raw)' fail\n +-batch_test_atom refs/mytrees/tree1 '%(raw)' fail\n +batch_test_atom refs/myblobs/blob1 '%(raw)'\n +batch_test_atom refs/myblobs/blob2 '%(raw)'\n +batch_test_atom refs/mytrees/tree1 '%(raw)'\n -+\n + \n +-batch_test_atom refs/myblobs/blob1 '%(raw:size)' fail\n +-batch_test_atom refs/myblobs/blob2 '%(raw:size)' fail\n +-batch_test_atom refs/mytrees/tree1 '%(raw:size)' fail\n +batch_test_atom refs/myblobs/blob1 '%(raw:size)'\n +batch_test_atom refs/myblobs/blob2 '%(raw:size)'\n +batch_test_atom refs/mytrees/tree1 '%(raw:size)'\n -+\n + \n +-batch_test_atom refs/myblobs/blob1 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)' fail\n +-batch_test_atom refs/myblobs/blob2 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)' fail\n +-batch_test_atom refs/mytrees/tree1 '%(if:equals=tree)%(objecttype)%(then)tree%(else)not tree%(end)' fail\n +batch_test_atom refs/myblobs/blob1 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)'\n +batch_test_atom refs/myblobs/blob2 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)'\n +batch_test_atom refs/mytrees/tree1 '%(if:equals=tree)%(objecttype)%(then)tree%(else)not tree%(end)'\n -+\n + \n +-batch_test_atom refs/heads/main '%(align:60) objectname is %(objectname)%(end)|%(objectname)' fail\n +-batch_test_atom refs/heads/main '%(align:left,60) objectname is %(objectname)%(end)|%(objectname)' fail\n +-batch_test_atom refs/heads/main '%(align:middle,60) objectname is %(objectname)%(end)|%(objectname)' fail\n +-batch_test_atom refs/heads/main '%(align:60,right) objectname is %(objectname)%(end)|%(objectname)' fail\n +batch_test_atom refs/heads/main '%(align:60) objectname is %(objectname)%(end)|%(objectname)'\n +batch_test_atom refs/heads/main '%(align:left,60) objectname is %(objectname)%(end)|%(objectname)'\n +batch_test_atom refs/heads/main '%(align:middle,60) objectname is %(objectname)%(end)|%(objectname)'\n +batch_test_atom refs/heads/main '%(align:60,right) objectname is %(objectname)%(end)|%(objectname)'\n -+\n -+batch_test_atom refs/heads/main 'VALID'\n -+batch_test_atom refs/heads/main '%(INVALID)' fail\n -+batch_test_atom refs/heads/main '%(authordate:INVALID)' fail\n -+\n -+test_expect_success '%(rest) works with both a branch and a tag' '\n -+\tcat >expected <<-EOF &&\n -+\t123 commit 123\n -+\t456 tag 456\n -+\tEOF\n -+\tgit cat-file --batch-check=\"%(rest) %(objecttype) %(rest)\" >actual <<-EOF &&\n -+\trefs/heads/main 123\n -+\trefs/tags/testtag 456\n -+\tEOF\n -+\ttest_cmp expected actual\n -+'\n -+\n -+batch_test_atom refs/heads/main '%(objectname) %(objecttype) %(objectsize)\n + \n + batch_test_atom refs/heads/main 'VALID'\n + batch_test_atom refs/heads/main '%(INVALID)' fail\n + batch_test_atom refs/heads/main '%(authordate:INVALID)' fail\n + \n + batch_test_atom refs/heads/main '%(objectname) %(objecttype) %(objectsize)\n +-%(raw)' fail\n +%(raw)'\n -+batch_test_atom refs/tags/testtag '%(objectname) %(objecttype) %(objectsize)\n + batch_test_atom refs/tags/testtag '%(objectname) %(objecttype) %(objectsize)\n +-%(raw)' fail\n +%(raw)'\n -+batch_test_atom refs/myblobs/blob1 '%(objectname) %(objecttype) %(objectsize)\n + batch_test_atom refs/myblobs/blob1 '%(objectname) %(objecttype) %(objectsize)\n +-%(raw)' fail\n +%(raw)'\n -+batch_test_atom refs/myblobs/blob2 '%(objectname) %(objecttype) %(objectsize)\n + batch_test_atom refs/myblobs/blob2 '%(objectname) %(objecttype) %(objectsize)\n +-%(raw)' fail\n +%(raw)'\n +\n -+\n +test_expect_success 'cat-file --batch equals to --batch-check with atoms' '\n +\tgit cat-file --batch-check=\"%(objectname) %(objecttype) %(objectsize)\n +%(raw)\" >expected <<-EOF &&\n 15: e20780e9a6c = 14: fd3901dfee6 [GSOC] cat-file: reuse err buf in batch_object_write()\n 16: fa74bf9451c = 15: ed556e5f31e [GSOC] cat-file: re-implement --textconv, --filters options\n 17: ff74fa9f2f2 = 16: 96ef6431a2b [GSOC] ref-filter: remove grab_oid() function\n 19: c35e7dfe542 = 17: 5903d02324f [GSOC] cat-file: use fast path when using default_format\n\n-- \ngitgitgadget\n"},{"id":"430211","messageId":"554d7653ee7a9c4f57136eb5bdcd924338268532.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 02/17] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:10Z","receivedAt":"2021-07-15T15:40:33Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd new formatting option `%(raw)`, which will print the raw\nobject data without any changes. It will help further to migrate\nall cat-file formatting logic from cat-file to ref-filter.\n\nThe raw data of blob, tree objects may contain '\\0', but most of\nthe logic in `ref-filter` depends on the output of the atom being\ntext (specifically, no embedded NULs in it).\n\nE.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()`\nadd the data to the buffer. The raw data of a tree object is\n`100644 one\\0...`, only the `100644 one` will be added to the buffer,\nwhich is incorrect.\n\nTherefore, we need to find a way to record the length of the\natom_value's member `s`. Although strbuf can already record the\nstring and its length, if we want to replace the type of atom_value's\nmember `s` with strbuf, many places in ref-filter that are filled\nwith dynamically allocated mermory in `v->s` are not easy to replace.\nAt the same time, we need to check if `v->s == NULL` in\npopulate_value(), and strbuf cannot easily distinguish NULL and empty\nstrings, but c-style \"const char *\" can do it. So add a new member in\n`struct atom_value`: `s_size`, which can record raw object size, it\ncan help us add raw object data to the buffer or compare two buffers\nwhich contain raw object data.\n\nNote that `--format=%(raw)` cannot be used with `--python`, `--shell`,\n`--tcl`, and `--perl` because if the binary raw data is passed to a\nvariable in such languages, these may not support arbitrary binary data\nin their string variable type.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Bagas Sanjaya \nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Felipe Contreras \nHelped-by: Phillip Wood \nHelped-by: Junio C Hamano \nBased-on-patch-by: Olga Telezhnaya \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 9 ++\n ref-filter.c | 140 +++++++++++++++----\n t/t6300-for-each-ref.sh | 216 +++++++++++++++++++++++++++++\n 3 files changed, 338 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 2ae2478de70..cbb6f87d13f 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`,\n without angle brackets, and `:localpart` to get the part before the `@` symbol\n out of the trimmed email.\n \n+The raw data in an object is `raw`.\n+\n+raw:size::\n+\tThe raw data size of the object.\n+\n+Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n+`--perl` because such language may not support arbitrary binary data in their\n+string variable type.\n+\n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5cee6512fba..506fbc3d691 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -144,6 +144,7 @@ enum atom_type {\n \tATOM_BODY,\n \tATOM_TRAILERS,\n \tATOM_CONTENTS,\n+\tATOM_RAW,\n \tATOM_UPSTREAM,\n \tATOM_PUSH,\n \tATOM_SYMREF,\n@@ -189,6 +190,9 @@ static struct used_atom {\n \t\t\tstruct process_trailer_options trailer_opts;\n \t\t\tunsigned int nlines;\n \t\t} contents;\n+\t\tstruct {\n+\t\t\tenum { RAW_BARE, RAW_LENGTH } option;\n+\t\t} raw_data;\n \t\tstruct {\n \t\t\tcmp_status cmp_status;\n \t\t\tconst char *str;\n@@ -426,6 +430,18 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n+static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\tconst char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\tatom->u.raw_data.option = RAW_BARE;\n+\telse if (!strcmp(arg, \"size\"))\n+\t\tatom->u.raw_data.option = RAW_LENGTH;\n+\telse\n+\t\treturn strbuf_addf_ret(err, -1, _(\"unrecognized %%(raw) argument: %s\"), arg);\n+\treturn 0;\n+}\n+\n static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -586,6 +602,7 @@ static struct {\n \t[ATOM_BODY] = { \"body\", SOURCE_OBJ, FIELD_STR, body_atom_parser },\n \t[ATOM_TRAILERS] = { \"trailers\", SOURCE_OBJ, FIELD_STR, trailers_atom_parser },\n \t[ATOM_CONTENTS] = { \"contents\", SOURCE_OBJ, FIELD_STR, contents_atom_parser },\n+\t[ATOM_RAW] = { \"raw\", SOURCE_OBJ, FIELD_STR, raw_atom_parser },\n \t[ATOM_UPSTREAM] = { \"upstream\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_PUSH] = { \"push\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_SYMREF] = { \"symref\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -620,12 +637,15 @@ struct ref_formatting_state {\n \n struct atom_value {\n \tconst char *s;\n+\tsize_t s_size;\n \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t struct strbuf *err);\n \tuintmax_t value; /* used for sorting when not FIELD_STR */\n \tstruct used_atom *atom;\n };\n \n+#define ATOM_VALUE_S_SIZE_INIT (-1)\n+\n /*\n * Used to parse format string and sort specifiers\n */\n@@ -644,13 +664,6 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \t\treturn strbuf_addf_ret(err, -1, _(\"malformed field name: %.*s\"),\n \t\t\t\t (int)(ep-atom), atom);\n \n-\t/* Do we have the atom already used elsewhere? */\n-\tfor (i = 0; i < used_atom_cnt; i++) {\n-\t\tint len = strlen(used_atom[i].name);\n-\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n-\t\t\treturn i;\n-\t}\n-\n \t/*\n \t * If the atom name has a colon, strip it and everything after\n \t * it off - it specifies the format for this entry, and\n@@ -660,6 +673,13 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \targ = memchr(sp, ':', ep - sp);\n \tatom_len = (arg ? arg : ep) - sp;\n \n+\t/* Do we have the atom already used elsewhere? */\n+\tfor (i = 0; i < used_atom_cnt; i++) {\n+\t\tint len = strlen(used_atom[i].name);\n+\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n+\t\t\treturn i;\n+\t}\n+\n \t/* Is the atom a valid one? */\n \tfor (i = 0; i < ARRAY_SIZE(valid_atom); i++) {\n \t\tint len = strlen(valid_atom[i].name);\n@@ -709,11 +729,14 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \treturn at;\n }\n \n-static void quote_formatting(struct strbuf *s, const char *str, int quote_style)\n+static void quote_formatting(struct strbuf *s, const char *str, size_t len, int quote_style)\n {\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n-\t\tstrbuf_addstr(s, str);\n+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tstrbuf_add(s, str, len);\n+\t\telse\n+\t\t\tstrbuf_addstr(s, str);\n \t\tbreak;\n \tcase QUOTE_SHELL:\n \t\tsq_quote_buf(s, str);\n@@ -740,9 +763,12 @@ static int append_atom(struct atom_value *v, struct ref_formatting_state *state,\n \t * encountered.\n \t */\n \tif (!state->stack->prev)\n-\t\tquote_formatting(&state->stack->output, v->s, state->quote_style);\n+\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n \telse\n-\t\tstrbuf_addstr(&state->stack->output, v->s);\n+\t\tif (v->s_size != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n+\t\telse\n+\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n \treturn 0;\n }\n \n@@ -842,21 +868,23 @@ static int if_atom_handler(struct atom_value *atomv, struct ref_formatting_state\n \treturn 0;\n }\n \n-static int is_empty(const char *s)\n+static int is_empty(struct strbuf *buf)\n {\n-\twhile (*s != '\\0') {\n-\t\tif (!isspace(*s))\n-\t\t\treturn 0;\n-\t\ts++;\n-\t}\n-\treturn 1;\n-}\n+\tconst char *cur = buf->buf;\n+\tconst char *end = buf->buf + buf->len;\n+\n+\twhile (cur != end && (isspace(*cur)))\n+\t\tcur++;\n+\n+\treturn cur == end;\n+ }\n \n static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t\t struct strbuf *err)\n {\n \tstruct ref_formatting_stack *cur = state->stack;\n \tstruct if_then_else *if_then_else = NULL;\n+\tsize_t str_len = 0;\n \n \tif (cur->at_end == if_then_else_handler)\n \t\tif_then_else = (struct if_then_else *)cur->at_end_data;\n@@ -867,18 +895,22 @@ static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_sta\n \tif (if_then_else->else_atom_seen)\n \t\treturn strbuf_addf_ret(err, -1, _(\"format: %%(then) atom used after %%(else)\"));\n \tif_then_else->then_atom_seen = 1;\n+\tif (if_then_else->str)\n+\t\tstr_len = strlen(if_then_else->str);\n \t/*\n \t * If the 'equals' or 'notequals' attribute is used then\n \t * perform the required comparison. If not, only non-empty\n \t * strings satisfy the 'if' condition.\n \t */\n \tif (if_then_else->cmp_status == COMPARE_EQUAL) {\n-\t\tif (!strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len == cur->output.len &&\n+\t\t !memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n \t} else if (if_then_else->cmp_status == COMPARE_UNEQUAL) {\n-\t\tif (strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len != cur->output.len ||\n+\t\t memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n-\t} else if (cur->output.len && !is_empty(cur->output.buf))\n+\t} else if (cur->output.len && !is_empty(&cur->output))\n \t\tif_then_else->condition_satisfied = 1;\n \tstrbuf_reset(&cur->output);\n \treturn 0;\n@@ -924,7 +956,7 @@ static int end_atom_handler(struct atom_value *atomv, struct ref_formatting_stat\n \t * only on the topmost supporting atom.\n \t */\n \tif (!current->prev->prev) {\n-\t\tquote_formatting(&s, current->output.buf, state->quote_style);\n+\t\tquote_formatting(&s, current->output.buf, current->output.len, state->quote_style);\n \t\tstrbuf_swap(¤t->output, &s);\n \t}\n \tstrbuf_release(&s);\n@@ -974,6 +1006,10 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\t\t\tdie(_(\"--format=%.*s cannot be used with\"\n+\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\n@@ -1367,12 +1403,25 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp\n \t\tstruct used_atom *atom = &used_atom[i];\n \t\tconst char *name = atom->name;\n \t\tstruct atom_value *v = &val[i];\n+\t\tenum atom_type atom_type = atom->atom_type;\n \n \t\tif (!!deref != (*name == '*'))\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n \n+\t\tif (atom_type == ATOM_RAW) {\n+\t\t\tunsigned long buf_size = data->size;\n+\n+\t\t\tif (atom->u.raw_data.option == RAW_BARE) {\n+\t\t\t\tv->s = xmemdupz(buf, buf_size);\n+\t\t\t\tv->s_size = buf_size;\n+\t\t\t} else if (atom->u.raw_data.option == RAW_LENGTH) {\n+\t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX, (uintmax_t)buf_size);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif ((data->type != OBJ_TAG &&\n \t\t data->type != OBJ_COMMIT) ||\n \t\t (strcmp(name, \"body\") &&\n@@ -1460,9 +1509,11 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"Eh? Object of type %d?\", obj->type);\n@@ -1766,6 +1817,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n \n+\t\tv->s_size = ATOM_VALUE_S_SIZE_INIT;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n@@ -2369,6 +2421,19 @@ static int compare_detached_head(struct ref_array_item *a, struct ref_array_item\n \treturn 0;\n }\n \n+static int memcasecmp(const void *vs1, const void *vs2, size_t n)\n+{\n+\tconst char *s1 = vs1, *s2 = vs2;\n+\tconst char *end = s1 + n;\n+\n+\tfor (; s1 < end; s1++, s2++) {\n+\t\tint diff = tolower(*s1) - tolower(*s2);\n+\t\tif (diff)\n+\t\t\treturn diff;\n+\t}\n+\treturn 0;\n+}\n+\n static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b)\n {\n \tstruct atom_value *va, *vb;\n@@ -2389,10 +2454,30 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n \t} else if (s->sort_flags & REF_SORTING_VERSION) {\n \t\tcmp = versioncmp(va->s, vb->s);\n \t} else if (cmp_type == FIELD_STR) {\n-\t\tint (*cmp_fn)(const char *, const char *);\n-\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n-\t\t\t? strcasecmp : strcmp;\n-\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\tif (va->s_size == ATOM_VALUE_S_SIZE_INIT &&\n+\t\t vb->s_size == ATOM_VALUE_S_SIZE_INIT) {\n+\t\t\tint (*cmp_fn)(const char *, const char *);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? strcasecmp : strcmp;\n+\t\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\t} else {\n+\t\t\tsize_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT ?\n+\t\t\t\t\tstrlen(va->s) : va->s_size;\n+\t\t\tsize_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT ?\n+\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n+\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? memcasecmp : memcmp;\n+\n+\t\t\tcmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n+\t\t\t\t a_size : b_size);\n+\t\t\tif (!cmp) {\n+\t\t\t\tif (a_size > b_size)\n+\t\t\t\t\tcmp = 1;\n+\t\t\t\telse if (a_size < b_size)\n+\t\t\t\t\tcmp = -1;\n+\t\t\t}\n+\t\t}\n \t} else {\n \t\tif (va->value < vb->value)\n \t\t\tcmp = -1;\n@@ -2492,6 +2577,7 @@ int format_ref_array_item(struct ref_array_item *info,\n \t}\n \tif (format->need_color_reset_at_eol) {\n \t\tstruct atom_value resetv;\n+\t\tresetv.s_size = ATOM_VALUE_S_SIZE_INIT;\n \t\tresetv.s = GIT_COLOR_RESET;\n \t\tif (append_atom(&resetv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 9e0214076b4..18554f62d94 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -130,6 +130,8 @@ test_atom head parent:short=10 ''\n test_atom head numparent 0\n test_atom head object ''\n test_atom head type ''\n+test_atom head raw \"$(git cat-file commit refs/heads/main)\n+\"\n test_atom head '*objectname' ''\n test_atom head '*objecttype' ''\n test_atom head author 'A U Thor 1151968724 +0200'\n@@ -221,6 +223,15 @@ test_atom tag contents 'Tagging at 1151968727\n '\n test_atom tag HEAD ' '\n \n+test_expect_success 'basic atom: refs/tags/testtag *raw' '\n+\tgit cat-file commit refs/tags/testtag^{} >expected &&\n+\tgit for-each-ref --format=\"%(*raw)\" refs/tags/testtag >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'Check invalid atoms names are errors' '\n \ttest_must_fail git for-each-ref --format=\"%(INVALID)\" refs/heads\n '\n@@ -686,6 +697,15 @@ test_atom refs/tags/signed-empty contents:body ''\n test_atom refs/tags/signed-empty contents:signature \"$sig\"\n test_atom refs/tags/signed-empty contents \"$sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-empty raw' '\n+\tgit cat-file tag refs/tags/signed-empty >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-empty >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-short subject 'subject line'\n test_atom refs/tags/signed-short subject:sanitize 'subject-line'\n test_atom refs/tags/signed-short contents:subject 'subject line'\n@@ -695,6 +715,15 @@ test_atom refs/tags/signed-short contents:signature \"$sig\"\n test_atom refs/tags/signed-short contents \"subject line\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-short raw' '\n+\tgit cat-file tag refs/tags/signed-short >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-short >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-long subject 'subject line'\n test_atom refs/tags/signed-long subject:sanitize 'subject-line'\n test_atom refs/tags/signed-long contents:subject 'subject line'\n@@ -708,6 +737,15 @@ test_atom refs/tags/signed-long contents \"subject line\n body contents\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-long raw' '\n+\tgit cat-file tag refs/tags/signed-long >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-long >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'set up refs pointing to tree and blob' '\n \tgit update-ref refs/mytrees/first refs/heads/main^{tree} &&\n \tgit update-ref refs/myblobs/first refs/heads/main:one\n@@ -720,6 +758,16 @@ test_atom refs/mytrees/first contents:body \"\"\n test_atom refs/mytrees/first contents:signature \"\"\n test_atom refs/mytrees/first contents \"\"\n \n+test_expect_success 'basic atom: refs/mytrees/first raw' '\n+\tgit cat-file tree refs/mytrees/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/mytrees/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_atom refs/myblobs/first subject \"\"\n test_atom refs/myblobs/first contents:subject \"\"\n test_atom refs/myblobs/first body \"\"\n@@ -727,6 +775,174 @@ test_atom refs/myblobs/first contents:body \"\"\n test_atom refs/myblobs/first contents:signature \"\"\n test_atom refs/myblobs/first contents \"\"\n \n+test_expect_success 'basic atom: refs/myblobs/first raw' '\n+\tgit cat-file blob refs/myblobs/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/myblobs/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'set up refs pointing to binary blob' '\n+\tprintf \"a\\0b\\0c\" >blob1 &&\n+\tprintf \"a\\0c\\0b\" >blob2 &&\n+\tprintf \"\\0a\\0b\\0c\" >blob3 &&\n+\tprintf \"abc\" >blob4 &&\n+\tprintf \"\\0 \\0 \\0 \" >blob5 &&\n+\tprintf \"\\0 \\0a\\0 \" >blob6 &&\n+\tprintf \" \" >blob7 &&\n+\t>blob8 &&\n+\tobj=$(git hash-object -w blob1) &&\n+\tgit update-ref refs/myblobs/blob1 \"$obj\" &&\n+\tobj=$(git hash-object -w blob2) &&\n+\tgit update-ref refs/myblobs/blob2 \"$obj\" &&\n+\tobj=$(git hash-object -w blob3) &&\n+\tgit update-ref refs/myblobs/blob3 \"$obj\" &&\n+\tobj=$(git hash-object -w blob4) &&\n+\tgit update-ref refs/myblobs/blob4 \"$obj\" &&\n+\tobj=$(git hash-object -w blob5) &&\n+\tgit update-ref refs/myblobs/blob5 \"$obj\" &&\n+\tobj=$(git hash-object -w blob6) &&\n+\tgit update-ref refs/myblobs/blob6 \"$obj\" &&\n+\tobj=$(git hash-object -w blob7) &&\n+\tgit update-ref refs/myblobs/blob7 \"$obj\" &&\n+\tobj=$(git hash-object -w blob8) &&\n+\tgit update-ref refs/myblobs/blob8 \"$obj\"\n+'\n+\n+test_expect_success 'Verify sorts with raw' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob7\n+\trefs/mytrees/first\n+\trefs/myblobs/first\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob4\n+\trefs/heads/main\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'Verify sorts with raw:size' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\trefs/myblobs/blob7\n+\trefs/heads/main\n+\trefs/myblobs/blob4\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/mytrees/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw:size \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:equals)' '\n+\tcat >expected <<-EOF &&\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\trefs/myblobs/blob4\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:equals=abc)%(raw)%(then)%(refname)%(else)not equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:notequals)' '\n+\tcat >expected <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/heads/main\n+\trefs/heads/newtag\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\tequals\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob7\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:notequals=abc)%(raw)%(then)%(refname)%(else)equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'empty raw refs with %(if)' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob1 not empty\n+\trefs/myblobs/blob2 not empty\n+\trefs/myblobs/blob3 not empty\n+\trefs/myblobs/blob4 not empty\n+\trefs/myblobs/blob5 not empty\n+\trefs/myblobs/blob6 not empty\n+\trefs/myblobs/blob7 empty\n+\trefs/myblobs/blob8 empty\n+\trefs/myblobs/first not empty\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname) %(if)%(raw)%(then)not empty%(else)empty%(end)\" \\\n+\t\trefs/myblobs/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success '%(raw) with --python must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --python\n+'\n+\n+test_expect_success '%(raw) with --tcl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n+'\n+\n+test_expect_success '%(raw) with --perl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+'\n+\n+test_expect_success '%(raw) with --shell must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --shell\n+'\n+\n+test_expect_success '%(raw) with --shell and --sort=raw must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --sort=raw --shell\n+'\n+\n+test_expect_success '%(raw:size) with --shell' '\n+\tgit for-each-ref --format=\"%(raw:size)\" | while read line\n+\tdo\n+\t\techo \"'\\''$line'\\''\" >>expect\n+\tdone &&\n+\tgit for-each-ref --format=\"%(raw:size)\" --shell >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for-each-ref --format compare with cat-file --batch' '\n+\tgit rev-parse refs/mytrees/first | git cat-file --batch >expected &&\n+\tgit for-each-ref --format=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'set up multiple-sort tags' '\n \tfor when in 100000 200000\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"430212","messageId":"94addd4676af68352d53d1b2bbb0ec911d8e70ce.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 03/17] [GSOC] ref-filter: --format=%(raw) re-support --perl","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:11Z","receivedAt":"2021-07-15T15:40:35Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause the perl language can handle binary data correctly,\nadd the function perl_quote_buf_with_len(), which can specify\nthe length of the data and prevent the data from being truncated\nat '\\0' to help `--format=\"%(raw)\"` re-support `--perl`.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 4 ++--\n quote.c | 17 +++++++++++++++++\n quote.h | 1 +\n ref-filter.c | 15 +++++++++++----\n t/t6300-for-each-ref.sh | 19 +++++++++++++++++--\n 5 files changed, 48 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex cbb6f87d13f..6da899c6296 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -241,8 +241,8 @@ raw:size::\n \tThe raw data size of the object.\n \n Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n-`--perl` because such language may not support arbitrary binary data in their\n-string variable type.\n+because such language may not support arbitrary binary data in their string\n+variable type.\n \n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\ndiff --git a/quote.c b/quote.c\nindex 8a3a5e39eb1..26719d21d1e 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src)\n \tstrbuf_addch(sb, sq);\n }\n \n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len)\n+{\n+\tconst char sq = '\\'';\n+\tconst char bq = '\\\\';\n+\tconst char *c = src;\n+\tconst char *end = src + len;\n+\n+\tstrbuf_addch(sb, sq);\n+\twhile (c != end) {\n+\t\tif (*c == sq || *c == bq)\n+\t\t\tstrbuf_addch(sb, bq);\n+\t\tstrbuf_addch(sb, *c);\n+\t\tc++;\n+\t}\n+\tstrbuf_addch(sb, sq);\n+}\n+\n void python_quote_buf(struct strbuf *sb, const char *src)\n {\n \tconst char sq = '\\'';\ndiff --git a/quote.h b/quote.h\nindex 768cc6338e2..0fe69e264b0 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -94,6 +94,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne\n \n /* quoting as a string literal for other languages */\n void perl_quote_buf(struct strbuf *sb, const char *src);\n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len);\n void python_quote_buf(struct strbuf *sb, const char *src);\n void tcl_quote_buf(struct strbuf *sb, const char *src);\n void basic_regex_quote_buf(struct strbuf *sb, const char *src);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 506fbc3d691..ba9ab35d7ec 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -742,7 +742,10 @@ static void quote_formatting(struct strbuf *s, const char *str, size_t len, int\n \t\tsq_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PERL:\n-\t\tperl_quote_buf(s, str);\n+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tperl_quote_buf_with_len(s, str, len);\n+\t\telse\n+\t\t\tperl_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PYTHON:\n \t\tpython_quote_buf(s, str);\n@@ -1006,10 +1009,14 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n-\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n-\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\n+\t\tif ((format->quote_style == QUOTE_PYTHON ||\n+\t\t format->quote_style == QUOTE_SHELL ||\n+\t\t format->quote_style == QUOTE_TCL) &&\n+\t\t used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n \t\t\tdie(_(\"--format=%.*s cannot be used with\"\n-\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n+\t\t\t \"--python, --shell, --tcl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 18554f62d94..3d15d0a5360 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' '\n \ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n '\n \n-test_expect_success '%(raw) with --perl must fail' '\n-\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+test_expect_success '%(raw) with --perl' '\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob1 --perl | perl >actual &&\n+\tcmp blob1 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob3 --perl | perl >actual &&\n+\tcmp blob3 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob8 --perl | perl >actual &&\n+\tcmp blob8 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/first --perl | perl >actual &&\n+\tcmp one actual &&\n+\tgit cat-file tree refs/mytrees/first > expected &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/mytrees/first --perl | perl >actual &&\n+\tcmp expected actual\n '\n \n test_expect_success '%(raw) with --shell must fail' '\n-- \ngitgitgadget\n\n"},{"id":"430213","messageId":"45984f94bf3514d244f4541390a0ec4f8d3b8c51.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 04/17] [GSOC] ref-filter: use non-const ref_format in *_atom_parser()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:12Z","receivedAt":"2021-07-15T15:40:37Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse non-const ref_format in *_atom_parser(), which can help us\nmodify the members of ref_format in *_atom_parser().\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/tag.c | 2 +-\n ref-filter.c | 44 ++++++++++++++++++++++----------------------\n ref-filter.h | 4 ++--\n 3 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 82fcfc09824..452558ec957 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref,\n \t\t const struct object_id *oid, void *cb_data)\n {\n \tint flags;\n-\tconst struct ref_format *format = cb_data;\n+\tstruct ref_format *format = cb_data;\n \tflags = GPG_VERIFY_VERBOSE;\n \n \tif (format->format)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex ba9ab35d7ec..c8e561a3687 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -226,7 +226,7 @@ static int strbuf_addf_ret(struct strbuf *sb, int ret, const char *fmt, ...)\n \treturn ret;\n }\n \n-static int color_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int color_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *color_value, struct strbuf *err)\n {\n \tif (!color_value)\n@@ -264,7 +264,7 @@ static int refname_atom_parser_internal(struct refname_atom *atom, const char *a\n \treturn 0;\n }\n \n-static int remote_ref_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int remote_ref_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct string_list params = STRING_LIST_INIT_DUP;\n@@ -311,7 +311,7 @@ static int remote_ref_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objecttype_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -323,7 +323,7 @@ static int objecttype_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objectsize_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -343,7 +343,7 @@ static int objectsize_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int deltabase_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int deltabase_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -355,7 +355,7 @@ static int deltabase_atom_parser(const struct ref_format *format, struct used_at\n \treturn 0;\n }\n \n-static int body_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int body_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -364,7 +364,7 @@ static int body_atom_parser(const struct ref_format *format, struct used_atom *a\n \treturn 0;\n }\n \n-static int subject_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int subject_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -376,7 +376,7 @@ static int subject_atom_parser(const struct ref_format *format, struct used_atom\n \treturn 0;\n }\n \n-static int trailers_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int trailers_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tatom->u.contents.trailer_opts.no_divider = 1;\n@@ -402,7 +402,7 @@ static int trailers_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int contents_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int contents_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -430,7 +430,7 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int raw_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -442,7 +442,7 @@ static int raw_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int oid_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -461,7 +461,7 @@ static int oid_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int person_email_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int person_email_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -475,7 +475,7 @@ static int person_email_atom_parser(const struct ref_format *format, struct used\n \treturn 0;\n }\n \n-static int refname_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int refname_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \treturn refname_atom_parser_internal(&atom->u.refname, arg, atom->name, err);\n@@ -492,7 +492,7 @@ static align_type parse_align_position(const char *s)\n \treturn -1;\n }\n \n-static int align_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int align_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct align *align = &atom->u.align;\n@@ -544,7 +544,7 @@ static int align_atom_parser(const struct ref_format *format, struct used_atom *\n \treturn 0;\n }\n \n-static int if_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -559,7 +559,7 @@ static int if_atom_parser(const struct ref_format *format, struct used_atom *ato\n \treturn 0;\n }\n \n-static int head_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n \tatom->u.head = resolve_refdup(\"HEAD\", RESOLVE_REF_READING, NULL, NULL);\n@@ -570,7 +570,7 @@ static struct {\n \tconst char *name;\n \tinfo_source source;\n \tcmp_type cmp_type;\n-\tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n+\tint (*parser)(struct ref_format *format, struct used_atom *atom,\n \t\t const char *arg, struct strbuf *err);\n } valid_atom[] = {\n \t[ATOM_REFNAME] = { \"refname\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -649,7 +649,7 @@ struct atom_value {\n /*\n * Used to parse format string and sort specifiers\n */\n-static int parse_ref_filter_atom(const struct ref_format *format,\n+static int parse_ref_filter_atom(struct ref_format *format,\n \t\t\t\t const char *atom, const char *ep,\n \t\t\t\t struct strbuf *err)\n {\n@@ -2554,9 +2554,9 @@ static void append_literal(const char *cp, const char *ep, struct ref_formatting\n }\n \n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n-\t\t\t struct strbuf *final_buf,\n-\t\t\t struct strbuf *error_buf)\n+\t\t\t struct ref_format *format,\n+\t\t\t struct strbuf *final_buf,\n+\t\t\t struct strbuf *error_buf)\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n@@ -2601,7 +2601,7 @@ int format_ref_array_item(struct ref_array_item *info,\n }\n \n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format)\n+\t\t struct ref_format *format)\n {\n \tstruct ref_array_item *ref_item;\n \tstruct strbuf output = STRBUF_INIT;\ndiff --git a/ref-filter.h b/ref-filter.h\nindex baf72a71896..74fb423fc89 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);\n void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting, unsigned int mask, int on);\n /* Based on the given format and quote_style, fill the strbuf */\n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n+\t\t\t struct ref_format *format,\n \t\t\t struct strbuf *final_buf,\n \t\t\t struct strbuf *error_buf);\n /* Parse a single sort specifier and add it to the list */\n@@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void);\n * name must be a fully qualified refname.\n */\n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format);\n+\t\t struct ref_format *format);\n \n /*\n * Push a single ref onto the array; this can be used to construct your own\n-- \ngitgitgadget\n\n"},{"id":"430214","messageId":"c3378dbfaed6b8d62ef46d8528770efa8365fb1a.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 06/17] [GSOC] ref-filter: pass get_object() return value to their callers","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:14Z","receivedAt":"2021-07-15T15:40:38Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause in the refactor of `git cat-file --batch` later,\noid_object_info_extended() in get_object() will be used to obtain\nthe info of an object with it's oid. When the object cannot be\nobtained in the git repository, `cat-file --batch` expects to output\n\" missing\" and continue the next oid query instead of letting\nGit exit. In other error conditions, Git should exit normally. So we\ncan achieve this function by passing the return value of get_object().\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 17 +++++++++++------\n 1 file changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex af8216dcd5b..cfcea0e507e 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1821,6 +1821,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n {\n \tstruct object *obj;\n \tint i;\n+\tint ret;\n \tstruct object_info empty = OBJECT_INFO_INIT;\n \n \tCALLOC_ARRAY(ref->value, used_atom_cnt);\n@@ -1977,8 +1978,9 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \n \n \toi.oid = ref->objectname;\n-\tif (get_object(ref, 0, &obj, &oi, err))\n-\t\treturn -1;\n+\tret = get_object(ref, 0, &obj, &oi, err);\n+\tif (ret)\n+\t\treturn ret;\n \n \t/*\n \t * If there is no atom that wants to know about tagged\n@@ -2010,8 +2012,10 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,\n \t\t\t struct atom_value **v, struct strbuf *err)\n {\n \tif (!ref->value) {\n-\t\tif (populate_value(ref, err))\n-\t\t\treturn -1;\n+\t\tint ret = populate_value(ref, err);\n+\n+\t\tif (ret)\n+\t\t\treturn ret;\n \t\tfill_missing_values(ref->value);\n \t}\n \t*v = &ref->value[atom];\n@@ -2585,6 +2589,7 @@ int format_ref_array_item(struct ref_array_item *info,\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n+\tint ret;\n \n \tstate.quote_style = format->quote_style;\n \tpush_stack_element(&state.stack);\n@@ -2597,10 +2602,10 @@ int format_ref_array_item(struct ref_array_item *info,\n \t\tif (cp < sp)\n \t\t\tappend_literal(cp, sp, &state);\n \t\tpos = parse_ref_filter_atom(format, sp + 2, ep, error_buf);\n-\t\tif (pos < 0 || get_ref_atom_value(info, pos, &atomv, error_buf) ||\n+\t\tif (pos < 0 || (ret = get_ref_atom_value(info, pos, &atomv, error_buf)) ||\n \t\t atomv->handler(atomv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\n-\t\t\treturn -1;\n+\t\t\treturn ret ? ret : -1;\n \t\t}\n \t}\n \tif (*cp) {\n-- \ngitgitgadget\n\n"},{"id":"430215","messageId":"e5cf5541024fb1744d1b9e8417d2148129c53f8e.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 07/17] [GSOC] ref-filter: introduce free_ref_array_item_value() function","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:15Z","receivedAt":"2021-07-15T15:40:39Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nWhen we use ref_array_item which is not dynamically allocated and\nwant to free the space of its member \"value\" after the end of use,\nfree_array_item() does not meet our needs, because it tries to free\nref_array_item itself and its member \"symref\".\n\nIntroduce free_ref_array_item_value() for freeing ref_array_item value.\nIt will be called internally by free_array_item(), and it will help\n`cat-file --batch` free ref_array_item's value memory later.\n\nHelped-by: Junio C Hamano \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 11 ++++++++---\n ref-filter.h | 2 ++\n 2 files changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex cfcea0e507e..d70b295672f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2296,16 +2296,21 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \treturn 0;\n }\n \n-/* Free memory allocated for a ref_array_item */\n-static void free_array_item(struct ref_array_item *item)\n+void free_ref_array_item_value(struct ref_array_item *item)\n {\n-\tfree((char *)item->symref);\n \tif (item->value) {\n \t\tint i;\n \t\tfor (i = 0; i < used_atom_cnt; i++)\n \t\t\tfree((char *)item->value[i].s);\n \t\tfree(item->value);\n \t}\n+}\n+\n+/* Free memory allocated for a ref_array_item */\n+static void free_array_item(struct ref_array_item *item)\n+{\n+\tfree((char *)item->symref);\n+\tfree_ref_array_item_value(item);\n \tfree(item);\n }\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex c15dee8d6b9..44e6dc05ac2 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -111,6 +111,8 @@ struct ref_format {\n int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type);\n /* Clear all memory allocated to ref_array */\n void ref_array_clear(struct ref_array *array);\n+/* Free ref_array_item's value */\n+void free_ref_array_item_value(struct ref_array_item *item);\n /* Used to verify if the given format is correct and to parse out the used atoms */\n int verify_ref_format(struct ref_format *format);\n /* Sort the given ref_array as per the ref_sorting provided */\n-- \ngitgitgadget\n\n"},{"id":"430216","messageId":"6fb9cfdeab1910f7fc2faf2ba6b34efea4399a85.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 05/17] [GSOC] ref-filter: add %(rest) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:13Z","receivedAt":"2021-07-15T15:40:39Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nIn order to let \"cat-file --batch=%(rest)\" use the ref-filter\ninterface, add %(rest) atom for ref-filter. Introduce the\nreject_atom() to reject the atom %(rest) for \"git for-each-ref\",\n\"git branch\", \"git tag\" and \"git verify-tag\".\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 25 +++++++++++++++++++++++++\n ref-filter.h | 5 ++++-\n t/t3203-branch-output.sh | 4 ++++\n t/t6300-for-each-ref.sh | 4 ++++\n t/t7004-tag.sh | 4 ++++\n t/t7030-verify-tag.sh | 4 ++++\n 6 files changed, 45 insertions(+), 1 deletion(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex c8e561a3687..af8216dcd5b 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -157,6 +157,7 @@ enum atom_type {\n \tATOM_IF,\n \tATOM_THEN,\n \tATOM_ELSE,\n+\tATOM_REST,\n };\n \n /*\n@@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"%%(rest) does not take arguments\"));\n+\tformat->use_rest = 1;\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n@@ -615,6 +625,7 @@ static struct {\n \t[ATOM_IF] = { \"if\", SOURCE_NONE, FIELD_STR, if_atom_parser },\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n+\t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -989,6 +1000,11 @@ static const char *find_next(const char *cp)\n \treturn NULL;\n }\n \n+static int reject_atom(enum atom_type atom_type)\n+{\n+\treturn atom_type == ATOM_REST;\n+}\n+\n /*\n * Make sure the format string is well formed, and parse out\n * the used atoms.\n@@ -1009,6 +1025,8 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (reject_atom(used_atom[at].atom_type))\n+\t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n \t\t format->quote_style == QUOTE_SHELL ||\n@@ -1928,6 +1946,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_REST) {\n+\t\t\tif (ref->rest)\n+\t\t\t\tv->s = xstrdup(ref->rest);\n+\t\t\telse\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2145,6 +2169,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n+\tref->rest = NULL;\n \n \treturn ref;\n }\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 74fb423fc89..c15dee8d6b9 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -38,6 +38,7 @@ struct ref_sorting {\n \n struct ref_array_item {\n \tstruct object_id objectname;\n+\tconst char *rest;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n@@ -76,14 +77,16 @@ struct ref_format {\n \t * verify_ref_format() afterwards to finalize.\n \t */\n \tconst char *format;\n+\tconst char *rest;\n \tint quote_style;\n+\tint use_rest;\n \tint use_color;\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n };\n \n-#define REF_FORMAT_INIT { NULL, 0, -1 }\n+#define REF_FORMAT_INIT { .use_color = -1 }\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\ndiff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh\nindex 5325b9f67a0..6e94c6db7b5 100755\n--- a/t/t3203-branch-output.sh\n+++ b/t/t3203-branch-output.sh\n@@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git branch with --format=%(rest) must fail' '\n+\ttest_must_fail git branch --format=\"%(rest)\" >actual\n+'\n+\n test_expect_success 'worktree colors correct' '\n \tcat >expect <<-EOF &&\n \t* (HEAD detached from fromtag)\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 3d15d0a5360..0d2e062f791 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' '\n \ttest_cmp expect actual.clean\n '\n \n+test_expect_success 'basic atom: rest must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(rest)\" refs/heads/main\n+'\n+\n test_expect_success 'trailer parsing not fooled by --- line' '\n \tgit commit --allow-empty -F - <<-\\EOF &&\n \tthis is the subject\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 2f72c5c6883..082be85dffc 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git tag -l with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git tag -l --format=\"%(rest)\" \"v1*\"\n+'\n+\n test_expect_success \"set up color tests\" '\n \techo \"v1.0\" >expect.color &&\n \techo \"v1.0\" >expect.bare &&\ndiff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh\nindex 3cefde9602b..10faa645157 100755\n--- a/t/t7030-verify-tag.sh\n+++ b/t/t7030-verify-tag.sh\n@@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success GPG 'verifying tag with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git verify-tag --format=\"%(rest)\" \"fourth-signed\"\n+'\n+\n test_expect_success GPG 'verifying a forged tag with --format should fail silently' '\n \ttest_must_fail git verify-tag --format=\"tagname : %(tag)\" $(cat forged1.tag) >actual-forged &&\n \ttest_must_be_empty actual-forged\n-- \ngitgitgadget\n\n"},{"id":"430217","messageId":"7058f47d41d74ad874b9e74f469dc473fd0d3413.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 09/17] [GSOC] ref-filter: modify the error message and value in get_object","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:17Z","receivedAt":"2021-07-15T15:40:40Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nLet get_object() return 1 and print \" missing\" instead\nof returning -1 and printing \"missing object for \"\nif oid_object_info_extended() unable to find the data corresponding\nto oid. When `cat-file --batch` use ref-filter logic later it can\nhelp `format_ref_array_item()` just report that the object is missing\nwithout letting Git exit.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 4 ++--\n t/t6301-for-each-ref-errors.sh | 2 +-\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 27199ba40f5..b4f41fec871 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1762,8 +1762,8 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t}\n \tif (oid_object_info_extended(the_repository, &oi->oid, &oi->info,\n \t\t\t\t OBJECT_INFO_LOOKUP_REPLACE))\n-\t\treturn strbuf_addf_ret(err, -1, _(\"missing object %s for %s\"),\n-\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n+\t\treturn strbuf_addf_ret(err, 1, _(\"%s missing\"),\n+\t\t\t\t oid_to_hex(&oi->oid));\n \tif (oi->info.disk_sizep && oi->disk_size < 0)\n \t\tBUG(\"Object size is less than zero.\");\n \ndiff --git a/t/t6301-for-each-ref-errors.sh b/t/t6301-for-each-ref-errors.sh\nindex 40edf9dab53..3553f84a00c 100755\n--- a/t/t6301-for-each-ref-errors.sh\n+++ b/t/t6301-for-each-ref-errors.sh\n@@ -41,7 +41,7 @@ test_expect_success 'Missing objects are reported correctly' '\n \tr=refs/heads/missing &&\n \techo $MISSING >.git/$r &&\n \ttest_when_finished \"rm -f .git/$r\" &&\n-\techo \"fatal: missing object $MISSING for $r\" >missing-err &&\n+\techo \"fatal: $MISSING missing\" >missing-err &&\n \ttest_must_fail git for-each-ref 2>err &&\n \ttest_cmp missing-err err &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"430218","messageId":"3af0def894c44cecd9f621daf8708a232bc0a143.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 10/17] [GSOC] cat-file: add has_object_file() check","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:18Z","receivedAt":"2021-07-15T15:40:41Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse `has_object_file()` in `batch_one_object()` to check\nwhether the input object exists. This can help us reject\nthe missing oid when we let `cat-file --batch` use ref-filter\nlogic later.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 243fe6844bc..59a86412fd0 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -428,6 +428,13 @@ static void batch_one_object(const char *obj_name,\n \t\treturn;\n \t}\n \n+\tif (!has_object_file(&data->oid)) {\n+\t\tprintf(\"%s missing\\n\",\n+\t\t obj_name ? obj_name : oid_to_hex(&data->oid));\n+\t\tfflush(stdout);\n+\t\treturn;\n+\t}\n+\n \tbatch_object_write(obj_name, scratch, opt, data);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"430219","messageId":"c445fa6520b296673b5e559dfb4d1c678e722b0d.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 11/17] [GSOC] cat-file: change batch_objects parameter name","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:19Z","receivedAt":"2021-07-15T15:40:42Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause later cat-file reuses ref-filter logic that will add\nparameter \"const struct option *options\" to batch_objects(),\nthe two synonymous parameters of \"opt\" and \"options\" may\nconfuse readers, so change batch_options parameter of\nbatch_objects() from \"opt\" to \"batch\".\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 59a86412fd0..41d407638d5 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -495,7 +495,7 @@ static int batch_unordered_packed(const struct object_id *oid,\n \treturn batch_unordered_object(oid, data);\n }\n \n-static int batch_objects(struct batch_options *opt)\n+static int batch_objects(struct batch_options *batch)\n {\n \tstruct strbuf input = STRBUF_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n@@ -503,8 +503,8 @@ static int batch_objects(struct batch_options *opt)\n \tint save_warning;\n \tint retval = 0;\n \n-\tif (!opt->format)\n-\t\topt->format = \"%(objectname) %(objecttype) %(objectsize)\";\n+\tif (!batch->format)\n+\t\tbatch->format = \"%(objectname) %(objecttype) %(objectsize)\";\n \n \t/*\n \t * Expand once with our special mark_query flag, which will prime the\n@@ -513,20 +513,20 @@ static int batch_objects(struct batch_options *opt)\n \t */\n \tmemset(&data, 0, sizeof(data));\n \tdata.mark_query = 1;\n-\tstrbuf_expand(&output, opt->format, expand_format, &data);\n+\tstrbuf_expand(&output, batch->format, expand_format, &data);\n \tdata.mark_query = 0;\n \tstrbuf_release(&output);\n-\tif (opt->cmdmode)\n+\tif (batch->cmdmode)\n \t\tdata.split_on_whitespace = 1;\n \n \t/*\n \t * If we are printing out the object, then always fill in the type,\n \t * since we will want to decide whether or not to stream.\n \t */\n-\tif (opt->print_contents)\n+\tif (batch->print_contents)\n \t\tdata.info.typep = &data.type;\n \n-\tif (opt->all_objects) {\n+\tif (batch->all_objects) {\n \t\tstruct object_cb_data cb;\n \t\tstruct object_info empty = OBJECT_INFO_INIT;\n \n@@ -536,11 +536,11 @@ static int batch_objects(struct batch_options *opt)\n \t\tif (has_promisor_remote())\n \t\t\twarning(\"This repository uses promisor remotes. Some objects may not be loaded.\");\n \n-\t\tcb.opt = opt;\n+\t\tcb.opt = batch;\n \t\tcb.expand = &data;\n \t\tcb.scratch = &output;\n \n-\t\tif (opt->unordered) {\n+\t\tif (batch->unordered) {\n \t\t\tstruct oidset seen = OIDSET_INIT;\n \n \t\t\tcb.seen = &seen;\n@@ -590,7 +590,7 @@ static int batch_objects(struct batch_options *opt)\n \t\t\tdata.rest = p;\n \t\t}\n \n-\t\tbatch_one_object(input.buf, &output, opt, &data);\n+\t\tbatch_one_object(input.buf, &output, batch, &data);\n \t}\n \n \tstrbuf_release(&input);\n-- \ngitgitgadget\n\n"},{"id":"430220","messageId":"ae3a7816013fbe1568e0bddff211b43ae41625bf.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 12/17] [GSOC] cat-file: create p1006-cat-file.sh","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:20Z","receivedAt":"2021-07-15T15:40:42Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nCreate p1006-cat-file.sh to provide performance testing for\n`git cat-file --batch` and `git cat-file --batch-check`. This\nwill help us compare the performance changes after we let\ncat-file reuse the ref-filter logic.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n t/perf/p1006-cat-file.sh | 28 ++++++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n create mode 100755 t/perf/p1006-cat-file.sh\n\ndiff --git a/t/perf/p1006-cat-file.sh b/t/perf/p1006-cat-file.sh\nnew file mode 100755\nindex 00000000000..b84ac31f9cc\n--- /dev/null\n+++ b/t/perf/p1006-cat-file.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='Basic sort performance tests'\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+test_expect_success 'setup' '\n+\tgit rev-list --all >rla\n+'\n+\n+test_perf 'cat-file --batch-check' '\n+\tgit cat-file --batch-check \n\nAdd `cat_file_mode` member to struct `ref_format`. When\n`cat-file --batch` use ref-filter logic later, it can help us\nreject atoms in verify_ref_format() which cat-file cannot use,\ne.g. `%(refname)`, `%(push)`, `%(upstream)\"...\n\nAdd batch_test_atom() to t/t1006-cat-file.sh and add check\nfor cat-file --batch, this can help us clearly show which\natoms cat-file accepts and which atoms it rejects.\n\nHelped-by: Eric Sunshine \nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Christian Couder \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 23 ++++-\n ref-filter.h | 1 +\n t/t1006-cat-file.sh | 226 ++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 246 insertions(+), 4 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex d70b295672f..27199ba40f5 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1000,9 +1000,24 @@ static const char *find_next(const char *cp)\n \treturn NULL;\n }\n \n-static int reject_atom(enum atom_type atom_type)\n-{\n-\treturn atom_type == ATOM_REST;\n+static int reject_atom(int cat_file_mode, enum atom_type atom_type)\n+{\n+\tif (!cat_file_mode)\n+\t\treturn atom_type == ATOM_REST;\n+\n+\t/* cat_file_mode */\n+\tswitch (atom_type) {\n+\tcase ATOM_FLAG:\n+\tcase ATOM_HEAD:\n+\tcase ATOM_PUSH:\n+\tcase ATOM_REFNAME:\n+\tcase ATOM_SYMREF:\n+\tcase ATOM_UPSTREAM:\n+\tcase ATOM_WORKTREEPATH:\n+\t\treturn 1;\n+\tdefault:\n+\t\treturn 0;\n+\t}\n }\n \n /*\n@@ -1025,7 +1040,7 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n-\t\tif (reject_atom(used_atom[at].atom_type))\n+\t\tif (reject_atom(format->cat_file_mode, used_atom[at].atom_type))\n \t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n \t\tif ((format->quote_style == QUOTE_PYTHON ||\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 44e6dc05ac2..053980a6a42 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -78,6 +78,7 @@ struct ref_format {\n \t */\n \tconst char *format;\n \tconst char *rest;\n+\tint cat_file_mode;\n \tint quote_style;\n \tint use_rest;\n \tint use_color;\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 18b3779ccb6..95d760652eb 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -608,4 +608,230 @@ test_expect_success 'cat-file --batch=\"batman\" with --batch-all-objects will wor\n \tcmp expect actual\n '\n \n+. \"$TEST_DIRECTORY\"/lib-gpg.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n+\n+test_expect_success 'cat-file --batch|--batch-check setup' '\n+\techo 1>blob1 &&\n+\tprintf \"a\\0b\\0\\c\" >blob2 &&\n+\tgit add blob1 blob2 &&\n+\tgit commit -m \"Commit Message\" &&\n+\tgit branch -M main &&\n+\tgit tag -a -m \"v0.0.0\" testtag &&\n+\tgit update-ref refs/myblobs/blob1 HEAD:blob1 &&\n+\tgit update-ref refs/myblobs/blob2 HEAD:blob2 &&\n+\tgit update-ref refs/mytrees/tree1 HEAD^{tree}\n+'\n+\n+batch_test_atom() {\n+\tif test \"$3\" = \"fail\"\n+\tthen\n+\t\ttest_expect_${4:-success} $PREREQ \"basic atom: $1 $2 must fail\" \"\n+\t\t\ttest_must_fail git cat-file --batch-check='$2' >bad <<-EOF\n+\t\t\t$1\n+\t\t\tEOF\n+\t\t\"\n+\telse\n+\t\ttest_expect_${4:-success} $PREREQ \"basic atom: $1 $2\" \"\n+\t\t\tgit for-each-ref --format='$2' $1 >expected &&\n+\t\t\tgit cat-file --batch-check='$2' >actual <<-EOF &&\n+\t\t\t$1\n+\t\t\tEOF\n+\t\t\tsanitize_pgp actual.clean &&\n+\t\t\tcmp expected actual.clean\n+\t\t\"\n+\tfi\n+}\n+\n+batch_test_atom refs/heads/main '%(refname)' fail\n+batch_test_atom refs/heads/main '%(refname:)' fail\n+batch_test_atom refs/heads/main '%(refname:short)' fail\n+batch_test_atom refs/heads/main '%(refname:lstrip=1)' fail\n+batch_test_atom refs/heads/main '%(refname:lstrip=2)' fail\n+batch_test_atom refs/heads/main '%(refname:lstrip=-1)' fail\n+batch_test_atom refs/heads/main '%(refname:lstrip=-2)' fail\n+batch_test_atom refs/heads/main '%(refname:rstrip=1)' fail\n+batch_test_atom refs/heads/main '%(refname:rstrip=2)' fail\n+batch_test_atom refs/heads/main '%(refname:rstrip=-1)' fail\n+batch_test_atom refs/heads/main '%(refname:rstrip=-2)' fail\n+batch_test_atom refs/heads/main '%(refname:strip=1)' fail\n+batch_test_atom refs/heads/main '%(refname:strip=2)' fail\n+batch_test_atom refs/heads/main '%(refname:strip=-1)' fail\n+batch_test_atom refs/heads/main '%(refname:strip=-2)' fail\n+batch_test_atom refs/heads/main '%(upstream)' fail\n+batch_test_atom refs/heads/main '%(upstream:short)' fail\n+batch_test_atom refs/heads/main '%(upstream:lstrip=2)' fail\n+batch_test_atom refs/heads/main '%(upstream:lstrip=-2)' fail\n+batch_test_atom refs/heads/main '%(upstream:rstrip=2)' fail\n+batch_test_atom refs/heads/main '%(upstream:rstrip=-2)' fail\n+batch_test_atom refs/heads/main '%(upstream:strip=2)' fail\n+batch_test_atom refs/heads/main '%(upstream:strip=-2)' fail\n+batch_test_atom refs/heads/main '%(push)' fail\n+batch_test_atom refs/heads/main '%(push:short)' fail\n+batch_test_atom refs/heads/main '%(push:lstrip=1)' fail\n+batch_test_atom refs/heads/main '%(push:lstrip=-1)' fail\n+batch_test_atom refs/heads/main '%(push:rstrip=1)' fail\n+batch_test_atom refs/heads/main '%(push:rstrip=-1)' fail\n+batch_test_atom refs/heads/main '%(push:strip=1)' fail\n+batch_test_atom refs/heads/main '%(push:strip=-1)' fail\n+batch_test_atom refs/heads/main '%(objecttype)'\n+batch_test_atom refs/heads/main '%(objectsize)'\n+batch_test_atom refs/heads/main '%(objectsize:disk)'\n+batch_test_atom refs/heads/main '%(deltabase)'\n+batch_test_atom refs/heads/main '%(objectname)'\n+batch_test_atom refs/heads/main '%(objectname:short)' fail\n+batch_test_atom refs/heads/main '%(objectname:short=1)' fail\n+batch_test_atom refs/heads/main '%(objectname:short=10)' fail\n+batch_test_atom refs/heads/main '%(tree)' fail\n+batch_test_atom refs/heads/main '%(tree:short)' fail\n+batch_test_atom refs/heads/main '%(tree:short=1)' fail\n+batch_test_atom refs/heads/main '%(tree:short=10)' fail\n+batch_test_atom refs/heads/main '%(parent)' fail\n+batch_test_atom refs/heads/main '%(parent:short)' fail\n+batch_test_atom refs/heads/main '%(parent:short=1)' fail\n+batch_test_atom refs/heads/main '%(parent:short=10)' fail\n+batch_test_atom refs/heads/main '%(numparent)' fail\n+batch_test_atom refs/heads/main '%(object)' fail\n+batch_test_atom refs/heads/main '%(type)' fail\n+batch_test_atom refs/heads/main '%(raw)' fail\n+batch_test_atom refs/heads/main '%(*objectname)' fail\n+batch_test_atom refs/heads/main '%(*objecttype)' fail\n+batch_test_atom refs/heads/main '%(author)' fail\n+batch_test_atom refs/heads/main '%(authorname)' fail\n+batch_test_atom refs/heads/main '%(authoremail)' fail\n+batch_test_atom refs/heads/main '%(authoremail:trim)' fail\n+batch_test_atom refs/heads/main '%(authoremail:localpart)' fail\n+batch_test_atom refs/heads/main '%(authordate)' fail\n+batch_test_atom refs/heads/main '%(committer)' fail\n+batch_test_atom refs/heads/main '%(committername)' fail\n+batch_test_atom refs/heads/main '%(committeremail)' fail\n+batch_test_atom refs/heads/main '%(committeremail:trim)' fail\n+batch_test_atom refs/heads/main '%(committeremail:localpart)' fail\n+batch_test_atom refs/heads/main '%(committerdate)' fail\n+batch_test_atom refs/heads/main '%(tag)' fail\n+batch_test_atom refs/heads/main '%(tagger)' fail\n+batch_test_atom refs/heads/main '%(taggername)' fail\n+batch_test_atom refs/heads/main '%(taggeremail)' fail\n+batch_test_atom refs/heads/main '%(taggeremail:trim)' fail\n+batch_test_atom refs/heads/main '%(taggeremail:localpart)' fail\n+batch_test_atom refs/heads/main '%(taggerdate)' fail\n+batch_test_atom refs/heads/main '%(creator)' fail\n+batch_test_atom refs/heads/main '%(creatordate)' fail\n+batch_test_atom refs/heads/main '%(subject)' fail\n+batch_test_atom refs/heads/main '%(subject:sanitize)' fail\n+batch_test_atom refs/heads/main '%(contents:subject)' fail\n+batch_test_atom refs/heads/main '%(body)' fail\n+batch_test_atom refs/heads/main '%(contents:body)' fail\n+batch_test_atom refs/heads/main '%(contents:signature)' fail\n+batch_test_atom refs/heads/main '%(contents)' fail\n+batch_test_atom refs/heads/main '%(HEAD)' fail\n+batch_test_atom refs/heads/main '%(upstream:track)' fail\n+batch_test_atom refs/heads/main '%(upstream:trackshort)' fail\n+batch_test_atom refs/heads/main '%(upstream:track,nobracket)' fail\n+batch_test_atom refs/heads/main '%(upstream:nobracket,track)' fail\n+batch_test_atom refs/heads/main '%(push:track)' fail\n+batch_test_atom refs/heads/main '%(push:trackshort)' fail\n+batch_test_atom refs/heads/main '%(worktreepath)' fail\n+batch_test_atom refs/heads/main '%(symref)' fail\n+batch_test_atom refs/heads/main '%(flag)' fail\n+\n+batch_test_atom refs/tags/testtag '%(refname)' fail\n+batch_test_atom refs/tags/testtag '%(refname:short)' fail\n+batch_test_atom refs/tags/testtag '%(upstream)' fail\n+batch_test_atom refs/tags/testtag '%(push)' fail\n+batch_test_atom refs/tags/testtag '%(objecttype)'\n+batch_test_atom refs/tags/testtag '%(objectsize)'\n+batch_test_atom refs/tags/testtag '%(objectsize:disk)'\n+batch_test_atom refs/tags/testtag '%(*objectsize:disk)' fail\n+batch_test_atom refs/tags/testtag '%(deltabase)'\n+batch_test_atom refs/tags/testtag '%(*deltabase)' fail\n+batch_test_atom refs/tags/testtag '%(objectname)'\n+batch_test_atom refs/tags/testtag '%(objectname:short)' fail\n+batch_test_atom refs/tags/testtag '%(tree)' fail\n+batch_test_atom refs/tags/testtag '%(tree:short)' fail\n+batch_test_atom refs/tags/testtag '%(tree:short=1)' fail\n+batch_test_atom refs/tags/testtag '%(tree:short=10)' fail\n+batch_test_atom refs/tags/testtag '%(parent)' fail\n+batch_test_atom refs/tags/testtag '%(parent:short)' fail\n+batch_test_atom refs/tags/testtag '%(parent:short=1)' fail\n+batch_test_atom refs/tags/testtag '%(parent:short=10)' fail\n+batch_test_atom refs/tags/testtag '%(numparent)' fail\n+batch_test_atom refs/tags/testtag '%(object)' fail\n+batch_test_atom refs/tags/testtag '%(type)' fail\n+batch_test_atom refs/tags/testtag '%(*objectname)' fail\n+batch_test_atom refs/tags/testtag '%(*objecttype)' fail\n+batch_test_atom refs/tags/testtag '%(author)' fail\n+batch_test_atom refs/tags/testtag '%(authorname)' fail\n+batch_test_atom refs/tags/testtag '%(authoremail)' fail\n+batch_test_atom refs/tags/testtag '%(authoremail:trim)' fail\n+batch_test_atom refs/tags/testtag '%(authoremail:localpart)' fail\n+batch_test_atom refs/tags/testtag '%(authordate)' fail\n+batch_test_atom refs/tags/testtag '%(committer)' fail\n+batch_test_atom refs/tags/testtag '%(committername)' fail\n+batch_test_atom refs/tags/testtag '%(committeremail)' fail\n+batch_test_atom refs/tags/testtag '%(committeremail:trim)' fail\n+batch_test_atom refs/tags/testtag '%(committeremail:localpart)' fail\n+batch_test_atom refs/tags/testtag '%(committerdate)' fail\n+batch_test_atom refs/tags/testtag '%(tag)' fail\n+batch_test_atom refs/tags/testtag '%(tagger)' fail\n+batch_test_atom refs/tags/testtag '%(taggername)' fail\n+batch_test_atom refs/tags/testtag '%(taggeremail)' fail\n+batch_test_atom refs/tags/testtag '%(taggeremail:trim)' fail\n+batch_test_atom refs/tags/testtag '%(taggeremail:localpart)' fail\n+batch_test_atom refs/tags/testtag '%(taggerdate)' fail\n+batch_test_atom refs/tags/testtag '%(creator)' fail\n+batch_test_atom refs/tags/testtag '%(creatordate)' fail\n+batch_test_atom refs/tags/testtag '%(subject)' fail\n+batch_test_atom refs/tags/testtag '%(subject:sanitize)' fail\n+batch_test_atom refs/tags/testtag '%(contents:subject)' fail\n+batch_test_atom refs/tags/testtag '%(body)' fail\n+batch_test_atom refs/tags/testtag '%(contents:body)' fail\n+batch_test_atom refs/tags/testtag '%(contents:signature)' fail\n+batch_test_atom refs/tags/testtag '%(contents)' fail\n+batch_test_atom refs/tags/testtag '%(HEAD)' fail\n+\n+batch_test_atom refs/myblobs/blob1 '%(refname)' fail\n+batch_test_atom refs/myblobs/blob1 '%(upstream)' fail\n+batch_test_atom refs/myblobs/blob1 '%(push)' fail\n+batch_test_atom refs/myblobs/blob1 '%(HEAD)' fail\n+\n+batch_test_atom refs/myblobs/blob1 '%(objectname)'\n+batch_test_atom refs/myblobs/blob1 '%(objecttype)'\n+batch_test_atom refs/myblobs/blob1 '%(objectsize)'\n+batch_test_atom refs/myblobs/blob1 '%(objectsize:disk)'\n+batch_test_atom refs/myblobs/blob1 '%(deltabase)'\n+\n+batch_test_atom refs/myblobs/blob1 '%(contents)' fail\n+batch_test_atom refs/myblobs/blob2 '%(contents)' fail\n+\n+batch_test_atom refs/myblobs/blob1 '%(raw)' fail\n+batch_test_atom refs/myblobs/blob2 '%(raw)' fail\n+batch_test_atom refs/mytrees/tree1 '%(raw)' fail\n+\n+batch_test_atom refs/myblobs/blob1 '%(raw:size)' fail\n+batch_test_atom refs/myblobs/blob2 '%(raw:size)' fail\n+batch_test_atom refs/mytrees/tree1 '%(raw:size)' fail\n+\n+batch_test_atom refs/myblobs/blob1 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)' fail\n+batch_test_atom refs/myblobs/blob2 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)' fail\n+batch_test_atom refs/mytrees/tree1 '%(if:equals=tree)%(objecttype)%(then)tree%(else)not tree%(end)' fail\n+\n+batch_test_atom refs/heads/main '%(align:60) objectname is %(objectname)%(end)|%(objectname)' fail\n+batch_test_atom refs/heads/main '%(align:left,60) objectname is %(objectname)%(end)|%(objectname)' fail\n+batch_test_atom refs/heads/main '%(align:middle,60) objectname is %(objectname)%(end)|%(objectname)' fail\n+batch_test_atom refs/heads/main '%(align:60,right) objectname is %(objectname)%(end)|%(objectname)' fail\n+\n+batch_test_atom refs/heads/main 'VALID'\n+batch_test_atom refs/heads/main '%(INVALID)' fail\n+batch_test_atom refs/heads/main '%(authordate:INVALID)' fail\n+\n+batch_test_atom refs/heads/main '%(objectname) %(objecttype) %(objectsize)\n+%(raw)' fail\n+batch_test_atom refs/tags/testtag '%(objectname) %(objecttype) %(objectsize)\n+%(raw)' fail\n+batch_test_atom refs/myblobs/blob1 '%(objectname) %(objecttype) %(objectsize)\n+%(raw)' fail\n+batch_test_atom refs/myblobs/blob2 '%(objectname) %(objecttype) %(objectsize)\n+%(raw)' fail\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"430222","messageId":"ed556e5f31e6398d1733cf9639c454b04978ab3a.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 15/17] [GSOC] cat-file: re-implement --textconv, --filters options","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:23Z","receivedAt":"2021-07-15T15:40:44Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAfter cat-file reuses the ref-filter logic, we re-implement the\nfunctions of --textconv and --filters options.\n\nAdd members `cat_file_cmdmode` to struct `ref_array_item`,\nso that struct `batch_option` member `cmdmode` will be passed\nto ref-filter, and then ref-filter will take use of it to filter\nthe content of the object in get_object().\n\nUse `actual_oi` to record the real expand_data: it may point to the\noriginal `oi` or the `act_oi` processed by `textconv_object()` or\n`convert_to_working_tree()`. `grab_values()` will grab the contents\nof `actual_oi` and `grab_common_values()` to grab the contents of origin\n`oi`, this ensures that `%(objectsize)` still uses the size of the\nunfiltered data.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 2 +-\n ref-filter.c | 35 +++++++++++++++++++++++++++++++++--\n ref-filter.h | 1 +\n 3 files changed, 35 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex dc604a9879d..3a6153e778f 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -217,7 +217,7 @@ static void batch_object_write(const char *obj_name,\n \t\t\t struct expand_data *data)\n {\n \tint ret;\n-\tstruct ref_array_item item = { data->oid, data->rest };\n+\tstruct ref_array_item item = { data->oid, data->rest, opt->cmdmode };\n \n \tstrbuf_reset(scratch);\n \tstrbuf_reset(err);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex b4f41fec871..91e26c9aba3 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1,3 +1,4 @@\n+#define USE_THE_INDEX_COMPATIBILITY_MACROS\n #include \"builtin.h\"\n #include \"cache.h\"\n #include \"parse-options.h\"\n@@ -1755,6 +1756,9 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n {\n \t/* parse_object_buffer() will set eaten to 0 if free() will be needed */\n \tint eaten = 1;\n+\tstruct expand_data *actual_oi = oi;\n+\tstruct expand_data act_oi = {0};\n+\n \tif (oi->info.contentp) {\n \t\t/* We need to know that to use parse_object_buffer properly */\n \t\toi->info.sizep = &oi->size;\n@@ -1768,19 +1772,45 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\tBUG(\"Object size is less than zero.\");\n \n \tif (oi->info.contentp) {\n-\t\t*obj = parse_object_buffer(the_repository, &oi->oid, oi->type, oi->size, oi->content, &eaten);\n+\t\tif ((ref->cat_file_cmdmode == 'c' || ref->cat_file_cmdmode == 'w') && !ref->rest)\n+\t\t\treturn strbuf_addf_ret(err, -1, _(\"missing path for '%s'\"),\n+\t\t\t\t\t oid_to_hex(&act_oi.oid));\n+\t\tif (oi->type == OBJ_BLOB) {\n+\t\t\tif (ref->cat_file_cmdmode == 'c') {\n+\t\t\t\tact_oi = *oi;\n+\t\t\t\tif (textconv_object(the_repository,\n+\t\t\t\t\t\t ref->rest, 0100644, &act_oi.oid,\n+\t\t\t\t\t\t 1, (char **)(&act_oi.content), &act_oi.size))\n+\t\t\t\t\tactual_oi = &act_oi;\n+\t\t\t} else if (ref->cat_file_cmdmode == 'w') {\n+\t\t\t\tstruct strbuf strbuf = STRBUF_INIT;\n+\t\t\t\tstruct checkout_metadata meta;\n+\t\t\t\tact_oi = *oi;\n+\n+\t\t\t\tinit_checkout_metadata(&meta, NULL, NULL, &act_oi.oid);\n+\t\t\t\tif (!convert_to_working_tree(&the_index, ref->rest, act_oi.content, act_oi.size, &strbuf, &meta))\n+\t\t\t\t\tdie(\"could not convert '%s' %s\",\n+\t\t\t\t\t oid_to_hex(&oi->oid), ref->rest);\n+\t\t\t\tact_oi.size = strbuf.len;\n+\t\t\t\tact_oi.content = strbuf_detach(&strbuf, NULL);\n+\t\t\t\tactual_oi = &act_oi;\n+\t\t\t}\n+\t\t}\n+\t\t*obj = parse_object_buffer(the_repository, &actual_oi->oid, actual_oi->type, actual_oi->size, actual_oi->content, &eaten);\n \t\tif (!*obj) {\n \t\t\tif (!eaten)\n \t\t\t\tfree(oi->content);\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi);\n+\t\tgrab_values(ref->value, deref, *obj, actual_oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n \tif (!eaten)\n \t\tfree(oi->content);\n+\tif (actual_oi != oi)\n+\t\tfree(actual_oi->content);\n \treturn 0;\n }\n \n@@ -2189,6 +2219,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n \tref->rest = NULL;\n+\tref->cat_file_cmdmode = 0;\n \n \treturn ref;\n }\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 053980a6a42..a93d5e4dd61 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -39,6 +39,7 @@ struct ref_sorting {\n struct ref_array_item {\n \tstruct object_id objectname;\n \tconst char *rest;\n+\tint cat_file_cmdmode;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n-- \ngitgitgadget\n\n"},{"id":"430223","messageId":"fd3901dfee6bc4046a83011ab74ad6292dfe3e85.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 14/17] [GSOC] cat-file: reuse err buf in batch_object_write()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:22Z","receivedAt":"2021-07-15T15:40:46Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nReuse the `err` buffer in batch_object_write(), as the\nbuffer `scratch` does. This will reduce the overhead\nof multiple allocations of memory of the err buffer.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 22 ++++++++++++++--------\n 1 file changed, 14 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5b163551fc6..dc604a9879d 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -212,35 +212,36 @@ static void batch_write(struct batch_options *opt, const void *data, int len)\n \n static void batch_object_write(const char *obj_name,\n \t\t\t struct strbuf *scratch,\n+\t\t\t struct strbuf *err,\n \t\t\t struct batch_options *opt,\n \t\t\t struct expand_data *data)\n {\n \tint ret;\n-\tstruct strbuf err = STRBUF_INIT;\n \tstruct ref_array_item item = { data->oid, data->rest };\n \n \tstrbuf_reset(scratch);\n+\tstrbuf_reset(err);\n \n-\tret = format_ref_array_item(&item, &opt->format, scratch, &err);\n+\tret = format_ref_array_item(&item, &opt->format, scratch, err);\n \tif (ret < 0)\n-\t\tdie(\"%s\\n\", err.buf);\n+\t\tdie(\"%s\\n\", err->buf);\n \tif (ret) {\n \t\t/* ret > 0 means when the object corresponding to oid\n \t\t * cannot be found in format_ref_array_item(), we only print\n \t\t * the error message.\n \t\t */\n-\t\tprintf(\"%s\\n\", err.buf);\n+\t\tprintf(\"%s\\n\", err->buf);\n \t\tfflush(stdout);\n \t} else {\n \t\tstrbuf_addch(scratch, '\\n');\n \t\tbatch_write(opt, scratch->buf, scratch->len);\n \t}\n \tfree_ref_array_item_value(&item);\n-\tstrbuf_release(&err);\n }\n \n static void batch_one_object(const char *obj_name,\n \t\t\t struct strbuf *scratch,\n+\t\t\t struct strbuf *err,\n \t\t\t struct batch_options *opt,\n \t\t\t struct expand_data *data)\n {\n@@ -294,7 +295,7 @@ static void batch_one_object(const char *obj_name,\n \t\treturn;\n \t}\n \n-\tbatch_object_write(obj_name, scratch, opt, data);\n+\tbatch_object_write(obj_name, scratch, err, opt, data);\n }\n \n struct object_cb_data {\n@@ -302,13 +303,14 @@ struct object_cb_data {\n \tstruct expand_data *expand;\n \tstruct oidset *seen;\n \tstruct strbuf *scratch;\n+\tstruct strbuf *err;\n };\n \n static int batch_object_cb(const struct object_id *oid, void *vdata)\n {\n \tstruct object_cb_data *data = vdata;\n \toidcpy(&data->expand->oid, oid);\n-\tbatch_object_write(NULL, data->scratch, data->opt, data->expand);\n+\tbatch_object_write(NULL, data->scratch, data->err, data->opt, data->expand);\n \treturn 0;\n }\n \n@@ -364,6 +366,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n {\n \tstruct strbuf input = STRBUF_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n+\tstruct strbuf err = STRBUF_INIT;\n \tstruct strbuf format = STRBUF_INIT;\n \tstruct expand_data data;\n \tint save_warning;\n@@ -392,6 +395,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n \t\tcb.opt = batch;\n \t\tcb.expand = &data;\n \t\tcb.scratch = &output;\n+\t\tcb.err = &err;\n \n \t\tif (batch->unordered) {\n \t\t\tstruct oidset seen = OIDSET_INIT;\n@@ -416,6 +420,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n \n \t\tstrbuf_release(&format);\n \t\tstrbuf_release(&output);\n+\t\tstrbuf_release(&err);\n \t\treturn 0;\n \t}\n \n@@ -444,12 +449,13 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n \t\t\tdata.rest = p;\n \t\t}\n \n-\t\tbatch_one_object(input.buf, &output, batch, &data);\n+\t\tbatch_one_object(input.buf, &output, &err, batch, &data);\n \t}\n \n \tstrbuf_release(&format);\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n+\tstrbuf_release(&err);\n \twarn_on_object_refname_ambiguity = save_warning;\n \treturn retval;\n }\n-- \ngitgitgadget\n\n"},{"id":"430224","messageId":"8b26c9e745523ab7b1ca07065d0c338fef9c4293.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 13/17] [GSOC] cat-file: reuse ref-filter logic","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:21Z","receivedAt":"2021-07-15T15:40:46Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nIn order to let cat-file use ref-filter logic, let's do the\nfollowing:\n\n1. Change the type of member `format` in struct `batch_options`\nto `ref_format`, we will pass it to ref-filter later.\n2. Let `batch_objects()` add atoms to format, and use\n`verify_ref_format()` to check atoms.\n3. Use `format_ref_array_item()` in `batch_object_write()` to\nget the formatted data corresponding to the object. If the\nreturn value of `format_ref_array_item()` is equals to zero,\nuse `batch_write()` to print object data; else if the return\nvalue is less than zero, use `die()` to print the error message\nand exit; else if return value is greater than zero, only print\nthe error message, but don't exit.\n4. Use free_ref_array_item_value() to free ref_array_item's\nvalue.\n\nMost of the atoms in `for-each-ref --format` are now supported,\nsuch as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`,\n`%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected:\n`%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`,\n`%(flag)`, `%(HEAD)`, because these atoms are unique to those objects\nthat pointed to by a ref, \"for-each-ref\"'s family can naturally use\nthese atoms, but not all objects are pointed to be a ref, so \"cat-file\"\nwill not be able to use them.\n\nThe performance for `git cat-file --batch-all-objects\n--batch-check` on the Git repository itself with performance\ntesting tool `hyperfine` changes from 616.7 ms ± 8.9 ms to\n758.7 ms ± 16.4 ms.\n\nThe performance for `git cat-file --batch-all-objects --batch\n>/dev/null` on the Git repository itself with performance testing\ntool `time` change from \"25.26s user 0.30s system 98% cpu 25.840 total\"\nto \"28.79s user 0.83s system 99% cpu 29.829 total\".\n\nThe reasons for the performance degradation are as follows:\n1. There are a lot of data copies in the logic of ref-filter.\n2, In order to be able to support more useful formats, complex\ndata structure and parsing process are used in ref-filter.\n\nA later patch will add a fast path which will mitigate the\nperformance regression introduced by this patch.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-cat-file.txt | 6 +\n builtin/cat-file.c | 242 +++++++--------------------------\n t/t1006-cat-file.sh | 229 ++++++++++++++++---------------\n 3 files changed, 174 insertions(+), 303 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex 4eb0421b3fd..ef8ab952b2f 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -226,6 +226,12 @@ newline. The available atoms are:\n \tafter that first run of whitespace (i.e., the \"rest\" of the\n \tline) are output in place of the `%(rest)` atom.\n \n+Note that most of the atoms in `for-each-ref --format` are now supported,\n+such as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`,\n+`%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected:\n+`%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`,\n+`%(flag)`, `%(HEAD)`. See linkgit:git-for-each-ref[1].\n+\n If no format is specified, the default format is `%(objectname)\n %(objecttype) %(objectsize)`.\n \ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 41d407638d5..5b163551fc6 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -16,6 +16,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n+#include \"ref-filter.h\"\n \n struct batch_options {\n \tint enabled;\n@@ -25,7 +26,7 @@ struct batch_options {\n \tint all_objects;\n \tint unordered;\n \tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n-\tconst char *format;\n+\tstruct ref_format format;\n };\n \n static const char *force_path;\n@@ -195,99 +196,10 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \n struct expand_data {\n \tstruct object_id oid;\n-\tenum object_type type;\n-\tunsigned long size;\n-\toff_t disk_size;\n \tconst char *rest;\n-\tstruct object_id delta_base_oid;\n-\n-\t/*\n-\t * If mark_query is true, we do not expand anything, but rather\n-\t * just mark the object_info with items we wish to query.\n-\t */\n-\tint mark_query;\n-\n-\t/*\n-\t * Whether to split the input on whitespace before feeding it to\n-\t * get_sha1; this is decided during the mark_query phase based on\n-\t * whether we have a %(rest) token in our format.\n-\t */\n \tint split_on_whitespace;\n-\n-\t/*\n-\t * After a mark_query run, this object_info is set up to be\n-\t * passed to oid_object_info_extended. It will point to the data\n-\t * elements above, so you can retrieve the response from there.\n-\t */\n-\tstruct object_info info;\n-\n-\t/*\n-\t * This flag will be true if the requested batch format and options\n-\t * don't require us to call oid_object_info, which can then be\n-\t * optimized out.\n-\t */\n-\tunsigned skip_object_info : 1;\n };\n \n-static int is_atom(const char *atom, const char *s, int slen)\n-{\n-\tint alen = strlen(atom);\n-\treturn alen == slen && !memcmp(atom, s, alen);\n-}\n-\n-static void expand_atom(struct strbuf *sb, const char *atom, int len,\n-\t\t\tvoid *vdata)\n-{\n-\tstruct expand_data *data = vdata;\n-\n-\tif (is_atom(\"objectname\", atom, len)) {\n-\t\tif (!data->mark_query)\n-\t\t\tstrbuf_addstr(sb, oid_to_hex(&data->oid));\n-\t} else if (is_atom(\"objecttype\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->info.typep = &data->type;\n-\t\telse\n-\t\t\tstrbuf_addstr(sb, type_name(data->type));\n-\t} else if (is_atom(\"objectsize\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->info.sizep = &data->size;\n-\t\telse\n-\t\t\tstrbuf_addf(sb, \"%\"PRIuMAX , (uintmax_t)data->size);\n-\t} else if (is_atom(\"objectsize:disk\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->info.disk_sizep = &data->disk_size;\n-\t\telse\n-\t\t\tstrbuf_addf(sb, \"%\"PRIuMAX, (uintmax_t)data->disk_size);\n-\t} else if (is_atom(\"rest\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->split_on_whitespace = 1;\n-\t\telse if (data->rest)\n-\t\t\tstrbuf_addstr(sb, data->rest);\n-\t} else if (is_atom(\"deltabase\", atom, len)) {\n-\t\tif (data->mark_query)\n-\t\t\tdata->info.delta_base_oid = &data->delta_base_oid;\n-\t\telse\n-\t\t\tstrbuf_addstr(sb,\n-\t\t\t\t oid_to_hex(&data->delta_base_oid));\n-\t} else\n-\t\tdie(\"unknown format element: %.*s\", len, atom);\n-}\n-\n-static size_t expand_format(struct strbuf *sb, const char *start, void *data)\n-{\n-\tconst char *end;\n-\n-\tif (*start != '(')\n-\t\treturn 0;\n-\tend = strchr(start + 1, ')');\n-\tif (!end)\n-\t\tdie(\"format element '%s' does not end in ')'\", start);\n-\n-\texpand_atom(sb, start + 1, end - start - 1, data);\n-\n-\treturn end - start + 1;\n-}\n-\n static void batch_write(struct batch_options *opt, const void *data, int len)\n {\n \tif (opt->buffer_output) {\n@@ -297,87 +209,34 @@ static void batch_write(struct batch_options *opt, const void *data, int len)\n \t\twrite_or_die(1, data, len);\n }\n \n-static void print_object_or_die(struct batch_options *opt, struct expand_data *data)\n-{\n-\tconst struct object_id *oid = &data->oid;\n-\n-\tassert(data->info.typep);\n-\n-\tif (data->type == OBJ_BLOB) {\n-\t\tif (opt->buffer_output)\n-\t\t\tfflush(stdout);\n-\t\tif (opt->cmdmode) {\n-\t\t\tchar *contents;\n-\t\t\tunsigned long size;\n-\n-\t\t\tif (!data->rest)\n-\t\t\t\tdie(\"missing path for '%s'\", oid_to_hex(oid));\n-\n-\t\t\tif (opt->cmdmode == 'w') {\n-\t\t\t\tif (filter_object(data->rest, 0100644, oid,\n-\t\t\t\t\t\t &contents, &size))\n-\t\t\t\t\tdie(\"could not convert '%s' %s\",\n-\t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else if (opt->cmdmode == 'c') {\n-\t\t\t\tenum object_type type;\n-\t\t\t\tif (!textconv_object(the_repository,\n-\t\t\t\t\t\t data->rest, 0100644, oid,\n-\t\t\t\t\t\t 1, &contents, &size))\n-\t\t\t\t\tcontents = read_object_file(oid,\n-\t\t\t\t\t\t\t\t &type,\n-\t\t\t\t\t\t\t\t &size);\n-\t\t\t\tif (!contents)\n-\t\t\t\t\tdie(\"could not convert '%s' %s\",\n-\t\t\t\t\t oid_to_hex(oid), data->rest);\n-\t\t\t} else\n-\t\t\t\tBUG(\"invalid cmdmode: %c\", opt->cmdmode);\n-\t\t\tbatch_write(opt, contents, size);\n-\t\t\tfree(contents);\n-\t\t} else {\n-\t\t\tstream_blob(oid);\n-\t\t}\n-\t}\n-\telse {\n-\t\tenum object_type type;\n-\t\tunsigned long size;\n-\t\tvoid *contents;\n-\n-\t\tcontents = read_object_file(oid, &type, &size);\n-\t\tif (!contents)\n-\t\t\tdie(\"object %s disappeared\", oid_to_hex(oid));\n-\t\tif (type != data->type)\n-\t\t\tdie(\"object %s changed type!?\", oid_to_hex(oid));\n-\t\tif (data->info.sizep && size != data->size)\n-\t\t\tdie(\"object %s changed size!?\", oid_to_hex(oid));\n-\n-\t\tbatch_write(opt, contents, size);\n-\t\tfree(contents);\n-\t}\n-}\n \n static void batch_object_write(const char *obj_name,\n \t\t\t struct strbuf *scratch,\n \t\t\t struct batch_options *opt,\n \t\t\t struct expand_data *data)\n {\n-\tif (!data->skip_object_info &&\n-\t oid_object_info_extended(the_repository, &data->oid, &data->info,\n-\t\t\t\t OBJECT_INFO_LOOKUP_REPLACE) < 0) {\n-\t\tprintf(\"%s missing\\n\",\n-\t\t obj_name ? obj_name : oid_to_hex(&data->oid));\n-\t\tfflush(stdout);\n-\t\treturn;\n-\t}\n+\tint ret;\n+\tstruct strbuf err = STRBUF_INIT;\n+\tstruct ref_array_item item = { data->oid, data->rest };\n \n \tstrbuf_reset(scratch);\n-\tstrbuf_expand(scratch, opt->format, expand_format, data);\n-\tstrbuf_addch(scratch, '\\n');\n-\tbatch_write(opt, scratch->buf, scratch->len);\n \n-\tif (opt->print_contents) {\n-\t\tprint_object_or_die(opt, data);\n-\t\tbatch_write(opt, \"\\n\", 1);\n+\tret = format_ref_array_item(&item, &opt->format, scratch, &err);\n+\tif (ret < 0)\n+\t\tdie(\"%s\\n\", err.buf);\n+\tif (ret) {\n+\t\t/* ret > 0 means when the object corresponding to oid\n+\t\t * cannot be found in format_ref_array_item(), we only print\n+\t\t * the error message.\n+\t\t */\n+\t\tprintf(\"%s\\n\", err.buf);\n+\t\tfflush(stdout);\n+\t} else {\n+\t\tstrbuf_addch(scratch, '\\n');\n+\t\tbatch_write(opt, scratch->buf, scratch->len);\n \t}\n+\tfree_ref_array_item_value(&item);\n+\tstrbuf_release(&err);\n }\n \n static void batch_one_object(const char *obj_name,\n@@ -495,43 +354,37 @@ static int batch_unordered_packed(const struct object_id *oid,\n \treturn batch_unordered_object(oid, data);\n }\n \n-static int batch_objects(struct batch_options *batch)\n+static const char * const cat_file_usage[] = {\n+\tN_(\"git cat-file (-t [--allow-unknown-type] | -s [--allow-unknown-type] | -e | -p | | --textconv | --filters) [--path=] \"),\n+\tN_(\"git cat-file (--batch[=] | --batch-check[=]) [--follow-symlinks] [--textconv | --filters]\"),\n+\tNULL\n+};\n+\n+static int batch_objects(struct batch_options *batch, const struct option *options)\n {\n \tstruct strbuf input = STRBUF_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n+\tstruct strbuf format = STRBUF_INIT;\n \tstruct expand_data data;\n \tint save_warning;\n \tint retval = 0;\n \n-\tif (!batch->format)\n-\t\tbatch->format = \"%(objectname) %(objecttype) %(objectsize)\";\n-\n-\t/*\n-\t * Expand once with our special mark_query flag, which will prime the\n-\t * object_info to be handed to oid_object_info_extended for each\n-\t * object.\n-\t */\n \tmemset(&data, 0, sizeof(data));\n-\tdata.mark_query = 1;\n-\tstrbuf_expand(&output, batch->format, expand_format, &data);\n-\tdata.mark_query = 0;\n-\tstrbuf_release(&output);\n-\tif (batch->cmdmode)\n-\t\tdata.split_on_whitespace = 1;\n-\n-\t/*\n-\t * If we are printing out the object, then always fill in the type,\n-\t * since we will want to decide whether or not to stream.\n-\t */\n+\tif (batch->format.format)\n+\t\tstrbuf_addstr(&format, batch->format.format);\n+\telse\n+\t\tstrbuf_addstr(&format, \"%(objectname) %(objecttype) %(objectsize)\");\n \tif (batch->print_contents)\n-\t\tdata.info.typep = &data.type;\n+\t\tstrbuf_addstr(&format, \"\\n%(raw)\");\n+\tbatch->format.format = format.buf;\n+\tif (verify_ref_format(&batch->format))\n+\t\tusage_with_options(cat_file_usage, options);\n+\n+\tif (batch->cmdmode || batch->format.use_rest)\n+\t\tdata.split_on_whitespace = 1;\n \n \tif (batch->all_objects) {\n \t\tstruct object_cb_data cb;\n-\t\tstruct object_info empty = OBJECT_INFO_INIT;\n-\n-\t\tif (!memcmp(&data.info, &empty, sizeof(empty)))\n-\t\t\tdata.skip_object_info = 1;\n \n \t\tif (has_promisor_remote())\n \t\t\twarning(\"This repository uses promisor remotes. Some objects may not be loaded.\");\n@@ -561,6 +414,7 @@ static int batch_objects(struct batch_options *batch)\n \t\t\toid_array_clear(&sa);\n \t\t}\n \n+\t\tstrbuf_release(&format);\n \t\tstrbuf_release(&output);\n \t\treturn 0;\n \t}\n@@ -593,18 +447,13 @@ static int batch_objects(struct batch_options *batch)\n \t\tbatch_one_object(input.buf, &output, batch, &data);\n \t}\n \n+\tstrbuf_release(&format);\n \tstrbuf_release(&input);\n \tstrbuf_release(&output);\n \twarn_on_object_refname_ambiguity = save_warning;\n \treturn retval;\n }\n \n-static const char * const cat_file_usage[] = {\n-\tN_(\"git cat-file (-t [--allow-unknown-type] | -s [--allow-unknown-type] | -e | -p | | --textconv | --filters) [--path=] \"),\n-\tN_(\"git cat-file (--batch[=] | --batch-check[=]) [--follow-symlinks] [--textconv | --filters]\"),\n-\tNULL\n-};\n-\n static int git_cat_file_config(const char *var, const char *value, void *cb)\n {\n \tif (userdiff_config(var, value) < 0)\n@@ -627,7 +476,7 @@ static int batch_option_callback(const struct option *opt,\n \n \tbo->enabled = 1;\n \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n-\tbo->format = arg;\n+\tbo->format.format = arg;\n \n \treturn 0;\n }\n@@ -636,7 +485,9 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n {\n \tint opt = 0;\n \tconst char *exp_type = NULL, *obj_name = NULL;\n-\tstruct batch_options batch = {0};\n+\tstruct batch_options batch = {\n+\t\t.format = REF_FORMAT_INIT\n+\t};\n \tint unknown_type = 0;\n \n \tconst struct option options[] = {\n@@ -675,6 +526,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \tgit_config(git_cat_file_config, NULL);\n \n \tbatch.buffer_output = -1;\n+\tbatch.format.cat_file_mode = 1;\n \targc = parse_options(argc, argv, prefix, options, cat_file_usage, 0);\n \n \tif (opt) {\n@@ -718,7 +570,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \t\tbatch.buffer_output = batch.all_objects;\n \n \tif (batch.enabled)\n-\t\treturn batch_objects(&batch);\n+\t\treturn batch_objects(&batch, options);\n \n \tif (unknown_type && opt != 't' && opt != 's')\n \t\tdie(\"git cat-file --allow-unknown-type: use with -s or -t\");\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 95d760652eb..8c1943011fb 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -679,51 +679,51 @@ batch_test_atom refs/heads/main '%(objectsize)'\n batch_test_atom refs/heads/main '%(objectsize:disk)'\n batch_test_atom refs/heads/main '%(deltabase)'\n batch_test_atom refs/heads/main '%(objectname)'\n-batch_test_atom refs/heads/main '%(objectname:short)' fail\n-batch_test_atom refs/heads/main '%(objectname:short=1)' fail\n-batch_test_atom refs/heads/main '%(objectname:short=10)' fail\n-batch_test_atom refs/heads/main '%(tree)' fail\n-batch_test_atom refs/heads/main '%(tree:short)' fail\n-batch_test_atom refs/heads/main '%(tree:short=1)' fail\n-batch_test_atom refs/heads/main '%(tree:short=10)' fail\n-batch_test_atom refs/heads/main '%(parent)' fail\n-batch_test_atom refs/heads/main '%(parent:short)' fail\n-batch_test_atom refs/heads/main '%(parent:short=1)' fail\n-batch_test_atom refs/heads/main '%(parent:short=10)' fail\n-batch_test_atom refs/heads/main '%(numparent)' fail\n-batch_test_atom refs/heads/main '%(object)' fail\n-batch_test_atom refs/heads/main '%(type)' fail\n-batch_test_atom refs/heads/main '%(raw)' fail\n-batch_test_atom refs/heads/main '%(*objectname)' fail\n-batch_test_atom refs/heads/main '%(*objecttype)' fail\n-batch_test_atom refs/heads/main '%(author)' fail\n-batch_test_atom refs/heads/main '%(authorname)' fail\n-batch_test_atom refs/heads/main '%(authoremail)' fail\n-batch_test_atom refs/heads/main '%(authoremail:trim)' fail\n-batch_test_atom refs/heads/main '%(authoremail:localpart)' fail\n-batch_test_atom refs/heads/main '%(authordate)' fail\n-batch_test_atom refs/heads/main '%(committer)' fail\n-batch_test_atom refs/heads/main '%(committername)' fail\n-batch_test_atom refs/heads/main '%(committeremail)' fail\n-batch_test_atom refs/heads/main '%(committeremail:trim)' fail\n-batch_test_atom refs/heads/main '%(committeremail:localpart)' fail\n-batch_test_atom refs/heads/main '%(committerdate)' fail\n-batch_test_atom refs/heads/main '%(tag)' fail\n-batch_test_atom refs/heads/main '%(tagger)' fail\n-batch_test_atom refs/heads/main '%(taggername)' fail\n-batch_test_atom refs/heads/main '%(taggeremail)' fail\n-batch_test_atom refs/heads/main '%(taggeremail:trim)' fail\n-batch_test_atom refs/heads/main '%(taggeremail:localpart)' fail\n-batch_test_atom refs/heads/main '%(taggerdate)' fail\n-batch_test_atom refs/heads/main '%(creator)' fail\n-batch_test_atom refs/heads/main '%(creatordate)' fail\n-batch_test_atom refs/heads/main '%(subject)' fail\n-batch_test_atom refs/heads/main '%(subject:sanitize)' fail\n-batch_test_atom refs/heads/main '%(contents:subject)' fail\n-batch_test_atom refs/heads/main '%(body)' fail\n-batch_test_atom refs/heads/main '%(contents:body)' fail\n-batch_test_atom refs/heads/main '%(contents:signature)' fail\n-batch_test_atom refs/heads/main '%(contents)' fail\n+batch_test_atom refs/heads/main '%(objectname:short)'\n+batch_test_atom refs/heads/main '%(objectname:short=1)'\n+batch_test_atom refs/heads/main '%(objectname:short=10)'\n+batch_test_atom refs/heads/main '%(tree)'\n+batch_test_atom refs/heads/main '%(tree:short)'\n+batch_test_atom refs/heads/main '%(tree:short=1)'\n+batch_test_atom refs/heads/main '%(tree:short=10)'\n+batch_test_atom refs/heads/main '%(parent)'\n+batch_test_atom refs/heads/main '%(parent:short)'\n+batch_test_atom refs/heads/main '%(parent:short=1)'\n+batch_test_atom refs/heads/main '%(parent:short=10)'\n+batch_test_atom refs/heads/main '%(numparent)'\n+batch_test_atom refs/heads/main '%(object)'\n+batch_test_atom refs/heads/main '%(type)'\n+batch_test_atom refs/heads/main '%(raw)'\n+batch_test_atom refs/heads/main '%(*objectname)'\n+batch_test_atom refs/heads/main '%(*objecttype)'\n+batch_test_atom refs/heads/main '%(author)'\n+batch_test_atom refs/heads/main '%(authorname)'\n+batch_test_atom refs/heads/main '%(authoremail)'\n+batch_test_atom refs/heads/main '%(authoremail:trim)'\n+batch_test_atom refs/heads/main '%(authoremail:localpart)'\n+batch_test_atom refs/heads/main '%(authordate)'\n+batch_test_atom refs/heads/main '%(committer)'\n+batch_test_atom refs/heads/main '%(committername)'\n+batch_test_atom refs/heads/main '%(committeremail)'\n+batch_test_atom refs/heads/main '%(committeremail:trim)'\n+batch_test_atom refs/heads/main '%(committeremail:localpart)'\n+batch_test_atom refs/heads/main '%(committerdate)'\n+batch_test_atom refs/heads/main '%(tag)'\n+batch_test_atom refs/heads/main '%(tagger)'\n+batch_test_atom refs/heads/main '%(taggername)'\n+batch_test_atom refs/heads/main '%(taggeremail)'\n+batch_test_atom refs/heads/main '%(taggeremail:trim)'\n+batch_test_atom refs/heads/main '%(taggeremail:localpart)'\n+batch_test_atom refs/heads/main '%(taggerdate)'\n+batch_test_atom refs/heads/main '%(creator)'\n+batch_test_atom refs/heads/main '%(creatordate)'\n+batch_test_atom refs/heads/main '%(subject)'\n+batch_test_atom refs/heads/main '%(subject:sanitize)'\n+batch_test_atom refs/heads/main '%(contents:subject)'\n+batch_test_atom refs/heads/main '%(body)'\n+batch_test_atom refs/heads/main '%(contents:body)'\n+batch_test_atom refs/heads/main '%(contents:signature)'\n+batch_test_atom refs/heads/main '%(contents)'\n batch_test_atom refs/heads/main '%(HEAD)' fail\n batch_test_atom refs/heads/main '%(upstream:track)' fail\n batch_test_atom refs/heads/main '%(upstream:trackshort)' fail\n@@ -742,52 +742,52 @@ batch_test_atom refs/tags/testtag '%(push)' fail\n batch_test_atom refs/tags/testtag '%(objecttype)'\n batch_test_atom refs/tags/testtag '%(objectsize)'\n batch_test_atom refs/tags/testtag '%(objectsize:disk)'\n-batch_test_atom refs/tags/testtag '%(*objectsize:disk)' fail\n+batch_test_atom refs/tags/testtag '%(*objectsize:disk)'\n batch_test_atom refs/tags/testtag '%(deltabase)'\n-batch_test_atom refs/tags/testtag '%(*deltabase)' fail\n+batch_test_atom refs/tags/testtag '%(*deltabase)'\n batch_test_atom refs/tags/testtag '%(objectname)'\n-batch_test_atom refs/tags/testtag '%(objectname:short)' fail\n-batch_test_atom refs/tags/testtag '%(tree)' fail\n-batch_test_atom refs/tags/testtag '%(tree:short)' fail\n-batch_test_atom refs/tags/testtag '%(tree:short=1)' fail\n-batch_test_atom refs/tags/testtag '%(tree:short=10)' fail\n-batch_test_atom refs/tags/testtag '%(parent)' fail\n-batch_test_atom refs/tags/testtag '%(parent:short)' fail\n-batch_test_atom refs/tags/testtag '%(parent:short=1)' fail\n-batch_test_atom refs/tags/testtag '%(parent:short=10)' fail\n-batch_test_atom refs/tags/testtag '%(numparent)' fail\n-batch_test_atom refs/tags/testtag '%(object)' fail\n-batch_test_atom refs/tags/testtag '%(type)' fail\n-batch_test_atom refs/tags/testtag '%(*objectname)' fail\n-batch_test_atom refs/tags/testtag '%(*objecttype)' fail\n-batch_test_atom refs/tags/testtag '%(author)' fail\n-batch_test_atom refs/tags/testtag '%(authorname)' fail\n-batch_test_atom refs/tags/testtag '%(authoremail)' fail\n-batch_test_atom refs/tags/testtag '%(authoremail:trim)' fail\n-batch_test_atom refs/tags/testtag '%(authoremail:localpart)' fail\n-batch_test_atom refs/tags/testtag '%(authordate)' fail\n-batch_test_atom refs/tags/testtag '%(committer)' fail\n-batch_test_atom refs/tags/testtag '%(committername)' fail\n-batch_test_atom refs/tags/testtag '%(committeremail)' fail\n-batch_test_atom refs/tags/testtag '%(committeremail:trim)' fail\n-batch_test_atom refs/tags/testtag '%(committeremail:localpart)' fail\n-batch_test_atom refs/tags/testtag '%(committerdate)' fail\n-batch_test_atom refs/tags/testtag '%(tag)' fail\n-batch_test_atom refs/tags/testtag '%(tagger)' fail\n-batch_test_atom refs/tags/testtag '%(taggername)' fail\n-batch_test_atom refs/tags/testtag '%(taggeremail)' fail\n-batch_test_atom refs/tags/testtag '%(taggeremail:trim)' fail\n-batch_test_atom refs/tags/testtag '%(taggeremail:localpart)' fail\n-batch_test_atom refs/tags/testtag '%(taggerdate)' fail\n-batch_test_atom refs/tags/testtag '%(creator)' fail\n-batch_test_atom refs/tags/testtag '%(creatordate)' fail\n-batch_test_atom refs/tags/testtag '%(subject)' fail\n-batch_test_atom refs/tags/testtag '%(subject:sanitize)' fail\n-batch_test_atom refs/tags/testtag '%(contents:subject)' fail\n-batch_test_atom refs/tags/testtag '%(body)' fail\n-batch_test_atom refs/tags/testtag '%(contents:body)' fail\n-batch_test_atom refs/tags/testtag '%(contents:signature)' fail\n-batch_test_atom refs/tags/testtag '%(contents)' fail\n+batch_test_atom refs/tags/testtag '%(objectname:short)'\n+batch_test_atom refs/tags/testtag '%(tree)'\n+batch_test_atom refs/tags/testtag '%(tree:short)'\n+batch_test_atom refs/tags/testtag '%(tree:short=1)'\n+batch_test_atom refs/tags/testtag '%(tree:short=10)'\n+batch_test_atom refs/tags/testtag '%(parent)'\n+batch_test_atom refs/tags/testtag '%(parent:short)'\n+batch_test_atom refs/tags/testtag '%(parent:short=1)'\n+batch_test_atom refs/tags/testtag '%(parent:short=10)'\n+batch_test_atom refs/tags/testtag '%(numparent)'\n+batch_test_atom refs/tags/testtag '%(object)'\n+batch_test_atom refs/tags/testtag '%(type)'\n+batch_test_atom refs/tags/testtag '%(*objectname)'\n+batch_test_atom refs/tags/testtag '%(*objecttype)'\n+batch_test_atom refs/tags/testtag '%(author)'\n+batch_test_atom refs/tags/testtag '%(authorname)'\n+batch_test_atom refs/tags/testtag '%(authoremail)'\n+batch_test_atom refs/tags/testtag '%(authoremail:trim)'\n+batch_test_atom refs/tags/testtag '%(authoremail:localpart)'\n+batch_test_atom refs/tags/testtag '%(authordate)'\n+batch_test_atom refs/tags/testtag '%(committer)'\n+batch_test_atom refs/tags/testtag '%(committername)'\n+batch_test_atom refs/tags/testtag '%(committeremail)'\n+batch_test_atom refs/tags/testtag '%(committeremail:trim)'\n+batch_test_atom refs/tags/testtag '%(committeremail:localpart)'\n+batch_test_atom refs/tags/testtag '%(committerdate)'\n+batch_test_atom refs/tags/testtag '%(tag)'\n+batch_test_atom refs/tags/testtag '%(tagger)'\n+batch_test_atom refs/tags/testtag '%(taggername)'\n+batch_test_atom refs/tags/testtag '%(taggeremail)'\n+batch_test_atom refs/tags/testtag '%(taggeremail:trim)'\n+batch_test_atom refs/tags/testtag '%(taggeremail:localpart)'\n+batch_test_atom refs/tags/testtag '%(taggerdate)'\n+batch_test_atom refs/tags/testtag '%(creator)'\n+batch_test_atom refs/tags/testtag '%(creatordate)'\n+batch_test_atom refs/tags/testtag '%(subject)'\n+batch_test_atom refs/tags/testtag '%(subject:sanitize)'\n+batch_test_atom refs/tags/testtag '%(contents:subject)'\n+batch_test_atom refs/tags/testtag '%(body)'\n+batch_test_atom refs/tags/testtag '%(contents:body)'\n+batch_test_atom refs/tags/testtag '%(contents:signature)'\n+batch_test_atom refs/tags/testtag '%(contents)'\n batch_test_atom refs/tags/testtag '%(HEAD)' fail\n \n batch_test_atom refs/myblobs/blob1 '%(refname)' fail\n@@ -801,37 +801,50 @@ batch_test_atom refs/myblobs/blob1 '%(objectsize)'\n batch_test_atom refs/myblobs/blob1 '%(objectsize:disk)'\n batch_test_atom refs/myblobs/blob1 '%(deltabase)'\n \n-batch_test_atom refs/myblobs/blob1 '%(contents)' fail\n-batch_test_atom refs/myblobs/blob2 '%(contents)' fail\n+batch_test_atom refs/myblobs/blob1 '%(contents)'\n+batch_test_atom refs/myblobs/blob2 '%(contents)'\n \n-batch_test_atom refs/myblobs/blob1 '%(raw)' fail\n-batch_test_atom refs/myblobs/blob2 '%(raw)' fail\n-batch_test_atom refs/mytrees/tree1 '%(raw)' fail\n+batch_test_atom refs/myblobs/blob1 '%(raw)'\n+batch_test_atom refs/myblobs/blob2 '%(raw)'\n+batch_test_atom refs/mytrees/tree1 '%(raw)'\n \n-batch_test_atom refs/myblobs/blob1 '%(raw:size)' fail\n-batch_test_atom refs/myblobs/blob2 '%(raw:size)' fail\n-batch_test_atom refs/mytrees/tree1 '%(raw:size)' fail\n+batch_test_atom refs/myblobs/blob1 '%(raw:size)'\n+batch_test_atom refs/myblobs/blob2 '%(raw:size)'\n+batch_test_atom refs/mytrees/tree1 '%(raw:size)'\n \n-batch_test_atom refs/myblobs/blob1 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)' fail\n-batch_test_atom refs/myblobs/blob2 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)' fail\n-batch_test_atom refs/mytrees/tree1 '%(if:equals=tree)%(objecttype)%(then)tree%(else)not tree%(end)' fail\n+batch_test_atom refs/myblobs/blob1 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)'\n+batch_test_atom refs/myblobs/blob2 '%(if:equals=blob)%(objecttype)%(then)commit%(else)not commit%(end)'\n+batch_test_atom refs/mytrees/tree1 '%(if:equals=tree)%(objecttype)%(then)tree%(else)not tree%(end)'\n \n-batch_test_atom refs/heads/main '%(align:60) objectname is %(objectname)%(end)|%(objectname)' fail\n-batch_test_atom refs/heads/main '%(align:left,60) objectname is %(objectname)%(end)|%(objectname)' fail\n-batch_test_atom refs/heads/main '%(align:middle,60) objectname is %(objectname)%(end)|%(objectname)' fail\n-batch_test_atom refs/heads/main '%(align:60,right) objectname is %(objectname)%(end)|%(objectname)' fail\n+batch_test_atom refs/heads/main '%(align:60) objectname is %(objectname)%(end)|%(objectname)'\n+batch_test_atom refs/heads/main '%(align:left,60) objectname is %(objectname)%(end)|%(objectname)'\n+batch_test_atom refs/heads/main '%(align:middle,60) objectname is %(objectname)%(end)|%(objectname)'\n+batch_test_atom refs/heads/main '%(align:60,right) objectname is %(objectname)%(end)|%(objectname)'\n \n batch_test_atom refs/heads/main 'VALID'\n batch_test_atom refs/heads/main '%(INVALID)' fail\n batch_test_atom refs/heads/main '%(authordate:INVALID)' fail\n \n batch_test_atom refs/heads/main '%(objectname) %(objecttype) %(objectsize)\n-%(raw)' fail\n+%(raw)'\n batch_test_atom refs/tags/testtag '%(objectname) %(objecttype) %(objectsize)\n-%(raw)' fail\n+%(raw)'\n batch_test_atom refs/myblobs/blob1 '%(objectname) %(objecttype) %(objectsize)\n-%(raw)' fail\n+%(raw)'\n batch_test_atom refs/myblobs/blob2 '%(objectname) %(objecttype) %(objectsize)\n-%(raw)' fail\n+%(raw)'\n+\n+test_expect_success 'cat-file --batch equals to --batch-check with atoms' '\n+\tgit cat-file --batch-check=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" >expected <<-EOF &&\n+\trefs/heads/main\n+\trefs/tags/testtag\n+\tEOF\n+\tgit cat-file --batch >actual <<-EOF &&\n+\trefs/heads/main\n+\trefs/tags/testtag\n+\tEOF\n+\tcmp expected actual\n+'\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"430225","messageId":"96ef6431a2b64bbf15a9fbe7755dc3e882db072e.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 16/17] [GSOC] ref-filter: remove grab_oid() function","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:24Z","receivedAt":"2021-07-15T15:40:48Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause \"atom_type == ATOM_OBJECTNAME\" implies the condition\nof `starts_with(name, \"objectname\")`, \"atom_type == ATOM_TREE\"\nimplies the condition of `starts_with(name, \"tree\")`, so the\ncheck for `starts_with(name, field)` in grab_oid() is redundant.\n\nSo Remove the grab_oid() from ref-filter, to reduce repeated check.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 26 +++++++++-----------------\n 1 file changed, 9 insertions(+), 17 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 91e26c9aba3..1c7287f1061 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1077,16 +1077,6 @@ static const char *do_grab_oid(const char *field, const struct object_id *oid,\n \t}\n }\n \n-static int grab_oid(const char *name, const char *field, const struct object_id *oid,\n-\t\t struct atom_value *v, struct used_atom *atom)\n-{\n-\tif (starts_with(name, field)) {\n-\t\tv->s = xstrdup(do_grab_oid(field, oid, atom));\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n /* See grab_values */\n static void grab_common_values(struct atom_value *val, int deref, struct expand_data *oi)\n {\n@@ -1112,8 +1102,9 @@ static void grab_common_values(struct atom_value *val, int deref, struct expand_\n \t\t\t}\n \t\t} else if (atom_type == ATOM_DELTABASE)\n \t\t\tv->s = xstrdup(oid_to_hex(&oi->delta_base_oid));\n-\t\telse if (atom_type == ATOM_OBJECTNAME && deref)\n-\t\t\tgrab_oid(name, \"objectname\", &oi->oid, v, &used_atom[i]);\n+\t\telse if (atom_type == ATOM_OBJECTNAME && deref) {\n+\t\t\tv->s = xstrdup(do_grab_oid(\"objectname\", &oi->oid, &used_atom[i]));\n+\t\t}\n \t}\n }\n \n@@ -1154,9 +1145,10 @@ static void grab_commit_values(struct atom_value *val, int deref, struct object\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (atom_type == ATOM_TREE &&\n-\t\t grab_oid(name, \"tree\", get_commit_tree_oid(commit), v, &used_atom[i]))\n+\t\tif (atom_type == ATOM_TREE) {\n+\t\t\tv->s = xstrdup(do_grab_oid(\"tree\", get_commit_tree_oid(commit), &used_atom[i]));\n \t\t\tcontinue;\n+\t\t}\n \t\tif (atom_type == ATOM_NUMPARENT) {\n \t\t\tv->value = commit_list_count(commit->parents);\n \t\t\tv->s = xstrfmt(\"%lu\", (unsigned long)v->value);\n@@ -1959,9 +1951,9 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\t\tv->s = xstrdup(buf + 1);\n \t\t\t}\n \t\t\tcontinue;\n-\t\t} else if (!deref && atom_type == ATOM_OBJECTNAME &&\n-\t\t\t grab_oid(name, \"objectname\", &ref->objectname, v, atom)) {\n-\t\t\t\tcontinue;\n+\t\t} else if (!deref && atom_type == ATOM_OBJECTNAME) {\n+\t\t\t v->s = xstrdup(do_grab_oid(\"objectname\", &ref->objectname, atom));\n+\t\t\t continue;\n \t\t} else if (atom_type == ATOM_HEAD) {\n \t\t\tif (atom->u.head && !strcmp(ref->refname, atom->u.head))\n \t\t\t\tv->s = xstrdup(\"*\");\n-- \ngitgitgadget\n\n"},{"id":"430226","messageId":"5903d02324f3275b3aa442bb3ca2602564c543dc.1626363626.git.gitgitgadget@gmail.com","threadId":"56095","inReplyTo":"pull.993.v2.git.1626363626.gitgitgadget@gmail.com","subject":"[PATCH v2 17/17] [GSOC] cat-file: use fast path when using default_format","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-15T15:40:25Z","receivedAt":"2021-07-15T15:40:50Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd the member `default_format` to struct `batch_options`,\nwhen we are using the default format on `git cat-file --batch`,\nor `git cat-file --batch-check`, `default_format` will be set,\nif we don't use `--textconv` or `--filter`, then we will not call\nverify_ref_format(), has_object_file() and format_ref_array_item().\nInstead, we get the object data directly through\noid_object_info_extended() and then output the data directly.\n\nBy using this fast path, we can reduce some of the extra overhead\nwhen `cat-file --batch` using ref-filter. The running time of\n`git cat-file --batch-check` will be similar to before, and the\nrunning time of `git cat-file --batch` will be 9.1% less than before.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/cat-file.c | 79 +++++++++++++++++++++++++++++++++-------------\n 1 file changed, 57 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 3a6153e778f..8edc19f2d5a 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -26,6 +26,7 @@ struct batch_options {\n \tint all_objects;\n \tint unordered;\n \tint cmdmode; /* may be 'w' or 'c' for --filters or --textconv */\n+\tint default_format;\n \tstruct ref_format format;\n };\n \n@@ -196,6 +197,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \n struct expand_data {\n \tstruct object_id oid;\n+\tstruct object_info info;\n \tconst char *rest;\n \tint split_on_whitespace;\n };\n@@ -216,27 +218,58 @@ static void batch_object_write(const char *obj_name,\n \t\t\t struct batch_options *opt,\n \t\t\t struct expand_data *data)\n {\n-\tint ret;\n-\tstruct ref_array_item item = { data->oid, data->rest, opt->cmdmode };\n-\n-\tstrbuf_reset(scratch);\n-\tstrbuf_reset(err);\n-\n-\tret = format_ref_array_item(&item, &opt->format, scratch, err);\n-\tif (ret < 0)\n-\t\tdie(\"%s\\n\", err->buf);\n-\tif (ret) {\n-\t\t/* ret > 0 means when the object corresponding to oid\n-\t\t * cannot be found in format_ref_array_item(), we only print\n-\t\t * the error message.\n-\t\t */\n-\t\tprintf(\"%s\\n\", err->buf);\n+\tif (opt->default_format && !opt->cmdmode) {\n+\t\tstruct strbuf type_name = STRBUF_INIT;\n+\t\tunsigned long size;\n+\t\tvoid *content;\n+\n+\t\tif (opt->print_contents)\n+\t\t\tdata->info.contentp = &content;\n+\n+\t\tdata->info.type_name = &type_name;\n+\t\tdata->info.sizep = &size;\n+\n+\t\tif (oid_object_info_extended(the_repository, &data->oid, &data->info,\n+\t\t\t\t\t OBJECT_INFO_LOOKUP_REPLACE) < 0) {\n+\t\t\tprintf(\"%s missing\\n\",\n+\t\t\t obj_name ? obj_name : oid_to_hex(&data->oid));\n+\t\t\tfflush(stdout);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tfprintf(stdout, \"%s %s %\"PRIuMAX\"\\n\", oid_to_hex(&data->oid),\n+\t\t\tdata->info.type_name->buf,\n+\t\t\t(uintmax_t)*data->info.sizep);\n \t\tfflush(stdout);\n+\t\tstrbuf_release(&type_name);\n+\t\tif (opt->print_contents) {\n+\t\t\tbatch_write(opt, content, *data->info.sizep);\n+\t\t\tbatch_write(opt, \"\\n\", 1);\n+\t\t\tfree(content);\n+\t\t}\n \t} else {\n-\t\tstrbuf_addch(scratch, '\\n');\n-\t\tbatch_write(opt, scratch->buf, scratch->len);\n+\t\tint ret;\n+\t\tstruct ref_array_item item = { data->oid, data->rest, opt->cmdmode };\n+\n+\t\tstrbuf_reset(scratch);\n+\t\tstrbuf_reset(err);\n+\n+\t\tret = format_ref_array_item(&item, &opt->format, scratch, err);\n+\t\tif (ret < 0)\n+\t\t\tdie(\"%s\\n\", err->buf);\n+\t\tif (ret) {\n+\t\t\t/* ret > 0 means when the object corresponding to oid\n+\t\t\t * cannot be found in format_ref_array_item(), we only print\n+\t\t\t * the error message.\n+\t\t\t */\n+\t\t\tprintf(\"%s\\n\", err->buf);\n+\t\t\tfflush(stdout);\n+\t\t} else {\n+\t\t\tstrbuf_addch(scratch, '\\n');\n+\t\t\tbatch_write(opt, scratch->buf, scratch->len);\n+\t\t}\n+\t\tfree_ref_array_item_value(&item);\n \t}\n-\tfree_ref_array_item_value(&item);\n }\n \n static void batch_one_object(const char *obj_name,\n@@ -288,7 +321,7 @@ static void batch_one_object(const char *obj_name,\n \t\treturn;\n \t}\n \n-\tif (!has_object_file(&data->oid)) {\n+\tif ((!opt->default_format || opt->cmdmode) && !has_object_file(&data->oid)) {\n \t\tprintf(\"%s missing\\n\",\n \t\t obj_name ? obj_name : oid_to_hex(&data->oid));\n \t\tfflush(stdout);\n@@ -380,7 +413,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio\n \tif (batch->print_contents)\n \t\tstrbuf_addstr(&format, \"\\n%(raw)\");\n \tbatch->format.format = format.buf;\n-\tif (verify_ref_format(&batch->format))\n+\tif ((!batch->default_format || batch->cmdmode) && verify_ref_format(&batch->format))\n \t\tusage_with_options(cat_file_usage, options);\n \n \tif (batch->cmdmode || batch->format.use_rest)\n@@ -483,7 +516,8 @@ static int batch_option_callback(const struct option *opt,\n \tbo->enabled = 1;\n \tbo->print_contents = !strcmp(opt->long_name, \"batch\");\n \tbo->format.format = arg;\n-\n+\tif (arg)\n+\t\tbo->default_format = 0;\n \treturn 0;\n }\n \n@@ -492,7 +526,8 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \tint opt = 0;\n \tconst char *exp_type = NULL, *obj_name = NULL;\n \tstruct batch_options batch = {\n-\t\t.format = REF_FORMAT_INIT\n+\t\t.format = REF_FORMAT_INIT,\n+\t\t.default_format = 1\n \t};\n \tint unknown_type = 0;\n \n-- \ngitgitgadget\n"}]}