{"thread":{"id":"56142","subject":"[PATCH 0/5] [GSOC] ref-filter: add %(raw) and %(rest) atoms","startedAt":"2021-07-22T07:39:23Z","lastAt":"2021-07-27T01:40:36Z","messageCount":45,"participants":["ZheNing Hu via GitGitGadget","Christian Couder","Ævar Arnfjörð Bjarmason","ZheNing Hu","Jacob Keller","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"430881","messageId":"pull.1000.git.1626939557.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":null,"subject":"[PATCH 0/5] [GSOC] ref-filter: add %(raw) and %(rest) atoms","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-22T07:39:11Z","receivedAt":"2021-07-22T07:39:23Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"This patch series is split from my main patch series zh/ref-filter-raw-data,\nto make things easier for reviewers to read.\n\nThis patch series provided %(raw) and %(rest) for ref-filter, which will be\nused by zh/cat-file-reuse-ref-filter-logic later.\n\nZheNing Hu (5):\n [GSOC] ref-filter: add obj-type check in grab contents\n [GSOC] ref-filter: add %(raw) atom\n [GSOC] ref-filter: --format=%(raw) re-support --perl\n [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n [GSOC] ref-filter: add %(rest) atom\n\n Documentation/git-for-each-ref.txt | 9 ++\n builtin/tag.c | 2 +-\n quote.c | 17 ++\n quote.h | 1 +\n ref-filter.c | 240 ++++++++++++++++++++++-------\n ref-filter.h | 9 +-\n t/t3203-branch-output.sh | 4 +\n t/t6300-for-each-ref.sh | 235 ++++++++++++++++++++++++++++\n t/t7004-tag.sh | 4 +\n t/t7030-verify-tag.sh | 4 +\n 10 files changed, 463 insertions(+), 62 deletions(-)\n\n\nbase-commit: daab8a564f8bbac55f70f8bf86c070e001a9b006\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1000%2Fadlternative%2Fref-filter-raw-data-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1000/adlternative/ref-filter-raw-data-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1000\n-- \ngitgitgadget\n"},{"id":"430882","messageId":"a7b4e45f14a27bd67172e24660f448e4c9cce5d8.1626939557.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.git.1626939557.gitgitgadget@gmail.com","subject":"[PATCH 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-22T07:39:12Z","receivedAt":"2021-07-22T07:39:24Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nOnly tag and commit objects use `grab_sub_body_contents()` to grab\nobject contents in the current codebase. We want to teach the\nfunction to also handle blobs and trees to get their raw data,\nwithout parsing a blob (whose contents looks like a commit or a tag)\nincorrectly as a commit or a tag.\n\nSkip the block of code that is specific to handling commits and tags\nearly when the given object is of a wrong type to help later\naddition to handle other types of objects in this function.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Junio C Hamano \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4db0e40ff4c..5cee6512fba 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n+static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n {\n \tint i;\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n+\tvoid *buf = data->content;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n@@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (strcmp(name, \"body\") &&\n-\t\t !starts_with(name, \"subject\") &&\n-\t\t !starts_with(name, \"trailers\") &&\n-\t\t !starts_with(name, \"contents\"))\n+\n+\t\tif ((data->type != OBJ_TAG &&\n+\t\t data->type != OBJ_COMMIT) ||\n+\t\t (strcmp(name, \"body\") &&\n+\t\t !starts_with(name, \"subject\") &&\n+\t\t !starts_with(name, \"trailers\") &&\n+\t\t !starts_with(name, \"contents\")))\n \t\t\tcontinue;\n \t\tif (!subpos)\n \t\t\tfind_subpos(buf,\n@@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n+static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n {\n+\tvoid *buf = data->content;\n+\n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n@@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi->content);\n+\t\tgrab_values(ref->value, deref, *obj, oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n-- \ngitgitgadget\n\n"},{"id":"430883","messageId":"ecd41b370e62cc88f1606569ec700eaca837fa1f.1626939557.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.git.1626939557.gitgitgadget@gmail.com","subject":"[PATCH 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-22T07:39:13Z","receivedAt":"2021-07-22T07:39:24Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd new formatting option `%(raw)`, which will print the raw\nobject data without any changes. It will help further to migrate\nall cat-file formatting logic from cat-file to ref-filter.\n\nThe raw data of blob, tree objects may contain '\\0', but most of\nthe logic in `ref-filter` depends on the output of the atom being\ntext (specifically, no embedded NULs in it).\n\nE.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()`\nadd the data to the buffer. The raw data of a tree object is\n`100644 one\\0...`, only the `100644 one` will be added to the buffer,\nwhich is incorrect.\n\nTherefore, we need to find a way to record the length of the\natom_value's member `s`. Although strbuf can already record the\nstring and its length, if we want to replace the type of atom_value's\nmember `s` with strbuf, many places in ref-filter that are filled\nwith dynamically allocated mermory in `v->s` are not easy to replace.\nAt the same time, we need to check if `v->s == NULL` in\npopulate_value(), and strbuf cannot easily distinguish NULL and empty\nstrings, but c-style \"const char *\" can do it. So add a new member in\n`struct atom_value`: `s_size`, which can record raw object size, it\ncan help us add raw object data to the buffer or compare two buffers\nwhich contain raw object data.\n\nNote that `--format=%(raw)` cannot be used with `--python`, `--shell`,\n`--tcl`, and `--perl` because if the binary raw data is passed to a\nvariable in such languages, these may not support arbitrary binary data\nin their string variable type.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Bagas Sanjaya \nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Felipe Contreras \nHelped-by: Phillip Wood \nHelped-by: Junio C Hamano \nBased-on-patch-by: Olga Telezhnaya \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 9 ++\n ref-filter.c | 140 +++++++++++++++----\n t/t6300-for-each-ref.sh | 216 +++++++++++++++++++++++++++++\n 3 files changed, 338 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 2ae2478de70..cbb6f87d13f 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`,\n without angle brackets, and `:localpart` to get the part before the `@` symbol\n out of the trimmed email.\n \n+The raw data in an object is `raw`.\n+\n+raw:size::\n+\tThe raw data size of the object.\n+\n+Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n+`--perl` because such language may not support arbitrary binary data in their\n+string variable type.\n+\n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5cee6512fba..506fbc3d691 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -144,6 +144,7 @@ enum atom_type {\n \tATOM_BODY,\n \tATOM_TRAILERS,\n \tATOM_CONTENTS,\n+\tATOM_RAW,\n \tATOM_UPSTREAM,\n \tATOM_PUSH,\n \tATOM_SYMREF,\n@@ -189,6 +190,9 @@ static struct used_atom {\n \t\t\tstruct process_trailer_options trailer_opts;\n \t\t\tunsigned int nlines;\n \t\t} contents;\n+\t\tstruct {\n+\t\t\tenum { RAW_BARE, RAW_LENGTH } option;\n+\t\t} raw_data;\n \t\tstruct {\n \t\t\tcmp_status cmp_status;\n \t\t\tconst char *str;\n@@ -426,6 +430,18 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n+static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\tconst char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\tatom->u.raw_data.option = RAW_BARE;\n+\telse if (!strcmp(arg, \"size\"))\n+\t\tatom->u.raw_data.option = RAW_LENGTH;\n+\telse\n+\t\treturn strbuf_addf_ret(err, -1, _(\"unrecognized %%(raw) argument: %s\"), arg);\n+\treturn 0;\n+}\n+\n static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -586,6 +602,7 @@ static struct {\n \t[ATOM_BODY] = { \"body\", SOURCE_OBJ, FIELD_STR, body_atom_parser },\n \t[ATOM_TRAILERS] = { \"trailers\", SOURCE_OBJ, FIELD_STR, trailers_atom_parser },\n \t[ATOM_CONTENTS] = { \"contents\", SOURCE_OBJ, FIELD_STR, contents_atom_parser },\n+\t[ATOM_RAW] = { \"raw\", SOURCE_OBJ, FIELD_STR, raw_atom_parser },\n \t[ATOM_UPSTREAM] = { \"upstream\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_PUSH] = { \"push\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_SYMREF] = { \"symref\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -620,12 +637,15 @@ struct ref_formatting_state {\n \n struct atom_value {\n \tconst char *s;\n+\tsize_t s_size;\n \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t struct strbuf *err);\n \tuintmax_t value; /* used for sorting when not FIELD_STR */\n \tstruct used_atom *atom;\n };\n \n+#define ATOM_VALUE_S_SIZE_INIT (-1)\n+\n /*\n * Used to parse format string and sort specifiers\n */\n@@ -644,13 +664,6 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \t\treturn strbuf_addf_ret(err, -1, _(\"malformed field name: %.*s\"),\n \t\t\t\t (int)(ep-atom), atom);\n \n-\t/* Do we have the atom already used elsewhere? */\n-\tfor (i = 0; i < used_atom_cnt; i++) {\n-\t\tint len = strlen(used_atom[i].name);\n-\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n-\t\t\treturn i;\n-\t}\n-\n \t/*\n \t * If the atom name has a colon, strip it and everything after\n \t * it off - it specifies the format for this entry, and\n@@ -660,6 +673,13 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \targ = memchr(sp, ':', ep - sp);\n \tatom_len = (arg ? arg : ep) - sp;\n \n+\t/* Do we have the atom already used elsewhere? */\n+\tfor (i = 0; i < used_atom_cnt; i++) {\n+\t\tint len = strlen(used_atom[i].name);\n+\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n+\t\t\treturn i;\n+\t}\n+\n \t/* Is the atom a valid one? */\n \tfor (i = 0; i < ARRAY_SIZE(valid_atom); i++) {\n \t\tint len = strlen(valid_atom[i].name);\n@@ -709,11 +729,14 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \treturn at;\n }\n \n-static void quote_formatting(struct strbuf *s, const char *str, int quote_style)\n+static void quote_formatting(struct strbuf *s, const char *str, size_t len, int quote_style)\n {\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n-\t\tstrbuf_addstr(s, str);\n+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tstrbuf_add(s, str, len);\n+\t\telse\n+\t\t\tstrbuf_addstr(s, str);\n \t\tbreak;\n \tcase QUOTE_SHELL:\n \t\tsq_quote_buf(s, str);\n@@ -740,9 +763,12 @@ static int append_atom(struct atom_value *v, struct ref_formatting_state *state,\n \t * encountered.\n \t */\n \tif (!state->stack->prev)\n-\t\tquote_formatting(&state->stack->output, v->s, state->quote_style);\n+\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n \telse\n-\t\tstrbuf_addstr(&state->stack->output, v->s);\n+\t\tif (v->s_size != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n+\t\telse\n+\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n \treturn 0;\n }\n \n@@ -842,21 +868,23 @@ static int if_atom_handler(struct atom_value *atomv, struct ref_formatting_state\n \treturn 0;\n }\n \n-static int is_empty(const char *s)\n+static int is_empty(struct strbuf *buf)\n {\n-\twhile (*s != '\\0') {\n-\t\tif (!isspace(*s))\n-\t\t\treturn 0;\n-\t\ts++;\n-\t}\n-\treturn 1;\n-}\n+\tconst char *cur = buf->buf;\n+\tconst char *end = buf->buf + buf->len;\n+\n+\twhile (cur != end && (isspace(*cur)))\n+\t\tcur++;\n+\n+\treturn cur == end;\n+ }\n \n static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t\t struct strbuf *err)\n {\n \tstruct ref_formatting_stack *cur = state->stack;\n \tstruct if_then_else *if_then_else = NULL;\n+\tsize_t str_len = 0;\n \n \tif (cur->at_end == if_then_else_handler)\n \t\tif_then_else = (struct if_then_else *)cur->at_end_data;\n@@ -867,18 +895,22 @@ static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_sta\n \tif (if_then_else->else_atom_seen)\n \t\treturn strbuf_addf_ret(err, -1, _(\"format: %%(then) atom used after %%(else)\"));\n \tif_then_else->then_atom_seen = 1;\n+\tif (if_then_else->str)\n+\t\tstr_len = strlen(if_then_else->str);\n \t/*\n \t * If the 'equals' or 'notequals' attribute is used then\n \t * perform the required comparison. If not, only non-empty\n \t * strings satisfy the 'if' condition.\n \t */\n \tif (if_then_else->cmp_status == COMPARE_EQUAL) {\n-\t\tif (!strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len == cur->output.len &&\n+\t\t !memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n \t} else if (if_then_else->cmp_status == COMPARE_UNEQUAL) {\n-\t\tif (strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len != cur->output.len ||\n+\t\t memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n-\t} else if (cur->output.len && !is_empty(cur->output.buf))\n+\t} else if (cur->output.len && !is_empty(&cur->output))\n \t\tif_then_else->condition_satisfied = 1;\n \tstrbuf_reset(&cur->output);\n \treturn 0;\n@@ -924,7 +956,7 @@ static int end_atom_handler(struct atom_value *atomv, struct ref_formatting_stat\n \t * only on the topmost supporting atom.\n \t */\n \tif (!current->prev->prev) {\n-\t\tquote_formatting(&s, current->output.buf, state->quote_style);\n+\t\tquote_formatting(&s, current->output.buf, current->output.len, state->quote_style);\n \t\tstrbuf_swap(¤t->output, &s);\n \t}\n \tstrbuf_release(&s);\n@@ -974,6 +1006,10 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\t\t\tdie(_(\"--format=%.*s cannot be used with\"\n+\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\n@@ -1367,12 +1403,25 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp\n \t\tstruct used_atom *atom = &used_atom[i];\n \t\tconst char *name = atom->name;\n \t\tstruct atom_value *v = &val[i];\n+\t\tenum atom_type atom_type = atom->atom_type;\n \n \t\tif (!!deref != (*name == '*'))\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n \n+\t\tif (atom_type == ATOM_RAW) {\n+\t\t\tunsigned long buf_size = data->size;\n+\n+\t\t\tif (atom->u.raw_data.option == RAW_BARE) {\n+\t\t\t\tv->s = xmemdupz(buf, buf_size);\n+\t\t\t\tv->s_size = buf_size;\n+\t\t\t} else if (atom->u.raw_data.option == RAW_LENGTH) {\n+\t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX, (uintmax_t)buf_size);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif ((data->type != OBJ_TAG &&\n \t\t data->type != OBJ_COMMIT) ||\n \t\t (strcmp(name, \"body\") &&\n@@ -1460,9 +1509,11 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"Eh? Object of type %d?\", obj->type);\n@@ -1766,6 +1817,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n \n+\t\tv->s_size = ATOM_VALUE_S_SIZE_INIT;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n@@ -2369,6 +2421,19 @@ static int compare_detached_head(struct ref_array_item *a, struct ref_array_item\n \treturn 0;\n }\n \n+static int memcasecmp(const void *vs1, const void *vs2, size_t n)\n+{\n+\tconst char *s1 = vs1, *s2 = vs2;\n+\tconst char *end = s1 + n;\n+\n+\tfor (; s1 < end; s1++, s2++) {\n+\t\tint diff = tolower(*s1) - tolower(*s2);\n+\t\tif (diff)\n+\t\t\treturn diff;\n+\t}\n+\treturn 0;\n+}\n+\n static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b)\n {\n \tstruct atom_value *va, *vb;\n@@ -2389,10 +2454,30 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n \t} else if (s->sort_flags & REF_SORTING_VERSION) {\n \t\tcmp = versioncmp(va->s, vb->s);\n \t} else if (cmp_type == FIELD_STR) {\n-\t\tint (*cmp_fn)(const char *, const char *);\n-\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n-\t\t\t? strcasecmp : strcmp;\n-\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\tif (va->s_size == ATOM_VALUE_S_SIZE_INIT &&\n+\t\t vb->s_size == ATOM_VALUE_S_SIZE_INIT) {\n+\t\t\tint (*cmp_fn)(const char *, const char *);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? strcasecmp : strcmp;\n+\t\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\t} else {\n+\t\t\tsize_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT ?\n+\t\t\t\t\tstrlen(va->s) : va->s_size;\n+\t\t\tsize_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT ?\n+\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n+\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? memcasecmp : memcmp;\n+\n+\t\t\tcmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n+\t\t\t\t a_size : b_size);\n+\t\t\tif (!cmp) {\n+\t\t\t\tif (a_size > b_size)\n+\t\t\t\t\tcmp = 1;\n+\t\t\t\telse if (a_size < b_size)\n+\t\t\t\t\tcmp = -1;\n+\t\t\t}\n+\t\t}\n \t} else {\n \t\tif (va->value < vb->value)\n \t\t\tcmp = -1;\n@@ -2492,6 +2577,7 @@ int format_ref_array_item(struct ref_array_item *info,\n \t}\n \tif (format->need_color_reset_at_eol) {\n \t\tstruct atom_value resetv;\n+\t\tresetv.s_size = ATOM_VALUE_S_SIZE_INIT;\n \t\tresetv.s = GIT_COLOR_RESET;\n \t\tif (append_atom(&resetv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 9e0214076b4..18554f62d94 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -130,6 +130,8 @@ test_atom head parent:short=10 ''\n test_atom head numparent 0\n test_atom head object ''\n test_atom head type ''\n+test_atom head raw \"$(git cat-file commit refs/heads/main)\n+\"\n test_atom head '*objectname' ''\n test_atom head '*objecttype' ''\n test_atom head author 'A U Thor 1151968724 +0200'\n@@ -221,6 +223,15 @@ test_atom tag contents 'Tagging at 1151968727\n '\n test_atom tag HEAD ' '\n \n+test_expect_success 'basic atom: refs/tags/testtag *raw' '\n+\tgit cat-file commit refs/tags/testtag^{} >expected &&\n+\tgit for-each-ref --format=\"%(*raw)\" refs/tags/testtag >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'Check invalid atoms names are errors' '\n \ttest_must_fail git for-each-ref --format=\"%(INVALID)\" refs/heads\n '\n@@ -686,6 +697,15 @@ test_atom refs/tags/signed-empty contents:body ''\n test_atom refs/tags/signed-empty contents:signature \"$sig\"\n test_atom refs/tags/signed-empty contents \"$sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-empty raw' '\n+\tgit cat-file tag refs/tags/signed-empty >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-empty >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-short subject 'subject line'\n test_atom refs/tags/signed-short subject:sanitize 'subject-line'\n test_atom refs/tags/signed-short contents:subject 'subject line'\n@@ -695,6 +715,15 @@ test_atom refs/tags/signed-short contents:signature \"$sig\"\n test_atom refs/tags/signed-short contents \"subject line\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-short raw' '\n+\tgit cat-file tag refs/tags/signed-short >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-short >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-long subject 'subject line'\n test_atom refs/tags/signed-long subject:sanitize 'subject-line'\n test_atom refs/tags/signed-long contents:subject 'subject line'\n@@ -708,6 +737,15 @@ test_atom refs/tags/signed-long contents \"subject line\n body contents\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-long raw' '\n+\tgit cat-file tag refs/tags/signed-long >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-long >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'set up refs pointing to tree and blob' '\n \tgit update-ref refs/mytrees/first refs/heads/main^{tree} &&\n \tgit update-ref refs/myblobs/first refs/heads/main:one\n@@ -720,6 +758,16 @@ test_atom refs/mytrees/first contents:body \"\"\n test_atom refs/mytrees/first contents:signature \"\"\n test_atom refs/mytrees/first contents \"\"\n \n+test_expect_success 'basic atom: refs/mytrees/first raw' '\n+\tgit cat-file tree refs/mytrees/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/mytrees/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_atom refs/myblobs/first subject \"\"\n test_atom refs/myblobs/first contents:subject \"\"\n test_atom refs/myblobs/first body \"\"\n@@ -727,6 +775,174 @@ test_atom refs/myblobs/first contents:body \"\"\n test_atom refs/myblobs/first contents:signature \"\"\n test_atom refs/myblobs/first contents \"\"\n \n+test_expect_success 'basic atom: refs/myblobs/first raw' '\n+\tgit cat-file blob refs/myblobs/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/myblobs/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'set up refs pointing to binary blob' '\n+\tprintf \"a\\0b\\0c\" >blob1 &&\n+\tprintf \"a\\0c\\0b\" >blob2 &&\n+\tprintf \"\\0a\\0b\\0c\" >blob3 &&\n+\tprintf \"abc\" >blob4 &&\n+\tprintf \"\\0 \\0 \\0 \" >blob5 &&\n+\tprintf \"\\0 \\0a\\0 \" >blob6 &&\n+\tprintf \" \" >blob7 &&\n+\t>blob8 &&\n+\tobj=$(git hash-object -w blob1) &&\n+\tgit update-ref refs/myblobs/blob1 \"$obj\" &&\n+\tobj=$(git hash-object -w blob2) &&\n+\tgit update-ref refs/myblobs/blob2 \"$obj\" &&\n+\tobj=$(git hash-object -w blob3) &&\n+\tgit update-ref refs/myblobs/blob3 \"$obj\" &&\n+\tobj=$(git hash-object -w blob4) &&\n+\tgit update-ref refs/myblobs/blob4 \"$obj\" &&\n+\tobj=$(git hash-object -w blob5) &&\n+\tgit update-ref refs/myblobs/blob5 \"$obj\" &&\n+\tobj=$(git hash-object -w blob6) &&\n+\tgit update-ref refs/myblobs/blob6 \"$obj\" &&\n+\tobj=$(git hash-object -w blob7) &&\n+\tgit update-ref refs/myblobs/blob7 \"$obj\" &&\n+\tobj=$(git hash-object -w blob8) &&\n+\tgit update-ref refs/myblobs/blob8 \"$obj\"\n+'\n+\n+test_expect_success 'Verify sorts with raw' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob7\n+\trefs/mytrees/first\n+\trefs/myblobs/first\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob4\n+\trefs/heads/main\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'Verify sorts with raw:size' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\trefs/myblobs/blob7\n+\trefs/heads/main\n+\trefs/myblobs/blob4\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/mytrees/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw:size \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:equals)' '\n+\tcat >expected <<-EOF &&\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\trefs/myblobs/blob4\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:equals=abc)%(raw)%(then)%(refname)%(else)not equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:notequals)' '\n+\tcat >expected <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/heads/main\n+\trefs/heads/newtag\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\tequals\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob7\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:notequals=abc)%(raw)%(then)%(refname)%(else)equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'empty raw refs with %(if)' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob1 not empty\n+\trefs/myblobs/blob2 not empty\n+\trefs/myblobs/blob3 not empty\n+\trefs/myblobs/blob4 not empty\n+\trefs/myblobs/blob5 not empty\n+\trefs/myblobs/blob6 not empty\n+\trefs/myblobs/blob7 empty\n+\trefs/myblobs/blob8 empty\n+\trefs/myblobs/first not empty\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname) %(if)%(raw)%(then)not empty%(else)empty%(end)\" \\\n+\t\trefs/myblobs/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success '%(raw) with --python must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --python\n+'\n+\n+test_expect_success '%(raw) with --tcl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n+'\n+\n+test_expect_success '%(raw) with --perl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+'\n+\n+test_expect_success '%(raw) with --shell must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --shell\n+'\n+\n+test_expect_success '%(raw) with --shell and --sort=raw must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --sort=raw --shell\n+'\n+\n+test_expect_success '%(raw:size) with --shell' '\n+\tgit for-each-ref --format=\"%(raw:size)\" | while read line\n+\tdo\n+\t\techo \"'\\''$line'\\''\" >>expect\n+\tdone &&\n+\tgit for-each-ref --format=\"%(raw:size)\" --shell >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for-each-ref --format compare with cat-file --batch' '\n+\tgit rev-parse refs/mytrees/first | git cat-file --batch >expected &&\n+\tgit for-each-ref --format=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'set up multiple-sort tags' '\n \tfor when in 100000 200000\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"430884","messageId":"b6438fd4308f204dd3f7cf68a97702bd8d0edcb5.1626939557.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.git.1626939557.gitgitgadget@gmail.com","subject":"[PATCH 3/5] [GSOC] ref-filter: --format=%(raw) re-support --perl","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-22T07:39:14Z","receivedAt":"2021-07-22T07:39:38Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause the perl language can handle binary data correctly,\nadd the function perl_quote_buf_with_len(), which can specify\nthe length of the data and prevent the data from being truncated\nat '\\0' to help `--format=\"%(raw)\"` re-support `--perl`.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 4 ++--\n quote.c | 17 +++++++++++++++++\n quote.h | 1 +\n ref-filter.c | 15 +++++++++++----\n t/t6300-for-each-ref.sh | 19 +++++++++++++++++--\n 5 files changed, 48 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex cbb6f87d13f..6da899c6296 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -241,8 +241,8 @@ raw:size::\n \tThe raw data size of the object.\n \n Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n-`--perl` because such language may not support arbitrary binary data in their\n-string variable type.\n+because such language may not support arbitrary binary data in their string\n+variable type.\n \n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\ndiff --git a/quote.c b/quote.c\nindex 8a3a5e39eb1..26719d21d1e 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src)\n \tstrbuf_addch(sb, sq);\n }\n \n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len)\n+{\n+\tconst char sq = '\\'';\n+\tconst char bq = '\\\\';\n+\tconst char *c = src;\n+\tconst char *end = src + len;\n+\n+\tstrbuf_addch(sb, sq);\n+\twhile (c != end) {\n+\t\tif (*c == sq || *c == bq)\n+\t\t\tstrbuf_addch(sb, bq);\n+\t\tstrbuf_addch(sb, *c);\n+\t\tc++;\n+\t}\n+\tstrbuf_addch(sb, sq);\n+}\n+\n void python_quote_buf(struct strbuf *sb, const char *src)\n {\n \tconst char sq = '\\'';\ndiff --git a/quote.h b/quote.h\nindex 768cc6338e2..0fe69e264b0 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -94,6 +94,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne\n \n /* quoting as a string literal for other languages */\n void perl_quote_buf(struct strbuf *sb, const char *src);\n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len);\n void python_quote_buf(struct strbuf *sb, const char *src);\n void tcl_quote_buf(struct strbuf *sb, const char *src);\n void basic_regex_quote_buf(struct strbuf *sb, const char *src);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 506fbc3d691..ba9ab35d7ec 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -742,7 +742,10 @@ static void quote_formatting(struct strbuf *s, const char *str, size_t len, int\n \t\tsq_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PERL:\n-\t\tperl_quote_buf(s, str);\n+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tperl_quote_buf_with_len(s, str, len);\n+\t\telse\n+\t\t\tperl_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PYTHON:\n \t\tpython_quote_buf(s, str);\n@@ -1006,10 +1009,14 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n-\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n-\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\n+\t\tif ((format->quote_style == QUOTE_PYTHON ||\n+\t\t format->quote_style == QUOTE_SHELL ||\n+\t\t format->quote_style == QUOTE_TCL) &&\n+\t\t used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n \t\t\tdie(_(\"--format=%.*s cannot be used with\"\n-\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n+\t\t\t \"--python, --shell, --tcl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 18554f62d94..3d15d0a5360 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' '\n \ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n '\n \n-test_expect_success '%(raw) with --perl must fail' '\n-\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+test_expect_success '%(raw) with --perl' '\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob1 --perl | perl >actual &&\n+\tcmp blob1 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob3 --perl | perl >actual &&\n+\tcmp blob3 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob8 --perl | perl >actual &&\n+\tcmp blob8 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/first --perl | perl >actual &&\n+\tcmp one actual &&\n+\tgit cat-file tree refs/mytrees/first > expected &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/mytrees/first --perl | perl >actual &&\n+\tcmp expected actual\n '\n \n test_expect_success '%(raw) with --shell must fail' '\n-- \ngitgitgadget\n\n"},{"id":"430885","messageId":"33d332c664e267e0141a44014c00168d8849c0d3.1626939557.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.git.1626939557.gitgitgadget@gmail.com","subject":"[PATCH 4/5] [GSOC] ref-filter: use non-const ref_format in *_atom_parser()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-22T07:39:15Z","receivedAt":"2021-07-22T07:39:39Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse non-const ref_format in *_atom_parser(), which can help us\nmodify the members of ref_format in *_atom_parser().\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/tag.c | 2 +-\n ref-filter.c | 44 ++++++++++++++++++++++----------------------\n ref-filter.h | 4 ++--\n 3 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 82fcfc09824..452558ec957 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref,\n \t\t const struct object_id *oid, void *cb_data)\n {\n \tint flags;\n-\tconst struct ref_format *format = cb_data;\n+\tstruct ref_format *format = cb_data;\n \tflags = GPG_VERIFY_VERBOSE;\n \n \tif (format->format)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex ba9ab35d7ec..c8e561a3687 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -226,7 +226,7 @@ static int strbuf_addf_ret(struct strbuf *sb, int ret, const char *fmt, ...)\n \treturn ret;\n }\n \n-static int color_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int color_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *color_value, struct strbuf *err)\n {\n \tif (!color_value)\n@@ -264,7 +264,7 @@ static int refname_atom_parser_internal(struct refname_atom *atom, const char *a\n \treturn 0;\n }\n \n-static int remote_ref_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int remote_ref_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct string_list params = STRING_LIST_INIT_DUP;\n@@ -311,7 +311,7 @@ static int remote_ref_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objecttype_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -323,7 +323,7 @@ static int objecttype_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objectsize_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -343,7 +343,7 @@ static int objectsize_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int deltabase_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int deltabase_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -355,7 +355,7 @@ static int deltabase_atom_parser(const struct ref_format *format, struct used_at\n \treturn 0;\n }\n \n-static int body_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int body_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -364,7 +364,7 @@ static int body_atom_parser(const struct ref_format *format, struct used_atom *a\n \treturn 0;\n }\n \n-static int subject_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int subject_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -376,7 +376,7 @@ static int subject_atom_parser(const struct ref_format *format, struct used_atom\n \treturn 0;\n }\n \n-static int trailers_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int trailers_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tatom->u.contents.trailer_opts.no_divider = 1;\n@@ -402,7 +402,7 @@ static int trailers_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int contents_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int contents_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -430,7 +430,7 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int raw_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -442,7 +442,7 @@ static int raw_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int oid_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -461,7 +461,7 @@ static int oid_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int person_email_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int person_email_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -475,7 +475,7 @@ static int person_email_atom_parser(const struct ref_format *format, struct used\n \treturn 0;\n }\n \n-static int refname_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int refname_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \treturn refname_atom_parser_internal(&atom->u.refname, arg, atom->name, err);\n@@ -492,7 +492,7 @@ static align_type parse_align_position(const char *s)\n \treturn -1;\n }\n \n-static int align_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int align_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct align *align = &atom->u.align;\n@@ -544,7 +544,7 @@ static int align_atom_parser(const struct ref_format *format, struct used_atom *\n \treturn 0;\n }\n \n-static int if_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -559,7 +559,7 @@ static int if_atom_parser(const struct ref_format *format, struct used_atom *ato\n \treturn 0;\n }\n \n-static int head_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n \tatom->u.head = resolve_refdup(\"HEAD\", RESOLVE_REF_READING, NULL, NULL);\n@@ -570,7 +570,7 @@ static struct {\n \tconst char *name;\n \tinfo_source source;\n \tcmp_type cmp_type;\n-\tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n+\tint (*parser)(struct ref_format *format, struct used_atom *atom,\n \t\t const char *arg, struct strbuf *err);\n } valid_atom[] = {\n \t[ATOM_REFNAME] = { \"refname\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -649,7 +649,7 @@ struct atom_value {\n /*\n * Used to parse format string and sort specifiers\n */\n-static int parse_ref_filter_atom(const struct ref_format *format,\n+static int parse_ref_filter_atom(struct ref_format *format,\n \t\t\t\t const char *atom, const char *ep,\n \t\t\t\t struct strbuf *err)\n {\n@@ -2554,9 +2554,9 @@ static void append_literal(const char *cp, const char *ep, struct ref_formatting\n }\n \n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n-\t\t\t struct strbuf *final_buf,\n-\t\t\t struct strbuf *error_buf)\n+\t\t\t struct ref_format *format,\n+\t\t\t struct strbuf *final_buf,\n+\t\t\t struct strbuf *error_buf)\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n@@ -2601,7 +2601,7 @@ int format_ref_array_item(struct ref_array_item *info,\n }\n \n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format)\n+\t\t struct ref_format *format)\n {\n \tstruct ref_array_item *ref_item;\n \tstruct strbuf output = STRBUF_INIT;\ndiff --git a/ref-filter.h b/ref-filter.h\nindex baf72a71896..74fb423fc89 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);\n void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting, unsigned int mask, int on);\n /* Based on the given format and quote_style, fill the strbuf */\n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n+\t\t\t struct ref_format *format,\n \t\t\t struct strbuf *final_buf,\n \t\t\t struct strbuf *error_buf);\n /* Parse a single sort specifier and add it to the list */\n@@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void);\n * name must be a fully qualified refname.\n */\n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format);\n+\t\t struct ref_format *format);\n \n /*\n * Push a single ref onto the array; this can be used to construct your own\n-- \ngitgitgadget\n\n"},{"id":"430886","messageId":"75eb2f6740eb5845afcb7d31956cc5b3e3957f97.1626939557.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.git.1626939557.gitgitgadget@gmail.com","subject":"[PATCH 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-22T07:39:16Z","receivedAt":"2021-07-22T07:39:40Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nIn order to let \"cat-file --batch=%(rest)\" use the ref-filter\ninterface, add %(rest) atom for ref-filter. Introduce the\nreject_atom() to reject the atom %(rest) for \"git for-each-ref\",\n\"git branch\", \"git tag\" and \"git verify-tag\".\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 25 +++++++++++++++++++++++++\n ref-filter.h | 5 ++++-\n t/t3203-branch-output.sh | 4 ++++\n t/t6300-for-each-ref.sh | 4 ++++\n t/t7004-tag.sh | 4 ++++\n t/t7030-verify-tag.sh | 4 ++++\n 6 files changed, 45 insertions(+), 1 deletion(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex c8e561a3687..af8216dcd5b 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -157,6 +157,7 @@ enum atom_type {\n \tATOM_IF,\n \tATOM_THEN,\n \tATOM_ELSE,\n+\tATOM_REST,\n };\n \n /*\n@@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"%%(rest) does not take arguments\"));\n+\tformat->use_rest = 1;\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n@@ -615,6 +625,7 @@ static struct {\n \t[ATOM_IF] = { \"if\", SOURCE_NONE, FIELD_STR, if_atom_parser },\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n+\t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -989,6 +1000,11 @@ static const char *find_next(const char *cp)\n \treturn NULL;\n }\n \n+static int reject_atom(enum atom_type atom_type)\n+{\n+\treturn atom_type == ATOM_REST;\n+}\n+\n /*\n * Make sure the format string is well formed, and parse out\n * the used atoms.\n@@ -1009,6 +1025,8 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (reject_atom(used_atom[at].atom_type))\n+\t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n \t\t format->quote_style == QUOTE_SHELL ||\n@@ -1928,6 +1946,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_REST) {\n+\t\t\tif (ref->rest)\n+\t\t\t\tv->s = xstrdup(ref->rest);\n+\t\t\telse\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2145,6 +2169,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n+\tref->rest = NULL;\n \n \treturn ref;\n }\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 74fb423fc89..c15dee8d6b9 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -38,6 +38,7 @@ struct ref_sorting {\n \n struct ref_array_item {\n \tstruct object_id objectname;\n+\tconst char *rest;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n@@ -76,14 +77,16 @@ struct ref_format {\n \t * verify_ref_format() afterwards to finalize.\n \t */\n \tconst char *format;\n+\tconst char *rest;\n \tint quote_style;\n+\tint use_rest;\n \tint use_color;\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n };\n \n-#define REF_FORMAT_INIT { NULL, 0, -1 }\n+#define REF_FORMAT_INIT { .use_color = -1 }\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\ndiff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh\nindex 5325b9f67a0..6e94c6db7b5 100755\n--- a/t/t3203-branch-output.sh\n+++ b/t/t3203-branch-output.sh\n@@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git branch with --format=%(rest) must fail' '\n+\ttest_must_fail git branch --format=\"%(rest)\" >actual\n+'\n+\n test_expect_success 'worktree colors correct' '\n \tcat >expect <<-EOF &&\n \t* (HEAD detached from fromtag)\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 3d15d0a5360..0d2e062f791 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' '\n \ttest_cmp expect actual.clean\n '\n \n+test_expect_success 'basic atom: rest must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(rest)\" refs/heads/main\n+'\n+\n test_expect_success 'trailer parsing not fooled by --- line' '\n \tgit commit --allow-empty -F - <<-\\EOF &&\n \tthis is the subject\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 2f72c5c6883..082be85dffc 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git tag -l with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git tag -l --format=\"%(rest)\" \"v1*\"\n+'\n+\n test_expect_success \"set up color tests\" '\n \techo \"v1.0\" >expect.color &&\n \techo \"v1.0\" >expect.bare &&\ndiff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh\nindex 3cefde9602b..10faa645157 100755\n--- a/t/t7030-verify-tag.sh\n+++ b/t/t7030-verify-tag.sh\n@@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success GPG 'verifying tag with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git verify-tag --format=\"%(rest)\" \"fourth-signed\"\n+'\n+\n test_expect_success GPG 'verifying a forged tag with --format should fail silently' '\n \ttest_must_fail git verify-tag --format=\"tagname : %(tag)\" $(cat forged1.tag) >actual-forged &&\n \ttest_must_be_empty actual-forged\n-- \ngitgitgadget\n"},{"id":"430889","messageId":"CAP8UFD3ENBZdm8yQn6J-yb4PvUsrzTA3KGxes42dE3bTrYbr9Q@mail.gmail.com","threadId":"56142","inReplyTo":"pull.1000.git.1626939557.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/5] [GSOC] ref-filter: add %(raw) and %(rest) atoms","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-22T08:08:15Z","receivedAt":"2021-07-22T08:08:32Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Jul 22, 2021 at 9:39 AM ZheNing Hu via GitGitGadget\n wrote:\n>\n> This patch series is split from my main patch series zh/ref-filter-raw-data,\n> to make things easier for reviewers to read.\n\nThanks for doing that!\n\n> This patch series provided %(raw) and %(rest) for ref-filter, which will be\n> used by zh/cat-file-reuse-ref-filter-logic later.\n\nGreat!\n\nIt would be nice if you could also point to the previous times when\nyou sent this to the mailing list, so that previous reviews can more\neasily be found and maybe checked.\n\nAnother nice improvement would be to link to a branch on GitHub or\nGitLab, so that people could more easily fetch your work and try it.\n"},{"id":"430890","messageId":"877dhivkei.fsf@evledraar.gmail.com","threadId":"56142","inReplyTo":"ecd41b370e62cc88f1606569ec700eaca837fa1f.1626939557.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T08:06:21Z","receivedAt":"2021-07-22T08:10:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 22 2021, ZheNing Hu via GitGitGadget wrote:\n\n> [..] add a new member in\n> `struct atom_value`: `s_size`, which can record raw object size, it\n> can help us add raw object data to the buffer or compare two buffers\n> which contain raw object data.\n\nI didn't look in detail, just one nit/comment, no need to re-roll for this:\n\n\n> struct atom_value {\n> \tconst char *s;\n> +\tsize_t s_size;\n> \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n> \t\t struct strbuf *err);\n> \tuintmax_t value; /* used for sorting when not FIELD_STR */\n> \tstruct used_atom *atom;\n> };\n> \n> +#define ATOM_VALUE_S_SIZE_INIT (-1)\n> +\n\nThis and then later doing (this appears to be the only initialization?):\n\n> \tif (format->need_color_reset_at_eol) {\n> \t\tstruct atom_value resetv;\n> +\t\tresetv.s_size = ATOM_VALUE_S_SIZE_INIT;\n> \t\tresetv.s = GIT_COLOR_RESET;\n> \t\tif (append_atom(&resetv, &state, error_buf)) {\n> \t\t\tpop_stack_element(&state.stack);\n\nIs a rather unusual pattern, more typical would be:\n\n\tstruct atom_value { .... }\n\t#define ATOM_VALUE_INIT { \\\n\t\t.s_size = -1, \\\n\t}\n\nThen:\n\n\tstruct atom_value resetv = ATOM_VALUE_INIT;\n\nYou could keep that \"#define ATOM_VALUE_S_SIZE_INIT (-1)\" to check the\ninit value, personally I think it's just fine to hardcode the -1 literal\nand do \"size < 0\" checks, which we us as convention in a lot of other\nplaces for \"not initialized\".\n"},{"id":"430892","messageId":"CAOLTT8QSUX6xDontk2xorDkrFAFYC7hiG=nOukMHgjpGdKzXww@mail.gmail.com","threadId":"56142","inReplyTo":"CAP8UFD3ENBZdm8yQn6J-yb4PvUsrzTA3KGxes42dE3bTrYbr9Q@mail.gmail.com","subject":"Re: [PATCH 0/5] [GSOC] ref-filter: add %(raw) and %(rest) atoms","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-22T08:22:02Z","receivedAt":"2021-07-22T08:21:40Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年7月22日周四 下午4:08写道：\n>\n> On Thu, Jul 22, 2021 at 9:39 AM ZheNing Hu via GitGitGadget\n> wrote:\n> >\n> > This patch series is split from my main patch series zh/ref-filter-raw-data,\n> > to make things easier for reviewers to read.\n>\n> Thanks for doing that!\n>\n> > This patch series provided %(raw) and %(rest) for ref-filter, which will be\n> > used by zh/cat-file-reuse-ref-filter-logic later.\n>\n> Great!\n>\n> It would be nice if you could also point to the previous times when\n> you sent this to the mailing list, so that previous reviews can more\n> easily be found and maybe checked.\n>\n\nOkey, I will do it next time.\n\n> Another nice improvement would be to link to a branch on GitHub or\n> GitLab, so that people could more easily fetch your work and try it.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"430893","messageId":"874kcmvjr6.fsf@evledraar.gmail.com","threadId":"56142","inReplyTo":"75eb2f6740eb5845afcb7d31956cc5b3e3957f97.1626939557.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T08:11:02Z","receivedAt":"2021-07-22T08:24:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 22 2021, ZheNing Hu via GitGitGadget wrote:\n\n\n> +\t\t} else if (atom_type == ATOM_REST) {\n> +\t\t\tif (ref->rest)\n> +\t\t\t\tv->s = xstrdup(ref->rest);\n> +\t\t\telse\n> +\t\t\t\tv->s = xstrdup(\"\");\n> +\t\t\tcontinue;\n> \t\t} else\n> \t\t\tcontinue;\n\nAnother light reading nit, maybe more readable as:\n\n } else if (atom_type == ATOM_REST && ref->rest) {\n ...\n } else if (atom_type == ATOM_REST) {\n ...\n }\n continue;\n\nBut we can't do that \"continue\" at the end because there's two cases\nwhere we don't continue, I wondered if elimanting that special case\nwould make it easier, patch below. Probably not worth it & feel free to\nignore:\n\n ref-filter.c | 63 +++++++++++++++++++++++++-----------------------------------\n 1 file changed, 26 insertions(+), 37 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 81e77b13ad2..189244fed6f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1890,18 +1890,26 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tname++;\n \t\t}\n \n-\t\tif (atom_type == ATOM_REFNAME)\n+\t\tif (atom_type == ATOM_REFNAME) {\n \t\t\trefname = get_refname(atom, ref);\n-\t\telse if (atom_type == ATOM_WORKTREEPATH) {\n-\t\t\tif (ref->kind == FILTER_REFS_BRANCHES)\n-\t\t\t\tv->s = get_worktree_path(atom, ref);\n+\t\t\tif (!deref)\n+\t\t\t\tv->s = xstrdup(refname);\n \t\t\telse\n-\t\t\t\tv->s = xstrdup(\"\");\n-\t\t\tcontinue;\n-\t\t}\n-\t\telse if (atom_type == ATOM_SYMREF)\n+\t\t\t\tv->s = xstrfmt(\"%s^{}\", refname);\n+\t\t\tfree((char *)refname);\n+\t\t} else if (atom_type == ATOM_WORKTREEPATH &&\n+\t\t\t ref->kind == FILTER_REFS_BRANCHES) {\n+\t\t\tv->s = get_worktree_path(atom, ref);\n+\t\t} else if (atom_type == ATOM_WORKTREEPATH) {\n+\t\t\tv->s = xstrdup(\"\");\n+\t\t} else if (atom_type == ATOM_SYMREF) {\n \t\t\trefname = get_symref(atom, ref);\n-\t\telse if (atom_type == ATOM_UPSTREAM) {\n+\t\t\tif (!deref)\n+\t\t\t\tv->s = xstrdup(refname);\n+\t\t\telse\n+\t\t\t\tv->s = xstrfmt(\"%s^{}\", refname);\n+\t\t\tfree((char *)refname);\n+\t\t} else if (atom_type == ATOM_UPSTREAM) {\n \t\t\tconst char *branch_name;\n \t\t\t/* only local branches may have an upstream */\n \t\t\tif (!skip_prefix(ref->refname, \"refs/heads/\",\n@@ -1916,7 +1924,6 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\t\tfill_remote_ref_details(atom, refname, branch, &v->s);\n \t\t\telse\n \t\t\t\tv->s = xstrdup(\"\");\n-\t\t\tcontinue;\n \t\t} else if (atom_type == ATOM_PUSH && atom->u.remote_ref.push) {\n \t\t\tconst char *branch_name;\n \t\t\tv->s = xstrdup(\"\");\n@@ -1935,10 +1942,8 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\t/* We will definitely re-init v->s on the next line. */\n \t\t\tfree((char *)v->s);\n \t\t\tfill_remote_ref_details(atom, refname, branch, &v->s);\n-\t\t\tcontinue;\n \t\t} else if (atom_type == ATOM_COLOR) {\n \t\t\tv->s = xstrdup(atom->u.color);\n-\t\t\tcontinue;\n \t\t} else if (atom_type == ATOM_FLAG) {\n \t\t\tchar buf[256], *cp = buf;\n \t\t\tif (ref->flag & REF_ISSYMREF)\n@@ -1951,24 +1956,20 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\t\t*cp = '\\0';\n \t\t\t\tv->s = xstrdup(buf + 1);\n \t\t\t}\n-\t\t\tcontinue;\n \t\t} else if (!deref && atom_type == ATOM_OBJECTNAME) {\n \t\t\t v->s = xstrdup(do_grab_oid(\"objectname\", &ref->objectname, atom));\n-\t\t\t continue;\n+\t\t} else if (atom_type == ATOM_HEAD &&\n+\t\t\t atom->u.head &&\n+\t\t\t !strcmp(ref->refname, atom->u.head)) {\n+\t\t\tv->s = xstrdup(\"*\");\n \t\t} else if (atom_type == ATOM_HEAD) {\n-\t\t\tif (atom->u.head && !strcmp(ref->refname, atom->u.head))\n-\t\t\t\tv->s = xstrdup(\"*\");\n-\t\t\telse\n-\t\t\t\tv->s = xstrdup(\" \");\n-\t\t\tcontinue;\n+\t\t\tv->s = xstrdup(\" \");\n \t\t} else if (atom_type == ATOM_ALIGN) {\n \t\t\tv->handler = align_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n-\t\t\tcontinue;\n \t\t} else if (atom_type == ATOM_END) {\n \t\t\tv->handler = end_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n-\t\t\tcontinue;\n \t\t} else if (atom_type == ATOM_IF) {\n \t\t\tconst char *s;\n \t\t\tif (skip_prefix(name, \"if:\", &s))\n@@ -1976,29 +1977,17 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\telse\n \t\t\t\tv->s = xstrdup(\"\");\n \t\t\tv->handler = if_atom_handler;\n-\t\t\tcontinue;\n \t\t} else if (atom_type == ATOM_THEN) {\n \t\t\tv->handler = then_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n-\t\t\tcontinue;\n \t\t} else if (atom_type == ATOM_ELSE) {\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n-\t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_REST && ref->rest) {\n+\t\t\tv->s = xstrdup(ref->rest);\n \t\t} else if (atom_type == ATOM_REST) {\n-\t\t\tif (ref->rest)\n-\t\t\t\tv->s = xstrdup(ref->rest);\n-\t\t\telse\n-\t\t\t\tv->s = xstrdup(\"\");\n-\t\t\tcontinue;\n-\t\t} else\n-\t\t\tcontinue;\n-\n-\t\tif (!deref)\n-\t\t\tv->s = xstrdup(refname);\n-\t\telse\n-\t\t\tv->s = xstrfmt(\"%s^{}\", refname);\n-\t\tfree((char *)refname);\n+\t\t\tv->s = xstrdup(\"\");\n+\t\t}\n \t}\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n"},{"id":"430894","messageId":"CAOLTT8TP4z1vR8i9DX09+oBiPVgaF1isTgD2ZQ4nfoBJwTd9_w@mail.gmail.com","threadId":"56142","inReplyTo":"877dhivkei.fsf@evledraar.gmail.com","subject":"Re: [PATCH 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-22T08:31:18Z","receivedAt":"2021-07-22T08:30:56Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Ævar Arnfjörð Bjarmason 于2021年7月22日周四 下午4:10写道：\n\n>\n>\n> On Thu, Jul 22 2021, ZheNing Hu via GitGitGadget wrote:\n>\n> > [..] add a new member in\n> > `struct atom_value`: `s_size`, which can record raw object size, it\n> > can help us add raw object data to the buffer or compare two buffers\n> > which contain raw object data.\n>\n> I didn't look in detail, just one nit/comment, no need to re-roll for this:\n>\n>\n> > struct atom_value {\n> > const char *s;\n> > + size_t s_size;\n> > int (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n> > struct strbuf *err);\n> > uintmax_t value; /* used for sorting when not FIELD_STR */\n> > struct used_atom *atom;\n> > };\n> >\n> > +#define ATOM_VALUE_S_SIZE_INIT (-1)\n> > +\n>\n> This and then later doing (this appears to be the only initialization?):\n\nYes, \"s_size = -1” is the only initialization.\n\n>\n> > if (format->need_color_reset_at_eol) {\n> > struct atom_value resetv;\n> > + resetv.s_size = ATOM_VALUE_S_SIZE_INIT;\n> > resetv.s = GIT_COLOR_RESET;\n> > if (append_atom(&resetv, &state, error_buf)) {\n> > pop_stack_element(&state.stack);\n>\n> Is a rather unusual pattern, more typical would be:\n>\n> struct atom_value { .... }\n> #define ATOM_VALUE_INIT { \\\n> .s_size = -1, \\\n> }\n>\n> Then:\n>\n> struct atom_value resetv = ATOM_VALUE_INIT;\n>\n> You could keep that \"#define ATOM_VALUE_S_SIZE_INIT (-1)\" to check the\n> init value, personally I think it's just fine to hardcode the -1 literal\n> and do \"size < 0\" checks, which we us as convention in a lot of other\n> places for \"not initialized\".\n\nMaybe you are right, hard-coding \"-1\" is ok.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"430897","messageId":"CAP8UFD3nmnu7fEOG75O46qEKtFHppbeZy3Y-NkPTCgphM5MPqA@mail.gmail.com","threadId":"56142","inReplyTo":"a7b4e45f14a27bd67172e24660f448e4c9cce5d8.1626939557.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-22T08:51:36Z","receivedAt":"2021-07-22T08:51:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Jul 22, 2021 at 9:39 AM ZheNing Hu via GitGitGadget\n wrote:\n>\n> From: ZheNing Hu \n>\n> Only tag and commit objects use `grab_sub_body_contents()` to grab\n> object contents in the current codebase. We want to teach the\n> function to also handle blobs and trees to get their raw data,\n> without parsing a blob (whose contents looks like a commit or a tag)\n> incorrectly as a commit or a tag.\n>\n> Skip the block of code that is specific to handling commits and tags\n> early when the given object is of a wrong type to help later\n> addition to handle other types of objects in this function.\n\nSmall nit in case the series is rerolled: it looks like this commit\nwould change only `grab_sub_body_contents()`, but there are a few\nchanges to grab_values() too. Maybe you could add that it's needed to\npass a `struct expand_data *data` instead of only `void *buf` to both\n`grab_sub_body_contents()` and `grab_values()` to be able to check the\nobject type.\n\n\n> Mentored-by: Christian Couder \n> Mentored-by: Hariom Verma \n> Helped-by: Junio C Hamano \n> Signed-off-by: ZheNing Hu \n> ---\n> ref-filter.c | 24 +++++++++++++++---------\n> 1 file changed, 15 insertions(+), 9 deletions(-)\n>\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 4db0e40ff4c..5cee6512fba 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n> }\n>\n> /* See grab_values */\n> -static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n> +static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n> {\n> int i;\n> const char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n> size_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n> + void *buf = data->content;\n>\n> for (i = 0; i < used_atom_cnt; i++) {\n> struct used_atom *atom = &used_atom[i];\n> @@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n> continue;\n> if (deref)\n> name++;\n> - if (strcmp(name, \"body\") &&\n> - !starts_with(name, \"subject\") &&\n> - !starts_with(name, \"trailers\") &&\n> - !starts_with(name, \"contents\"))\n> +\n> + if ((data->type != OBJ_TAG &&\n> + data->type != OBJ_COMMIT) ||\n> + (strcmp(name, \"body\") &&\n> + !starts_with(name, \"subject\") &&\n> + !starts_with(name, \"trailers\") &&\n> + !starts_with(name, \"contents\")))\n> continue;\n> if (!subpos)\n> find_subpos(buf,\n> @@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n> * pointed at by the ref itself; otherwise it is the object the\n> * ref (which is a tag) refers to.\n> */\n> -static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n> +static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n> {\n> + void *buf = data->content;\n> +\n> switch (obj->type) {\n> case OBJ_TAG:\n> grab_tag_values(val, deref, obj);\n> - grab_sub_body_contents(val, deref, buf);\n> + grab_sub_body_contents(val, deref, data);\n> grab_person(\"tagger\", val, deref, buf);\n> break;\n> case OBJ_COMMIT:\n> grab_commit_values(val, deref, obj);\n> - grab_sub_body_contents(val, deref, buf);\n> + grab_sub_body_contents(val, deref, data);\n> grab_person(\"author\", val, deref, buf);\n> grab_person(\"committer\", val, deref, buf);\n> break;\n> @@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n> return strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n> oid_to_hex(&oi->oid), ref->refname);\n> }\n> - grab_values(ref->value, deref, *obj, oi->content);\n> + grab_values(ref->value, deref, *obj, oi);\n> }\n>\n> grab_common_values(ref->value, deref, oi);\n> --\n> gitgitgadget\n>\n"},{"id":"430898","messageId":"CAOLTT8RULMLiUwRDMRO-W6gzB8+57XHT=Hn3owoQF-fyK8_HMw@mail.gmail.com","threadId":"56142","inReplyTo":"874kcmvjr6.fsf@evledraar.gmail.com","subject":"Re: [PATCH 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-22T09:10:47Z","receivedAt":"2021-07-22T09:10:28Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Ævar Arnfjörð Bjarmason 于2021年7月22日周四 下午4:24写道：\n>\n>\n> On Thu, Jul 22 2021, ZheNing Hu via GitGitGadget wrote:\n>\n>\n> > + } else if (atom_type == ATOM_REST) {\n> > + if (ref->rest)\n> > + v->s = xstrdup(ref->rest);\n> > + else\n> > + v->s = xstrdup(\"\");\n> > + continue;\n> > } else\n> > continue;\n>\n> Another light reading nit, maybe more readable as:\n>\n> } else if (atom_type == ATOM_REST && ref->rest) {\n> ...\n> } else if (atom_type == ATOM_REST) {\n> ...\n> }\n> continue;\n>\n> But we can't do that \"continue\" at the end because there's two cases\n> where we don't continue, I wondered if elimanting that special case\n> would make it easier, patch below. Probably not worth it & feel free to\n> ignore:\n\nYeah, the readability here is really bad, so many \"continue\" just to not execute\npart of the operation on refname. In fact, I have a similar patch\n(which will not\nbe sent to the mailing list for the time being), it use switch and\ncase to replace if\nand else. [1]\n\n>\n> ref-filter.c | 63 +++++++++++++++++++++++++-----------------------------------\n> 1 file changed, 26 insertions(+), 37 deletions(-)\n>\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 81e77b13ad2..189244fed6f 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -1890,18 +1890,26 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n> name++;\n> }\n>\n> - if (atom_type == ATOM_REFNAME)\n> + if (atom_type == ATOM_REFNAME) {\n> refname = get_refname(atom, ref);\n> - else if (atom_type == ATOM_WORKTREEPATH) {\n> - if (ref->kind == FILTER_REFS_BRANCHES)\n> - v->s = get_worktree_path(atom, ref);\n> + if (!deref)\n> + v->s = xstrdup(refname);\n> else\n> - v->s = xstrdup(\"\");\n> - continue;\n> - }\n> - else if (atom_type == ATOM_SYMREF)\n> + v->s = xstrfmt(\"%s^{}\", refname);\n> + free((char *)refname);\n> + } else if (atom_type == ATOM_WORKTREEPATH &&\n> + ref->kind == FILTER_REFS_BRANCHES) {\n> + v->s = get_worktree_path(atom, ref);\n> + } else if (atom_type == ATOM_WORKTREEPATH) {\n> + v->s = xstrdup(\"\");\n> + } else if (atom_type == ATOM_SYMREF) {\n> refname = get_symref(atom, ref);\n> - else if (atom_type == ATOM_UPSTREAM) {\n> + if (!deref)\n> + v->s = xstrdup(refname);\n> + else\n> + v->s = xstrfmt(\"%s^{}\", refname);\n> + free((char *)refname);\n> + } else if (atom_type == ATOM_UPSTREAM) {\n> const char *branch_name;\n> /* only local branches may have an upstream */\n> if (!skip_prefix(ref->refname, \"refs/heads/\",\n> @@ -1916,7 +1924,6 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n> fill_remote_ref_details(atom, refname, branch, &v->s);\n> else\n> v->s = xstrdup(\"\");\n> - continue;\n> } else if (atom_type == ATOM_PUSH && atom->u.remote_ref.push) {\n> const char *branch_name;\n> v->s = xstrdup(\"\");\n> @@ -1935,10 +1942,8 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n> /* We will definitely re-init v->s on the next line. */\n> free((char *)v->s);\n> fill_remote_ref_details(atom, refname, branch, &v->s);\n> - continue;\n> } else if (atom_type == ATOM_COLOR) {\n> v->s = xstrdup(atom->u.color);\n> - continue;\n> } else if (atom_type == ATOM_FLAG) {\n> char buf[256], *cp = buf;\n> if (ref->flag & REF_ISSYMREF)\n> @@ -1951,24 +1956,20 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n> *cp = '\\0';\n> v->s = xstrdup(buf + 1);\n> }\n> - continue;\n> } else if (!deref && atom_type == ATOM_OBJECTNAME) {\n> v->s = xstrdup(do_grab_oid(\"objectname\", &ref->objectname, atom));\n> - continue;\n> + } else if (atom_type == ATOM_HEAD &&\n> + atom->u.head &&\n> + !strcmp(ref->refname, atom->u.head)) {\n> + v->s = xstrdup(\"*\");\n> } else if (atom_type == ATOM_HEAD) {\n> - if (atom->u.head && !strcmp(ref->refname, atom->u.head))\n> - v->s = xstrdup(\"*\");\n> - else\n> - v->s = xstrdup(\" \");\n> - continue;\n> + v->s = xstrdup(\" \");\n> } else if (atom_type == ATOM_ALIGN) {\n> v->handler = align_atom_handler;\n> v->s = xstrdup(\"\");\n> - continue;\n> } else if (atom_type == ATOM_END) {\n> v->handler = end_atom_handler;\n> v->s = xstrdup(\"\");\n> - continue;\n> } else if (atom_type == ATOM_IF) {\n> const char *s;\n> if (skip_prefix(name, \"if:\", &s))\n> @@ -1976,29 +1977,17 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n> else\n> v->s = xstrdup(\"\");\n> v->handler = if_atom_handler;\n> - continue;\n> } else if (atom_type == ATOM_THEN) {\n> v->handler = then_atom_handler;\n> v->s = xstrdup(\"\");\n> - continue;\n> } else if (atom_type == ATOM_ELSE) {\n> v->handler = else_atom_handler;\n> v->s = xstrdup(\"\");\n> - continue;\n> + } else if (atom_type == ATOM_REST && ref->rest) {\n> + v->s = xstrdup(ref->rest);\n> } else if (atom_type == ATOM_REST) {\n> - if (ref->rest)\n> - v->s = xstrdup(ref->rest);\n> - else\n> - v->s = xstrdup(\"\");\n> - continue;\n> - } else\n> - continue;\n> -\n> - if (!deref)\n> - v->s = xstrdup(refname);\n> - else\n> - v->s = xstrfmt(\"%s^{}\", refname);\n> - free((char *)refname);\n> + v->s = xstrdup(\"\");\n> + }\n> }\n>\n> for (i = 0; i < used_atom_cnt; i++) {\n\n[1]: https://github.com/adlternative/git/commit/bbc6ecba4c0f65e7328e571b0d38ff99f800dc09\n"},{"id":"430900","messageId":"CAOLTT8SNgbJRnmtfL4J=ypo1Q98gY9DE3g6BjBnX895tkMLcng@mail.gmail.com","threadId":"56142","inReplyTo":"CAP8UFD3nmnu7fEOG75O46qEKtFHppbeZy3Y-NkPTCgphM5MPqA@mail.gmail.com","subject":"Re: [PATCH 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-22T09:15:15Z","receivedAt":"2021-07-22T09:14:54Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年7月22日周四 下午4:51写道：\n>\n> On Thu, Jul 22, 2021 at 9:39 AM ZheNing Hu via GitGitGadget\n> wrote:\n> >\n> > From: ZheNing Hu \n> >\n> > Only tag and commit objects use `grab_sub_body_contents()` to grab\n> > object contents in the current codebase. We want to teach the\n> > function to also handle blobs and trees to get their raw data,\n> > without parsing a blob (whose contents looks like a commit or a tag)\n> > incorrectly as a commit or a tag.\n> >\n> > Skip the block of code that is specific to handling commits and tags\n> > early when the given object is of a wrong type to help later\n> > addition to handle other types of objects in this function.\n>\n> Small nit in case the series is rerolled: it looks like this commit\n> would change only `grab_sub_body_contents()`, but there are a few\n> changes to grab_values() too. Maybe you could add that it's needed to\n> pass a `struct expand_data *data` instead of only `void *buf` to both\n> `grab_sub_body_contents()` and `grab_values()` to be able to check the\n> object type.\n>\nIndeed so. I will add them.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"430997","messageId":"pull.1000.v2.git.1627031043.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.git.1626939557.gitgitgadget@gmail.com","subject":"[PATCH v2 0/5] [GSOC] ref-filter: add %(raw) and %(rest) atoms","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T09:03:58Z","receivedAt":"2021-07-23T09:04:08Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"This patch series is split from my main patch series zh/ref-filter-raw-data,\nto make things easier for reviewers to read. It's last version is here:\nhttps://lore.kernel.org/git/pull.993.v2.git.1626363626.gitgitgadget@gmail.com/\n\nThis patch series provided %(raw) and %(rest) for ref-filter, which will be\nused by zh/cat-file-reuse-ref-filter-logic later.\n\nChange from last version:\n\n 1. Define ATOM_VALUE_INIT used to initialize struct atom_value and delete\n ATOM_VALUE_S_SIZE_INIT macro.\n 2. Provide more complete commit message.\n\nZheNing Hu (5):\n [GSOC] ref-filter: add obj-type check in grab contents\n [GSOC] ref-filter: add %(raw) atom\n [GSOC] ref-filter: --format=%(raw) re-support --perl\n [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n [GSOC] ref-filter: add %(rest) atom\n\n Documentation/git-for-each-ref.txt | 9 ++\n builtin/tag.c | 2 +-\n quote.c | 17 ++\n quote.h | 1 +\n ref-filter.c | 242 ++++++++++++++++++++++-------\n ref-filter.h | 9 +-\n t/t3203-branch-output.sh | 4 +\n t/t6300-for-each-ref.sh | 235 ++++++++++++++++++++++++++++\n t/t7004-tag.sh | 4 +\n t/t7030-verify-tag.sh | 4 +\n 10 files changed, 464 insertions(+), 63 deletions(-)\n\n\nbase-commit: daab8a564f8bbac55f70f8bf86c070e001a9b006\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1000%2Fadlternative%2Fref-filter-raw-data-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1000/adlternative/ref-filter-raw-data-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1000\n\nRange-diff vs v1:\n\n 1: a7b4e45f14a ! 1: d77268bb188 [GSOC] ref-filter: add obj-type check in grab contents\n @@ Commit message\n object contents in the current codebase. We want to teach the\n function to also handle blobs and trees to get their raw data,\n without parsing a blob (whose contents looks like a commit or a tag)\n - incorrectly as a commit or a tag.\n + incorrectly as a commit or a tag. So it's needed to pass a\n + `struct expand_data *data` instread of only `void *buf` to both\n + `grab_sub_body_contents()` and `grab_values()` to be able to check\n + the object type.\n \n Skip the block of code that is specific to handling commits and tags\n early when the given object is of a wrong type to help later\n 2: ecd41b370e6 ! 2: eafb79bad62 [GSOC] ref-filter: add %(raw) atom\n @@ ref-filter.c: struct ref_formatting_state {\n \tstruct used_atom *atom;\n };\n \n -+#define ATOM_VALUE_S_SIZE_INIT (-1)\n ++#define ATOM_VALUE_INIT { \\\n ++\t.s_size = -1 \\\n ++}\n +\n /*\n * Used to parse format string and sort specifiers\n @@ ref-filter.c: static int parse_ref_filter_atom(const struct ref_format *format,\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n -\t\tstrbuf_addstr(s, str);\n -+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n ++\t\tif (len != -1)\n +\t\t\tstrbuf_add(s, str, len);\n +\t\telse\n +\t\t\tstrbuf_addstr(s, str);\n @@ ref-filter.c: static int append_atom(struct atom_value *v, struct ref_formatting\n +\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n \telse\n -\t\tstrbuf_addstr(&state->stack->output, v->s);\n -+\t\tif (v->s_size != ATOM_VALUE_S_SIZE_INIT)\n ++\t\tif (v->s_size != -1)\n +\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n +\t\telse\n +\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n @@ ref-filter.c: static int populate_value(struct ref_array_item *ref, struct strbu\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n \n -+\t\tv->s_size = ATOM_VALUE_S_SIZE_INIT;\n ++\t\tv->s_size = -1;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n @@ ref-filter.c: static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array\n -\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n -\t\t\t? strcasecmp : strcmp;\n -\t\tcmp = cmp_fn(va->s, vb->s);\n -+\t\tif (va->s_size == ATOM_VALUE_S_SIZE_INIT &&\n -+\t\t vb->s_size == ATOM_VALUE_S_SIZE_INIT) {\n ++\t\tif (va->s_size == -1 && vb->s_size == -1) {\n +\t\t\tint (*cmp_fn)(const char *, const char *);\n +\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n +\t\t\t\t? strcasecmp : strcmp;\n +\t\t\tcmp = cmp_fn(va->s, vb->s);\n +\t\t} else {\n -+\t\t\tsize_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT ?\n ++\t\t\tsize_t a_size = va->s_size == -1 ?\n +\t\t\t\t\tstrlen(va->s) : va->s_size;\n -+\t\t\tsize_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT ?\n ++\t\t\tsize_t b_size = vb->s_size == -1 ?\n +\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n +\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n +\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n @@ ref-filter.c: static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array\n \t\tif (va->value < vb->value)\n \t\t\tcmp = -1;\n @@ ref-filter.c: int format_ref_array_item(struct ref_array_item *info,\n + \t\tappend_literal(cp, sp, &state);\n \t}\n \tif (format->need_color_reset_at_eol) {\n - \t\tstruct atom_value resetv;\n -+\t\tresetv.s_size = ATOM_VALUE_S_SIZE_INIT;\n +-\t\tstruct atom_value resetv;\n ++\t\tstruct atom_value resetv = ATOM_VALUE_INIT;\n \t\tresetv.s = GIT_COLOR_RESET;\n \t\tif (append_atom(&resetv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\n 3: b6438fd4308 ! 3: 00a29dd16d7 [GSOC] ref-filter: --format=%(raw) re-support --perl\n @@ ref-filter.c: static void quote_formatting(struct strbuf *s, const char *str, si\n \t\tbreak;\n \tcase QUOTE_PERL:\n -\t\tperl_quote_buf(s, str);\n -+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n ++\t\tif (len != -1)\n +\t\t\tperl_quote_buf_with_len(s, str, len);\n +\t\telse\n +\t\t\tperl_quote_buf(s, str);\n 4: 33d332c664e = 4: 3a8173b42ed [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n 5: 75eb2f6740e = 5: 71249dfd4e0 [GSOC] ref-filter: add %(rest) atom\n\n-- \ngitgitgadget\n"},{"id":"430998","messageId":"d77268bb1882a6cfbea740053baac5546ca5c5f1.1627031043.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v2.git.1627031043.gitgitgadget@gmail.com","subject":"[PATCH v2 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T09:03:59Z","receivedAt":"2021-07-23T09:04:09Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nOnly tag and commit objects use `grab_sub_body_contents()` to grab\nobject contents in the current codebase. We want to teach the\nfunction to also handle blobs and trees to get their raw data,\nwithout parsing a blob (whose contents looks like a commit or a tag)\nincorrectly as a commit or a tag. So it's needed to pass a\n`struct expand_data *data` instread of only `void *buf` to both\n`grab_sub_body_contents()` and `grab_values()` to be able to check\nthe object type.\n\nSkip the block of code that is specific to handling commits and tags\nearly when the given object is of a wrong type to help later\naddition to handle other types of objects in this function.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Junio C Hamano \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4db0e40ff4c..5cee6512fba 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n+static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n {\n \tint i;\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n+\tvoid *buf = data->content;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n@@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (strcmp(name, \"body\") &&\n-\t\t !starts_with(name, \"subject\") &&\n-\t\t !starts_with(name, \"trailers\") &&\n-\t\t !starts_with(name, \"contents\"))\n+\n+\t\tif ((data->type != OBJ_TAG &&\n+\t\t data->type != OBJ_COMMIT) ||\n+\t\t (strcmp(name, \"body\") &&\n+\t\t !starts_with(name, \"subject\") &&\n+\t\t !starts_with(name, \"trailers\") &&\n+\t\t !starts_with(name, \"contents\")))\n \t\t\tcontinue;\n \t\tif (!subpos)\n \t\t\tfind_subpos(buf,\n@@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n+static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n {\n+\tvoid *buf = data->content;\n+\n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n@@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi->content);\n+\t\tgrab_values(ref->value, deref, *obj, oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n-- \ngitgitgadget\n\n"},{"id":"430999","messageId":"00a29dd16d7c5e176e560bec1dc335b1eb6ccaf8.1627031043.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v2.git.1627031043.gitgitgadget@gmail.com","subject":"[PATCH v2 3/5] [GSOC] ref-filter: --format=%(raw) re-support --perl","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T09:04:01Z","receivedAt":"2021-07-23T09:04:12Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause the perl language can handle binary data correctly,\nadd the function perl_quote_buf_with_len(), which can specify\nthe length of the data and prevent the data from being truncated\nat '\\0' to help `--format=\"%(raw)\"` re-support `--perl`.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 4 ++--\n quote.c | 17 +++++++++++++++++\n quote.h | 1 +\n ref-filter.c | 15 +++++++++++----\n t/t6300-for-each-ref.sh | 19 +++++++++++++++++--\n 5 files changed, 48 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex cbb6f87d13f..6da899c6296 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -241,8 +241,8 @@ raw:size::\n \tThe raw data size of the object.\n \n Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n-`--perl` because such language may not support arbitrary binary data in their\n-string variable type.\n+because such language may not support arbitrary binary data in their string\n+variable type.\n \n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\ndiff --git a/quote.c b/quote.c\nindex 8a3a5e39eb1..26719d21d1e 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src)\n \tstrbuf_addch(sb, sq);\n }\n \n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len)\n+{\n+\tconst char sq = '\\'';\n+\tconst char bq = '\\\\';\n+\tconst char *c = src;\n+\tconst char *end = src + len;\n+\n+\tstrbuf_addch(sb, sq);\n+\twhile (c != end) {\n+\t\tif (*c == sq || *c == bq)\n+\t\t\tstrbuf_addch(sb, bq);\n+\t\tstrbuf_addch(sb, *c);\n+\t\tc++;\n+\t}\n+\tstrbuf_addch(sb, sq);\n+}\n+\n void python_quote_buf(struct strbuf *sb, const char *src)\n {\n \tconst char sq = '\\'';\ndiff --git a/quote.h b/quote.h\nindex 768cc6338e2..0fe69e264b0 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -94,6 +94,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne\n \n /* quoting as a string literal for other languages */\n void perl_quote_buf(struct strbuf *sb, const char *src);\n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len);\n void python_quote_buf(struct strbuf *sb, const char *src);\n void tcl_quote_buf(struct strbuf *sb, const char *src);\n void basic_regex_quote_buf(struct strbuf *sb, const char *src);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex a713507d783..0ff8574a00c 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -744,7 +744,10 @@ static void quote_formatting(struct strbuf *s, const char *str, size_t len, int\n \t\tsq_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PERL:\n-\t\tperl_quote_buf(s, str);\n+\t\tif (len != -1)\n+\t\t\tperl_quote_buf_with_len(s, str, len);\n+\t\telse\n+\t\t\tperl_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PYTHON:\n \t\tpython_quote_buf(s, str);\n@@ -1008,10 +1011,14 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n-\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n-\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\n+\t\tif ((format->quote_style == QUOTE_PYTHON ||\n+\t\t format->quote_style == QUOTE_SHELL ||\n+\t\t format->quote_style == QUOTE_TCL) &&\n+\t\t used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n \t\t\tdie(_(\"--format=%.*s cannot be used with\"\n-\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n+\t\t\t \"--python, --shell, --tcl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 18554f62d94..3d15d0a5360 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' '\n \ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n '\n \n-test_expect_success '%(raw) with --perl must fail' '\n-\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+test_expect_success '%(raw) with --perl' '\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob1 --perl | perl >actual &&\n+\tcmp blob1 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob3 --perl | perl >actual &&\n+\tcmp blob3 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob8 --perl | perl >actual &&\n+\tcmp blob8 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/first --perl | perl >actual &&\n+\tcmp one actual &&\n+\tgit cat-file tree refs/mytrees/first > expected &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/mytrees/first --perl | perl >actual &&\n+\tcmp expected actual\n '\n \n test_expect_success '%(raw) with --shell must fail' '\n-- \ngitgitgadget\n\n"},{"id":"431000","messageId":"eafb79bad62f13fc8fd70ba1dce3e8fbab870e52.1627031043.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v2.git.1627031043.gitgitgadget@gmail.com","subject":"[PATCH v2 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T09:04:00Z","receivedAt":"2021-07-23T09:04:13Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd new formatting option `%(raw)`, which will print the raw\nobject data without any changes. It will help further to migrate\nall cat-file formatting logic from cat-file to ref-filter.\n\nThe raw data of blob, tree objects may contain '\\0', but most of\nthe logic in `ref-filter` depends on the output of the atom being\ntext (specifically, no embedded NULs in it).\n\nE.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()`\nadd the data to the buffer. The raw data of a tree object is\n`100644 one\\0...`, only the `100644 one` will be added to the buffer,\nwhich is incorrect.\n\nTherefore, we need to find a way to record the length of the\natom_value's member `s`. Although strbuf can already record the\nstring and its length, if we want to replace the type of atom_value's\nmember `s` with strbuf, many places in ref-filter that are filled\nwith dynamically allocated mermory in `v->s` are not easy to replace.\nAt the same time, we need to check if `v->s == NULL` in\npopulate_value(), and strbuf cannot easily distinguish NULL and empty\nstrings, but c-style \"const char *\" can do it. So add a new member in\n`struct atom_value`: `s_size`, which can record raw object size, it\ncan help us add raw object data to the buffer or compare two buffers\nwhich contain raw object data.\n\nNote that `--format=%(raw)` cannot be used with `--python`, `--shell`,\n`--tcl`, and `--perl` because if the binary raw data is passed to a\nvariable in such languages, these may not support arbitrary binary data\nin their string variable type.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Bagas Sanjaya \nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Felipe Contreras \nHelped-by: Phillip Wood \nHelped-by: Junio C Hamano \nBased-on-patch-by: Olga Telezhnaya \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 9 ++\n ref-filter.c | 142 +++++++++++++++----\n t/t6300-for-each-ref.sh | 216 +++++++++++++++++++++++++++++\n 3 files changed, 339 insertions(+), 28 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 2ae2478de70..cbb6f87d13f 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`,\n without angle brackets, and `:localpart` to get the part before the `@` symbol\n out of the trimmed email.\n \n+The raw data in an object is `raw`.\n+\n+raw:size::\n+\tThe raw data size of the object.\n+\n+Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n+`--perl` because such language may not support arbitrary binary data in their\n+string variable type.\n+\n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5cee6512fba..a713507d783 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -144,6 +144,7 @@ enum atom_type {\n \tATOM_BODY,\n \tATOM_TRAILERS,\n \tATOM_CONTENTS,\n+\tATOM_RAW,\n \tATOM_UPSTREAM,\n \tATOM_PUSH,\n \tATOM_SYMREF,\n@@ -189,6 +190,9 @@ static struct used_atom {\n \t\t\tstruct process_trailer_options trailer_opts;\n \t\t\tunsigned int nlines;\n \t\t} contents;\n+\t\tstruct {\n+\t\t\tenum { RAW_BARE, RAW_LENGTH } option;\n+\t\t} raw_data;\n \t\tstruct {\n \t\t\tcmp_status cmp_status;\n \t\t\tconst char *str;\n@@ -426,6 +430,18 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n+static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\tconst char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\tatom->u.raw_data.option = RAW_BARE;\n+\telse if (!strcmp(arg, \"size\"))\n+\t\tatom->u.raw_data.option = RAW_LENGTH;\n+\telse\n+\t\treturn strbuf_addf_ret(err, -1, _(\"unrecognized %%(raw) argument: %s\"), arg);\n+\treturn 0;\n+}\n+\n static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -586,6 +602,7 @@ static struct {\n \t[ATOM_BODY] = { \"body\", SOURCE_OBJ, FIELD_STR, body_atom_parser },\n \t[ATOM_TRAILERS] = { \"trailers\", SOURCE_OBJ, FIELD_STR, trailers_atom_parser },\n \t[ATOM_CONTENTS] = { \"contents\", SOURCE_OBJ, FIELD_STR, contents_atom_parser },\n+\t[ATOM_RAW] = { \"raw\", SOURCE_OBJ, FIELD_STR, raw_atom_parser },\n \t[ATOM_UPSTREAM] = { \"upstream\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_PUSH] = { \"push\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_SYMREF] = { \"symref\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -620,12 +637,17 @@ struct ref_formatting_state {\n \n struct atom_value {\n \tconst char *s;\n+\tsize_t s_size;\n \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t struct strbuf *err);\n \tuintmax_t value; /* used for sorting when not FIELD_STR */\n \tstruct used_atom *atom;\n };\n \n+#define ATOM_VALUE_INIT { \\\n+\t.s_size = -1 \\\n+}\n+\n /*\n * Used to parse format string and sort specifiers\n */\n@@ -644,13 +666,6 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \t\treturn strbuf_addf_ret(err, -1, _(\"malformed field name: %.*s\"),\n \t\t\t\t (int)(ep-atom), atom);\n \n-\t/* Do we have the atom already used elsewhere? */\n-\tfor (i = 0; i < used_atom_cnt; i++) {\n-\t\tint len = strlen(used_atom[i].name);\n-\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n-\t\t\treturn i;\n-\t}\n-\n \t/*\n \t * If the atom name has a colon, strip it and everything after\n \t * it off - it specifies the format for this entry, and\n@@ -660,6 +675,13 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \targ = memchr(sp, ':', ep - sp);\n \tatom_len = (arg ? arg : ep) - sp;\n \n+\t/* Do we have the atom already used elsewhere? */\n+\tfor (i = 0; i < used_atom_cnt; i++) {\n+\t\tint len = strlen(used_atom[i].name);\n+\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n+\t\t\treturn i;\n+\t}\n+\n \t/* Is the atom a valid one? */\n \tfor (i = 0; i < ARRAY_SIZE(valid_atom); i++) {\n \t\tint len = strlen(valid_atom[i].name);\n@@ -709,11 +731,14 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \treturn at;\n }\n \n-static void quote_formatting(struct strbuf *s, const char *str, int quote_style)\n+static void quote_formatting(struct strbuf *s, const char *str, size_t len, int quote_style)\n {\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n-\t\tstrbuf_addstr(s, str);\n+\t\tif (len != -1)\n+\t\t\tstrbuf_add(s, str, len);\n+\t\telse\n+\t\t\tstrbuf_addstr(s, str);\n \t\tbreak;\n \tcase QUOTE_SHELL:\n \t\tsq_quote_buf(s, str);\n@@ -740,9 +765,12 @@ static int append_atom(struct atom_value *v, struct ref_formatting_state *state,\n \t * encountered.\n \t */\n \tif (!state->stack->prev)\n-\t\tquote_formatting(&state->stack->output, v->s, state->quote_style);\n+\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n \telse\n-\t\tstrbuf_addstr(&state->stack->output, v->s);\n+\t\tif (v->s_size != -1)\n+\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n+\t\telse\n+\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n \treturn 0;\n }\n \n@@ -842,21 +870,23 @@ static int if_atom_handler(struct atom_value *atomv, struct ref_formatting_state\n \treturn 0;\n }\n \n-static int is_empty(const char *s)\n+static int is_empty(struct strbuf *buf)\n {\n-\twhile (*s != '\\0') {\n-\t\tif (!isspace(*s))\n-\t\t\treturn 0;\n-\t\ts++;\n-\t}\n-\treturn 1;\n-}\n+\tconst char *cur = buf->buf;\n+\tconst char *end = buf->buf + buf->len;\n+\n+\twhile (cur != end && (isspace(*cur)))\n+\t\tcur++;\n+\n+\treturn cur == end;\n+ }\n \n static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t\t struct strbuf *err)\n {\n \tstruct ref_formatting_stack *cur = state->stack;\n \tstruct if_then_else *if_then_else = NULL;\n+\tsize_t str_len = 0;\n \n \tif (cur->at_end == if_then_else_handler)\n \t\tif_then_else = (struct if_then_else *)cur->at_end_data;\n@@ -867,18 +897,22 @@ static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_sta\n \tif (if_then_else->else_atom_seen)\n \t\treturn strbuf_addf_ret(err, -1, _(\"format: %%(then) atom used after %%(else)\"));\n \tif_then_else->then_atom_seen = 1;\n+\tif (if_then_else->str)\n+\t\tstr_len = strlen(if_then_else->str);\n \t/*\n \t * If the 'equals' or 'notequals' attribute is used then\n \t * perform the required comparison. If not, only non-empty\n \t * strings satisfy the 'if' condition.\n \t */\n \tif (if_then_else->cmp_status == COMPARE_EQUAL) {\n-\t\tif (!strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len == cur->output.len &&\n+\t\t !memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n \t} else if (if_then_else->cmp_status == COMPARE_UNEQUAL) {\n-\t\tif (strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len != cur->output.len ||\n+\t\t memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n-\t} else if (cur->output.len && !is_empty(cur->output.buf))\n+\t} else if (cur->output.len && !is_empty(&cur->output))\n \t\tif_then_else->condition_satisfied = 1;\n \tstrbuf_reset(&cur->output);\n \treturn 0;\n@@ -924,7 +958,7 @@ static int end_atom_handler(struct atom_value *atomv, struct ref_formatting_stat\n \t * only on the topmost supporting atom.\n \t */\n \tif (!current->prev->prev) {\n-\t\tquote_formatting(&s, current->output.buf, state->quote_style);\n+\t\tquote_formatting(&s, current->output.buf, current->output.len, state->quote_style);\n \t\tstrbuf_swap(¤t->output, &s);\n \t}\n \tstrbuf_release(&s);\n@@ -974,6 +1008,10 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\t\t\tdie(_(\"--format=%.*s cannot be used with\"\n+\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\n@@ -1367,12 +1405,25 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp\n \t\tstruct used_atom *atom = &used_atom[i];\n \t\tconst char *name = atom->name;\n \t\tstruct atom_value *v = &val[i];\n+\t\tenum atom_type atom_type = atom->atom_type;\n \n \t\tif (!!deref != (*name == '*'))\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n \n+\t\tif (atom_type == ATOM_RAW) {\n+\t\t\tunsigned long buf_size = data->size;\n+\n+\t\t\tif (atom->u.raw_data.option == RAW_BARE) {\n+\t\t\t\tv->s = xmemdupz(buf, buf_size);\n+\t\t\t\tv->s_size = buf_size;\n+\t\t\t} else if (atom->u.raw_data.option == RAW_LENGTH) {\n+\t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX, (uintmax_t)buf_size);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif ((data->type != OBJ_TAG &&\n \t\t data->type != OBJ_COMMIT) ||\n \t\t (strcmp(name, \"body\") &&\n@@ -1460,9 +1511,11 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"Eh? Object of type %d?\", obj->type);\n@@ -1766,6 +1819,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n \n+\t\tv->s_size = -1;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n@@ -2369,6 +2423,19 @@ static int compare_detached_head(struct ref_array_item *a, struct ref_array_item\n \treturn 0;\n }\n \n+static int memcasecmp(const void *vs1, const void *vs2, size_t n)\n+{\n+\tconst char *s1 = vs1, *s2 = vs2;\n+\tconst char *end = s1 + n;\n+\n+\tfor (; s1 < end; s1++, s2++) {\n+\t\tint diff = tolower(*s1) - tolower(*s2);\n+\t\tif (diff)\n+\t\t\treturn diff;\n+\t}\n+\treturn 0;\n+}\n+\n static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b)\n {\n \tstruct atom_value *va, *vb;\n@@ -2389,10 +2456,29 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n \t} else if (s->sort_flags & REF_SORTING_VERSION) {\n \t\tcmp = versioncmp(va->s, vb->s);\n \t} else if (cmp_type == FIELD_STR) {\n-\t\tint (*cmp_fn)(const char *, const char *);\n-\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n-\t\t\t? strcasecmp : strcmp;\n-\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\tif (va->s_size == -1 && vb->s_size == -1) {\n+\t\t\tint (*cmp_fn)(const char *, const char *);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? strcasecmp : strcmp;\n+\t\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\t} else {\n+\t\t\tsize_t a_size = va->s_size == -1 ?\n+\t\t\t\t\tstrlen(va->s) : va->s_size;\n+\t\t\tsize_t b_size = vb->s_size == -1 ?\n+\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n+\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? memcasecmp : memcmp;\n+\n+\t\t\tcmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n+\t\t\t\t a_size : b_size);\n+\t\t\tif (!cmp) {\n+\t\t\t\tif (a_size > b_size)\n+\t\t\t\t\tcmp = 1;\n+\t\t\t\telse if (a_size < b_size)\n+\t\t\t\t\tcmp = -1;\n+\t\t\t}\n+\t\t}\n \t} else {\n \t\tif (va->value < vb->value)\n \t\t\tcmp = -1;\n@@ -2491,7 +2577,7 @@ int format_ref_array_item(struct ref_array_item *info,\n \t\tappend_literal(cp, sp, &state);\n \t}\n \tif (format->need_color_reset_at_eol) {\n-\t\tstruct atom_value resetv;\n+\t\tstruct atom_value resetv = ATOM_VALUE_INIT;\n \t\tresetv.s = GIT_COLOR_RESET;\n \t\tif (append_atom(&resetv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 9e0214076b4..18554f62d94 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -130,6 +130,8 @@ test_atom head parent:short=10 ''\n test_atom head numparent 0\n test_atom head object ''\n test_atom head type ''\n+test_atom head raw \"$(git cat-file commit refs/heads/main)\n+\"\n test_atom head '*objectname' ''\n test_atom head '*objecttype' ''\n test_atom head author 'A U Thor 1151968724 +0200'\n@@ -221,6 +223,15 @@ test_atom tag contents 'Tagging at 1151968727\n '\n test_atom tag HEAD ' '\n \n+test_expect_success 'basic atom: refs/tags/testtag *raw' '\n+\tgit cat-file commit refs/tags/testtag^{} >expected &&\n+\tgit for-each-ref --format=\"%(*raw)\" refs/tags/testtag >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'Check invalid atoms names are errors' '\n \ttest_must_fail git for-each-ref --format=\"%(INVALID)\" refs/heads\n '\n@@ -686,6 +697,15 @@ test_atom refs/tags/signed-empty contents:body ''\n test_atom refs/tags/signed-empty contents:signature \"$sig\"\n test_atom refs/tags/signed-empty contents \"$sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-empty raw' '\n+\tgit cat-file tag refs/tags/signed-empty >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-empty >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-short subject 'subject line'\n test_atom refs/tags/signed-short subject:sanitize 'subject-line'\n test_atom refs/tags/signed-short contents:subject 'subject line'\n@@ -695,6 +715,15 @@ test_atom refs/tags/signed-short contents:signature \"$sig\"\n test_atom refs/tags/signed-short contents \"subject line\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-short raw' '\n+\tgit cat-file tag refs/tags/signed-short >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-short >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-long subject 'subject line'\n test_atom refs/tags/signed-long subject:sanitize 'subject-line'\n test_atom refs/tags/signed-long contents:subject 'subject line'\n@@ -708,6 +737,15 @@ test_atom refs/tags/signed-long contents \"subject line\n body contents\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-long raw' '\n+\tgit cat-file tag refs/tags/signed-long >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-long >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'set up refs pointing to tree and blob' '\n \tgit update-ref refs/mytrees/first refs/heads/main^{tree} &&\n \tgit update-ref refs/myblobs/first refs/heads/main:one\n@@ -720,6 +758,16 @@ test_atom refs/mytrees/first contents:body \"\"\n test_atom refs/mytrees/first contents:signature \"\"\n test_atom refs/mytrees/first contents \"\"\n \n+test_expect_success 'basic atom: refs/mytrees/first raw' '\n+\tgit cat-file tree refs/mytrees/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/mytrees/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_atom refs/myblobs/first subject \"\"\n test_atom refs/myblobs/first contents:subject \"\"\n test_atom refs/myblobs/first body \"\"\n@@ -727,6 +775,174 @@ test_atom refs/myblobs/first contents:body \"\"\n test_atom refs/myblobs/first contents:signature \"\"\n test_atom refs/myblobs/first contents \"\"\n \n+test_expect_success 'basic atom: refs/myblobs/first raw' '\n+\tgit cat-file blob refs/myblobs/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/myblobs/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'set up refs pointing to binary blob' '\n+\tprintf \"a\\0b\\0c\" >blob1 &&\n+\tprintf \"a\\0c\\0b\" >blob2 &&\n+\tprintf \"\\0a\\0b\\0c\" >blob3 &&\n+\tprintf \"abc\" >blob4 &&\n+\tprintf \"\\0 \\0 \\0 \" >blob5 &&\n+\tprintf \"\\0 \\0a\\0 \" >blob6 &&\n+\tprintf \" \" >blob7 &&\n+\t>blob8 &&\n+\tobj=$(git hash-object -w blob1) &&\n+\tgit update-ref refs/myblobs/blob1 \"$obj\" &&\n+\tobj=$(git hash-object -w blob2) &&\n+\tgit update-ref refs/myblobs/blob2 \"$obj\" &&\n+\tobj=$(git hash-object -w blob3) &&\n+\tgit update-ref refs/myblobs/blob3 \"$obj\" &&\n+\tobj=$(git hash-object -w blob4) &&\n+\tgit update-ref refs/myblobs/blob4 \"$obj\" &&\n+\tobj=$(git hash-object -w blob5) &&\n+\tgit update-ref refs/myblobs/blob5 \"$obj\" &&\n+\tobj=$(git hash-object -w blob6) &&\n+\tgit update-ref refs/myblobs/blob6 \"$obj\" &&\n+\tobj=$(git hash-object -w blob7) &&\n+\tgit update-ref refs/myblobs/blob7 \"$obj\" &&\n+\tobj=$(git hash-object -w blob8) &&\n+\tgit update-ref refs/myblobs/blob8 \"$obj\"\n+'\n+\n+test_expect_success 'Verify sorts with raw' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob7\n+\trefs/mytrees/first\n+\trefs/myblobs/first\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob4\n+\trefs/heads/main\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'Verify sorts with raw:size' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\trefs/myblobs/blob7\n+\trefs/heads/main\n+\trefs/myblobs/blob4\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/mytrees/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw:size \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:equals)' '\n+\tcat >expected <<-EOF &&\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\trefs/myblobs/blob4\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:equals=abc)%(raw)%(then)%(refname)%(else)not equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:notequals)' '\n+\tcat >expected <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/heads/main\n+\trefs/heads/newtag\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\tequals\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob7\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:notequals=abc)%(raw)%(then)%(refname)%(else)equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'empty raw refs with %(if)' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob1 not empty\n+\trefs/myblobs/blob2 not empty\n+\trefs/myblobs/blob3 not empty\n+\trefs/myblobs/blob4 not empty\n+\trefs/myblobs/blob5 not empty\n+\trefs/myblobs/blob6 not empty\n+\trefs/myblobs/blob7 empty\n+\trefs/myblobs/blob8 empty\n+\trefs/myblobs/first not empty\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname) %(if)%(raw)%(then)not empty%(else)empty%(end)\" \\\n+\t\trefs/myblobs/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success '%(raw) with --python must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --python\n+'\n+\n+test_expect_success '%(raw) with --tcl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n+'\n+\n+test_expect_success '%(raw) with --perl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+'\n+\n+test_expect_success '%(raw) with --shell must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --shell\n+'\n+\n+test_expect_success '%(raw) with --shell and --sort=raw must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --sort=raw --shell\n+'\n+\n+test_expect_success '%(raw:size) with --shell' '\n+\tgit for-each-ref --format=\"%(raw:size)\" | while read line\n+\tdo\n+\t\techo \"'\\''$line'\\''\" >>expect\n+\tdone &&\n+\tgit for-each-ref --format=\"%(raw:size)\" --shell >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for-each-ref --format compare with cat-file --batch' '\n+\tgit rev-parse refs/mytrees/first | git cat-file --batch >expected &&\n+\tgit for-each-ref --format=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'set up multiple-sort tags' '\n \tfor when in 100000 200000\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"431001","messageId":"3a8173b42ed1d3a3be30bb3b53ef604518486e38.1627031043.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v2.git.1627031043.gitgitgadget@gmail.com","subject":"[PATCH v2 4/5] [GSOC] ref-filter: use non-const ref_format in *_atom_parser()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T09:04:02Z","receivedAt":"2021-07-23T09:04:14Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse non-const ref_format in *_atom_parser(), which can help us\nmodify the members of ref_format in *_atom_parser().\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/tag.c | 2 +-\n ref-filter.c | 44 ++++++++++++++++++++++----------------------\n ref-filter.h | 4 ++--\n 3 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 82fcfc09824..452558ec957 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref,\n \t\t const struct object_id *oid, void *cb_data)\n {\n \tint flags;\n-\tconst struct ref_format *format = cb_data;\n+\tstruct ref_format *format = cb_data;\n \tflags = GPG_VERIFY_VERBOSE;\n \n \tif (format->format)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 0ff8574a00c..1277ed483a7 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -226,7 +226,7 @@ static int strbuf_addf_ret(struct strbuf *sb, int ret, const char *fmt, ...)\n \treturn ret;\n }\n \n-static int color_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int color_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *color_value, struct strbuf *err)\n {\n \tif (!color_value)\n@@ -264,7 +264,7 @@ static int refname_atom_parser_internal(struct refname_atom *atom, const char *a\n \treturn 0;\n }\n \n-static int remote_ref_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int remote_ref_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct string_list params = STRING_LIST_INIT_DUP;\n@@ -311,7 +311,7 @@ static int remote_ref_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objecttype_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -323,7 +323,7 @@ static int objecttype_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objectsize_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -343,7 +343,7 @@ static int objectsize_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int deltabase_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int deltabase_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -355,7 +355,7 @@ static int deltabase_atom_parser(const struct ref_format *format, struct used_at\n \treturn 0;\n }\n \n-static int body_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int body_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -364,7 +364,7 @@ static int body_atom_parser(const struct ref_format *format, struct used_atom *a\n \treturn 0;\n }\n \n-static int subject_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int subject_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -376,7 +376,7 @@ static int subject_atom_parser(const struct ref_format *format, struct used_atom\n \treturn 0;\n }\n \n-static int trailers_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int trailers_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tatom->u.contents.trailer_opts.no_divider = 1;\n@@ -402,7 +402,7 @@ static int trailers_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int contents_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int contents_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -430,7 +430,7 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int raw_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -442,7 +442,7 @@ static int raw_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int oid_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -461,7 +461,7 @@ static int oid_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int person_email_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int person_email_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -475,7 +475,7 @@ static int person_email_atom_parser(const struct ref_format *format, struct used\n \treturn 0;\n }\n \n-static int refname_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int refname_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \treturn refname_atom_parser_internal(&atom->u.refname, arg, atom->name, err);\n@@ -492,7 +492,7 @@ static align_type parse_align_position(const char *s)\n \treturn -1;\n }\n \n-static int align_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int align_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct align *align = &atom->u.align;\n@@ -544,7 +544,7 @@ static int align_atom_parser(const struct ref_format *format, struct used_atom *\n \treturn 0;\n }\n \n-static int if_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -559,7 +559,7 @@ static int if_atom_parser(const struct ref_format *format, struct used_atom *ato\n \treturn 0;\n }\n \n-static int head_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n \tatom->u.head = resolve_refdup(\"HEAD\", RESOLVE_REF_READING, NULL, NULL);\n@@ -570,7 +570,7 @@ static struct {\n \tconst char *name;\n \tinfo_source source;\n \tcmp_type cmp_type;\n-\tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n+\tint (*parser)(struct ref_format *format, struct used_atom *atom,\n \t\t const char *arg, struct strbuf *err);\n } valid_atom[] = {\n \t[ATOM_REFNAME] = { \"refname\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -651,7 +651,7 @@ struct atom_value {\n /*\n * Used to parse format string and sort specifiers\n */\n-static int parse_ref_filter_atom(const struct ref_format *format,\n+static int parse_ref_filter_atom(struct ref_format *format,\n \t\t\t\t const char *atom, const char *ep,\n \t\t\t\t struct strbuf *err)\n {\n@@ -2555,9 +2555,9 @@ static void append_literal(const char *cp, const char *ep, struct ref_formatting\n }\n \n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n-\t\t\t struct strbuf *final_buf,\n-\t\t\t struct strbuf *error_buf)\n+\t\t\t struct ref_format *format,\n+\t\t\t struct strbuf *final_buf,\n+\t\t\t struct strbuf *error_buf)\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n@@ -2601,7 +2601,7 @@ int format_ref_array_item(struct ref_array_item *info,\n }\n \n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format)\n+\t\t struct ref_format *format)\n {\n \tstruct ref_array_item *ref_item;\n \tstruct strbuf output = STRBUF_INIT;\ndiff --git a/ref-filter.h b/ref-filter.h\nindex baf72a71896..74fb423fc89 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);\n void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting, unsigned int mask, int on);\n /* Based on the given format and quote_style, fill the strbuf */\n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n+\t\t\t struct ref_format *format,\n \t\t\t struct strbuf *final_buf,\n \t\t\t struct strbuf *error_buf);\n /* Parse a single sort specifier and add it to the list */\n@@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void);\n * name must be a fully qualified refname.\n */\n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format);\n+\t\t struct ref_format *format);\n \n /*\n * Push a single ref onto the array; this can be used to construct your own\n-- \ngitgitgadget\n\n"},{"id":"431002","messageId":"71249dfd4e08077c954ae512eccfd98ccf8f9e1c.1627031043.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v2.git.1627031043.gitgitgadget@gmail.com","subject":"[PATCH v2 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T09:04:03Z","receivedAt":"2021-07-23T09:04:15Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nIn order to let \"cat-file --batch=%(rest)\" use the ref-filter\ninterface, add %(rest) atom for ref-filter. Introduce the\nreject_atom() to reject the atom %(rest) for \"git for-each-ref\",\n\"git branch\", \"git tag\" and \"git verify-tag\".\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 25 +++++++++++++++++++++++++\n ref-filter.h | 5 ++++-\n t/t3203-branch-output.sh | 4 ++++\n t/t6300-for-each-ref.sh | 4 ++++\n t/t7004-tag.sh | 4 ++++\n t/t7030-verify-tag.sh | 4 ++++\n 6 files changed, 45 insertions(+), 1 deletion(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 1277ed483a7..c4cfff10822 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -157,6 +157,7 @@ enum atom_type {\n \tATOM_IF,\n \tATOM_THEN,\n \tATOM_ELSE,\n+\tATOM_REST,\n };\n \n /*\n@@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"%%(rest) does not take arguments\"));\n+\tformat->use_rest = 1;\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n@@ -615,6 +625,7 @@ static struct {\n \t[ATOM_IF] = { \"if\", SOURCE_NONE, FIELD_STR, if_atom_parser },\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n+\t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -991,6 +1002,11 @@ static const char *find_next(const char *cp)\n \treturn NULL;\n }\n \n+static int reject_atom(enum atom_type atom_type)\n+{\n+\treturn atom_type == ATOM_REST;\n+}\n+\n /*\n * Make sure the format string is well formed, and parse out\n * the used atoms.\n@@ -1011,6 +1027,8 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (reject_atom(used_atom[at].atom_type))\n+\t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n \t\t format->quote_style == QUOTE_SHELL ||\n@@ -1930,6 +1948,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_REST) {\n+\t\t\tif (ref->rest)\n+\t\t\t\tv->s = xstrdup(ref->rest);\n+\t\t\telse\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2147,6 +2171,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n+\tref->rest = NULL;\n \n \treturn ref;\n }\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 74fb423fc89..c15dee8d6b9 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -38,6 +38,7 @@ struct ref_sorting {\n \n struct ref_array_item {\n \tstruct object_id objectname;\n+\tconst char *rest;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n@@ -76,14 +77,16 @@ struct ref_format {\n \t * verify_ref_format() afterwards to finalize.\n \t */\n \tconst char *format;\n+\tconst char *rest;\n \tint quote_style;\n+\tint use_rest;\n \tint use_color;\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n };\n \n-#define REF_FORMAT_INIT { NULL, 0, -1 }\n+#define REF_FORMAT_INIT { .use_color = -1 }\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\ndiff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh\nindex 5325b9f67a0..6e94c6db7b5 100755\n--- a/t/t3203-branch-output.sh\n+++ b/t/t3203-branch-output.sh\n@@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git branch with --format=%(rest) must fail' '\n+\ttest_must_fail git branch --format=\"%(rest)\" >actual\n+'\n+\n test_expect_success 'worktree colors correct' '\n \tcat >expect <<-EOF &&\n \t* (HEAD detached from fromtag)\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 3d15d0a5360..0d2e062f791 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' '\n \ttest_cmp expect actual.clean\n '\n \n+test_expect_success 'basic atom: rest must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(rest)\" refs/heads/main\n+'\n+\n test_expect_success 'trailer parsing not fooled by --- line' '\n \tgit commit --allow-empty -F - <<-\\EOF &&\n \tthis is the subject\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 2f72c5c6883..082be85dffc 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git tag -l with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git tag -l --format=\"%(rest)\" \"v1*\"\n+'\n+\n test_expect_success \"set up color tests\" '\n \techo \"v1.0\" >expect.color &&\n \techo \"v1.0\" >expect.bare &&\ndiff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh\nindex 3cefde9602b..10faa645157 100755\n--- a/t/t7030-verify-tag.sh\n+++ b/t/t7030-verify-tag.sh\n@@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success GPG 'verifying tag with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git verify-tag --format=\"%(rest)\" \"fourth-signed\"\n+'\n+\n test_expect_success GPG 'verifying a forged tag with --format should fail silently' '\n \ttest_must_fail git verify-tag --format=\"tagname : %(tag)\" $(cat forged1.tag) >actual-forged &&\n \ttest_must_be_empty actual-forged\n-- \ngitgitgadget\n"},{"id":"431003","messageId":"CA+P7+xr7GCZCk44e_CSYFDr_ow3-BBMUfP5t6=cErFBES4N8+Q@mail.gmail.com","threadId":"56142","inReplyTo":"75eb2f6740eb5845afcb7d31956cc5b3e3957f97.1626939557.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2021-07-23T09:09:40Z","receivedAt":"2021-07-23T09:09:52Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Thu, Jul 22, 2021 at 12:40 AM ZheNing Hu via GitGitGadget\n wrote:\n>\n> From: ZheNing Hu \n>\n> In order to let \"cat-file --batch=%(rest)\" use the ref-filter\n> interface, add %(rest) atom for ref-filter. Introduce the\n> reject_atom() to reject the atom %(rest) for \"git for-each-ref\",\n> \"git branch\", \"git tag\" and \"git verify-tag\".\n>\n\nTo a reader intimately familiar with cat-file, %(rest) might be\nobvious.. but it would help the commit message read more easily if\nthere was a short explanation of what %(rest) did. It is not obvious\njust from the name to me.\n\nThanks,\nJake\n"},{"id":"431038","messageId":"CAOLTT8Q1A8FBiWdEqF90_yQfT11uE3OwRy-2MGV9qm-4O4S1QQ@mail.gmail.com","threadId":"56142","inReplyTo":"CA+P7+xr7GCZCk44e_CSYFDr_ow3-BBMUfP5t6=cErFBES4N8+Q@mail.gmail.com","subject":"Re: [PATCH 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-23T14:11:06Z","receivedAt":"2021-07-23T14:10:42Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Jacob Keller 于2021年7月23日周五 下午5:09写道：\n>\n> On Thu, Jul 22, 2021 at 12:40 AM ZheNing Hu via GitGitGadget\n> wrote:\n> >\n> > From: ZheNing Hu \n> >\n> > In order to let \"cat-file --batch=%(rest)\" use the ref-filter\n> > interface, add %(rest) atom for ref-filter. Introduce the\n> > reject_atom() to reject the atom %(rest) for \"git for-each-ref\",\n> > \"git branch\", \"git tag\" and \"git verify-tag\".\n> >\n>\n> To a reader intimately familiar with cat-file, %(rest) might be\n> obvious.. but it would help the commit message read more easily if\n> there was a short explanation of what %(rest) did. It is not obvious\n> just from the name to me.\n>\n\nYou are right, I may subconsciously think that everyone knows the\n%(rest) in cat-file.\nI will change it.\n\n> Thanks,\n> Jake\n\nThanks.\n--\nZheNing Hu\n"},{"id":"431051","messageId":"xmqqh7glouiw.fsf@gitster.g","threadId":"56142","inReplyTo":"eafb79bad62f13fc8fd70ba1dce3e8fbab870e52.1627031043.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-23T16:38:15Z","receivedAt":"2021-07-23T16:38:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"ZheNing Hu via GitGitGadget\" writes:\n\n> struct atom_value {\n> \tconst char *s;\n> +\tsize_t s_size;\n> \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n> \t\t struct strbuf *err);\n> \tuintmax_t value; /* used for sorting when not FIELD_STR */\n> \tstruct used_atom *atom;\n> };\n> \n> +#define ATOM_VALUE_INIT { \\\n> +\t.s_size = -1 \\\n> +}\n\nThis will get turned into size_t that is unsigned, and comparisons\nlike this one and ...\n\n> \tcase QUOTE_NONE:\n> -\t\tstrbuf_addstr(s, str);\n> +\t\tif (len != -1)\n> +\t\t\tstrbuf_add(s, str, len);\n> +\t\telse\n> +\t\t\tstrbuf_addstr(s, str);\n\nthis one ...\n\n> +\t\tif (v->s_size != -1)\n> +\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n> +\t\telse\n> +\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n> \treturn 0;\n> }\n\nmay work as expected, but it makes readers wonder if there is\nsignificance to negative values other than -1 (in other words, what\ndoes it mean if v->s_size == -2, for example?).\n\nIt may make sense to \n\n * Turn atom_value.s_size field into ssize_t instead of size_t\n\n * Rewrite (v->s_size != -1) comparison to (v->s_size < 0)\n\n\nOptionally we could introduce #define ATOM_SIZE_UNSPECIFIED (-1) and\nuse it to initialize .s_size in ATOM_VALUE_INIT, but if we are just\ngoing to test \"is it negative, then it is not given\", then it probably\nis not needed.\n\nOther changes in the whole series relative to what has been queued\nlooked all sensible to me.\n\nTHanks.\n\n\n\n\n \n"},{"id":"431110","messageId":"CAOLTT8SggCXkajPG3om+6zhM_K8fyAb2qTBDj40JJa1pszshzg@mail.gmail.com","threadId":"56142","inReplyTo":"xmqqh7glouiw.fsf@gitster.g","subject":"Re: [PATCH v2 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-24T07:57:16Z","receivedAt":"2021-07-24T07:56:51Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年7月24日周六 上午12:38写道：\n>\n> \"ZheNing Hu via GitGitGadget\" writes:\n>\n> > struct atom_value {\n> > const char *s;\n> > + size_t s_size;\n> > int (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n> > struct strbuf *err);\n> > uintmax_t value; /* used for sorting when not FIELD_STR */\n> > struct used_atom *atom;\n> > };\n> >\n> > +#define ATOM_VALUE_INIT { \\\n> > + .s_size = -1 \\\n> > +}\n>\n> This will get turned into size_t that is unsigned, and comparisons\n> like this one and ...\n>\n> > case QUOTE_NONE:\n> > - strbuf_addstr(s, str);\n> > + if (len != -1)\n> > + strbuf_add(s, str, len);\n> > + else\n> > + strbuf_addstr(s, str);\n>\n> this one ...\n>\n> > + if (v->s_size != -1)\n> > + strbuf_add(&state->stack->output, v->s, v->s_size);\n> > + else\n> > + strbuf_addstr(&state->stack->output, v->s);\n> > return 0;\n> > }\n>\n> may work as expected, but it makes readers wonder if there is\n> significance to negative values other than -1 (in other words, what\n> does it mean if v->s_size == -2, for example?).\n>\n\nYes, any value less than -1 is meaningless here.\n\n> It may make sense to\n>\n> * Turn atom_value.s_size field into ssize_t instead of size_t\n>\n\nWill the conversion of size_t and ssize_t break -Wsign-conversion?\nAlthough there is a lot of code in Git that has broken it, but I am not\nsure if it is wise to use ssize_t here.\n\n> * Rewrite (v->s_size != -1) comparison to (v->s_size < 0)\n>\n\nFor size_t, this scheme is wrong.\n\n>\n> Optionally we could introduce #define ATOM_SIZE_UNSPECIFIED (-1) and\n> use it to initialize .s_size in ATOM_VALUE_INIT, but if we are just\n> going to test \"is it negative, then it is not given\", then it probably\n> is not needed.\n>\n\nIt seems that this is the only method left. Although I think\nATOM_SIZE_UNSPECIFIED\nis not very useful becasue we already have ATOM_VALUE_INIT.\n\n> Other changes in the whole series relative to what has been queued\n> looked all sensible to me.\n>\n> THanks.\n>\n\nThanks.\n--\nZheNing Hu\n"},{"id":"431113","messageId":"pull.1000.v3.git.1627136061.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v2.git.1627031043.gitgitgadget@gmail.com","subject":"[PATCH v3 0/5] [GSOC] ref-filter: add %(raw) and %(rest) atoms","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-24T14:14:16Z","receivedAt":"2021-07-24T14:14:26Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"This patch series is split from my main patch series zh/ref-filter-raw-data,\nto make things easier for reviewers to read. It's last version is here:\nhttps://lore.kernel.org/git/pull.993.v2.git.1626363626.gitgitgadget@gmail.com/\n\nThis patch series provided %(raw) and %(rest) for ref-filter, which will be\nused by zh/cat-file-reuse-ref-filter-logic later.\n\nv2--->v3:\n\n 1. Add macro ATOM_SIZE_UNSPECIFIED to represent the initial value of\n atom_value->s_size.\n 2. Add a description about %(rest) in the commit message. (copy from\n https://git-scm.com/docs/git-cat-file)\n\nv2:\nhttps://lore.kernel.org/git/CAP8UFD3ENBZdm8yQn6J-yb4PvUsrzTA3KGxes42dE3bTrYbr9Q@mail.gmail.com/\n\nZheNing Hu (5):\n [GSOC] ref-filter: add obj-type check in grab contents\n [GSOC] ref-filter: add %(raw) atom\n [GSOC] ref-filter: --format=%(raw) re-support --perl\n [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n [GSOC] ref-filter: add %(rest) atom\n\n Documentation/git-for-each-ref.txt | 9 ++\n builtin/tag.c | 2 +-\n quote.c | 17 ++\n quote.h | 1 +\n ref-filter.c | 244 ++++++++++++++++++++++-------\n ref-filter.h | 9 +-\n t/t3203-branch-output.sh | 4 +\n t/t6300-for-each-ref.sh | 235 +++++++++++++++++++++++++++\n t/t7004-tag.sh | 4 +\n t/t7030-verify-tag.sh | 4 +\n 10 files changed, 466 insertions(+), 63 deletions(-)\n\n\nbase-commit: daab8a564f8bbac55f70f8bf86c070e001a9b006\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1000%2Fadlternative%2Fref-filter-raw-data-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1000/adlternative/ref-filter-raw-data-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1000\n\nRange-diff vs v2:\n\n 1: d77268bb188 = 1: d77268bb188 [GSOC] ref-filter: add obj-type check in grab contents\n 2: eafb79bad62 ! 2: 0ffa549da6e [GSOC] ref-filter: add %(raw) atom\n @@ ref-filter.c: struct ref_formatting_state {\n \tstruct used_atom *atom;\n };\n \n ++#define ATOM_SIZE_UNSPECIFIED (-1)\n ++\n +#define ATOM_VALUE_INIT { \\\n -+\t.s_size = -1 \\\n ++\t.s_size = ATOM_SIZE_UNSPECIFIED \\\n +}\n +\n /*\n @@ ref-filter.c: static int append_atom(struct atom_value *v, struct ref_formatting\n +\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n \telse\n -\t\tstrbuf_addstr(&state->stack->output, v->s);\n -+\t\tif (v->s_size != -1)\n ++\t\tif (v->s_size != ATOM_SIZE_UNSPECIFIED)\n +\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n +\t\telse\n +\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n @@ ref-filter.c: static int populate_value(struct ref_array_item *ref, struct strbu\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n \n -+\t\tv->s_size = -1;\n ++\t\tv->s_size = ATOM_SIZE_UNSPECIFIED;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n @@ ref-filter.c: static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array\n -\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n -\t\t\t? strcasecmp : strcmp;\n -\t\tcmp = cmp_fn(va->s, vb->s);\n -+\t\tif (va->s_size == -1 && vb->s_size == -1) {\n ++\t\tif (va->s_size == ATOM_SIZE_UNSPECIFIED && vb->s_size == ATOM_SIZE_UNSPECIFIED) {\n +\t\t\tint (*cmp_fn)(const char *, const char *);\n +\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n +\t\t\t\t? strcasecmp : strcmp;\n +\t\t\tcmp = cmp_fn(va->s, vb->s);\n +\t\t} else {\n -+\t\t\tsize_t a_size = va->s_size == -1 ?\n ++\t\t\tsize_t a_size = va->s_size == ATOM_SIZE_UNSPECIFIED ?\n +\t\t\t\t\tstrlen(va->s) : va->s_size;\n -+\t\t\tsize_t b_size = vb->s_size == -1 ?\n ++\t\t\tsize_t b_size = vb->s_size == ATOM_SIZE_UNSPECIFIED ?\n +\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n +\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n +\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n 3: 00a29dd16d7 = 3: ac5d98647da [GSOC] ref-filter: --format=%(raw) re-support --perl\n 4: 3a8173b42ed = 4: 0722f546df9 [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n 5: 71249dfd4e0 ! 5: 2300b0463f2 [GSOC] ref-filter: add %(rest) atom\n @@ Metadata\n ## Commit message ##\n [GSOC] ref-filter: add %(rest) atom\n \n + %(rest) is a atom used for cat-file batch mode, which can split\n + the input lines at the first whitespace boundary, all characters\n + before that whitespace are considered to be the object name;\n + characters after that first run of whitespace (i.e., the \"rest\"\n + of the line) are output in place of the %(rest) atom.\n +\n In order to let \"cat-file --batch=%(rest)\" use the ref-filter\n - interface, add %(rest) atom for ref-filter. Introduce the\n - reject_atom() to reject the atom %(rest) for \"git for-each-ref\",\n - \"git branch\", \"git tag\" and \"git verify-tag\".\n + interface, add %(rest) atom for ref-filter.\n +\n + Introduce the reject_atom() to reject the atom %(rest) for\n + \"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n \n + Suggected-by: Jacob Keller \n Mentored-by: Christian Couder \n Mentored-by: Hariom Verma \n Signed-off-by: ZheNing Hu \n\n-- \ngitgitgadget\n"},{"id":"431114","messageId":"d77268bb1882a6cfbea740053baac5546ca5c5f1.1627136062.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v3.git.1627136061.gitgitgadget@gmail.com","subject":"[PATCH v3 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-24T14:14:17Z","receivedAt":"2021-07-24T14:14:28Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nOnly tag and commit objects use `grab_sub_body_contents()` to grab\nobject contents in the current codebase. We want to teach the\nfunction to also handle blobs and trees to get their raw data,\nwithout parsing a blob (whose contents looks like a commit or a tag)\nincorrectly as a commit or a tag. So it's needed to pass a\n`struct expand_data *data` instread of only `void *buf` to both\n`grab_sub_body_contents()` and `grab_values()` to be able to check\nthe object type.\n\nSkip the block of code that is specific to handling commits and tags\nearly when the given object is of a wrong type to help later\naddition to handle other types of objects in this function.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Junio C Hamano \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4db0e40ff4c..5cee6512fba 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n+static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n {\n \tint i;\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n+\tvoid *buf = data->content;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n@@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (strcmp(name, \"body\") &&\n-\t\t !starts_with(name, \"subject\") &&\n-\t\t !starts_with(name, \"trailers\") &&\n-\t\t !starts_with(name, \"contents\"))\n+\n+\t\tif ((data->type != OBJ_TAG &&\n+\t\t data->type != OBJ_COMMIT) ||\n+\t\t (strcmp(name, \"body\") &&\n+\t\t !starts_with(name, \"subject\") &&\n+\t\t !starts_with(name, \"trailers\") &&\n+\t\t !starts_with(name, \"contents\")))\n \t\t\tcontinue;\n \t\tif (!subpos)\n \t\t\tfind_subpos(buf,\n@@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n+static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n {\n+\tvoid *buf = data->content;\n+\n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n@@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi->content);\n+\t\tgrab_values(ref->value, deref, *obj, oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n-- \ngitgitgadget\n\n"},{"id":"431115","messageId":"0ffa549da6ef5c17e491970ca9b15719077e1f51.1627136062.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v3.git.1627136061.gitgitgadget@gmail.com","subject":"[PATCH v3 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-24T14:14:18Z","receivedAt":"2021-07-24T14:14:29Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd new formatting option `%(raw)`, which will print the raw\nobject data without any changes. It will help further to migrate\nall cat-file formatting logic from cat-file to ref-filter.\n\nThe raw data of blob, tree objects may contain '\\0', but most of\nthe logic in `ref-filter` depends on the output of the atom being\ntext (specifically, no embedded NULs in it).\n\nE.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()`\nadd the data to the buffer. The raw data of a tree object is\n`100644 one\\0...`, only the `100644 one` will be added to the buffer,\nwhich is incorrect.\n\nTherefore, we need to find a way to record the length of the\natom_value's member `s`. Although strbuf can already record the\nstring and its length, if we want to replace the type of atom_value's\nmember `s` with strbuf, many places in ref-filter that are filled\nwith dynamically allocated mermory in `v->s` are not easy to replace.\nAt the same time, we need to check if `v->s == NULL` in\npopulate_value(), and strbuf cannot easily distinguish NULL and empty\nstrings, but c-style \"const char *\" can do it. So add a new member in\n`struct atom_value`: `s_size`, which can record raw object size, it\ncan help us add raw object data to the buffer or compare two buffers\nwhich contain raw object data.\n\nNote that `--format=%(raw)` cannot be used with `--python`, `--shell`,\n`--tcl`, and `--perl` because if the binary raw data is passed to a\nvariable in such languages, these may not support arbitrary binary data\nin their string variable type.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Bagas Sanjaya \nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Felipe Contreras \nHelped-by: Phillip Wood \nHelped-by: Junio C Hamano \nBased-on-patch-by: Olga Telezhnaya \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 9 ++\n ref-filter.c | 144 +++++++++++++++----\n t/t6300-for-each-ref.sh | 216 +++++++++++++++++++++++++++++\n 3 files changed, 341 insertions(+), 28 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 2ae2478de70..cbb6f87d13f 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`,\n without angle brackets, and `:localpart` to get the part before the `@` symbol\n out of the trimmed email.\n \n+The raw data in an object is `raw`.\n+\n+raw:size::\n+\tThe raw data size of the object.\n+\n+Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n+`--perl` because such language may not support arbitrary binary data in their\n+string variable type.\n+\n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5cee6512fba..01545618101 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -144,6 +144,7 @@ enum atom_type {\n \tATOM_BODY,\n \tATOM_TRAILERS,\n \tATOM_CONTENTS,\n+\tATOM_RAW,\n \tATOM_UPSTREAM,\n \tATOM_PUSH,\n \tATOM_SYMREF,\n@@ -189,6 +190,9 @@ static struct used_atom {\n \t\t\tstruct process_trailer_options trailer_opts;\n \t\t\tunsigned int nlines;\n \t\t} contents;\n+\t\tstruct {\n+\t\t\tenum { RAW_BARE, RAW_LENGTH } option;\n+\t\t} raw_data;\n \t\tstruct {\n \t\t\tcmp_status cmp_status;\n \t\t\tconst char *str;\n@@ -426,6 +430,18 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n+static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\tconst char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\tatom->u.raw_data.option = RAW_BARE;\n+\telse if (!strcmp(arg, \"size\"))\n+\t\tatom->u.raw_data.option = RAW_LENGTH;\n+\telse\n+\t\treturn strbuf_addf_ret(err, -1, _(\"unrecognized %%(raw) argument: %s\"), arg);\n+\treturn 0;\n+}\n+\n static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -586,6 +602,7 @@ static struct {\n \t[ATOM_BODY] = { \"body\", SOURCE_OBJ, FIELD_STR, body_atom_parser },\n \t[ATOM_TRAILERS] = { \"trailers\", SOURCE_OBJ, FIELD_STR, trailers_atom_parser },\n \t[ATOM_CONTENTS] = { \"contents\", SOURCE_OBJ, FIELD_STR, contents_atom_parser },\n+\t[ATOM_RAW] = { \"raw\", SOURCE_OBJ, FIELD_STR, raw_atom_parser },\n \t[ATOM_UPSTREAM] = { \"upstream\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_PUSH] = { \"push\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_SYMREF] = { \"symref\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -620,12 +637,19 @@ struct ref_formatting_state {\n \n struct atom_value {\n \tconst char *s;\n+\tsize_t s_size;\n \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t struct strbuf *err);\n \tuintmax_t value; /* used for sorting when not FIELD_STR */\n \tstruct used_atom *atom;\n };\n \n+#define ATOM_SIZE_UNSPECIFIED (-1)\n+\n+#define ATOM_VALUE_INIT { \\\n+\t.s_size = ATOM_SIZE_UNSPECIFIED \\\n+}\n+\n /*\n * Used to parse format string and sort specifiers\n */\n@@ -644,13 +668,6 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \t\treturn strbuf_addf_ret(err, -1, _(\"malformed field name: %.*s\"),\n \t\t\t\t (int)(ep-atom), atom);\n \n-\t/* Do we have the atom already used elsewhere? */\n-\tfor (i = 0; i < used_atom_cnt; i++) {\n-\t\tint len = strlen(used_atom[i].name);\n-\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n-\t\t\treturn i;\n-\t}\n-\n \t/*\n \t * If the atom name has a colon, strip it and everything after\n \t * it off - it specifies the format for this entry, and\n@@ -660,6 +677,13 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \targ = memchr(sp, ':', ep - sp);\n \tatom_len = (arg ? arg : ep) - sp;\n \n+\t/* Do we have the atom already used elsewhere? */\n+\tfor (i = 0; i < used_atom_cnt; i++) {\n+\t\tint len = strlen(used_atom[i].name);\n+\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n+\t\t\treturn i;\n+\t}\n+\n \t/* Is the atom a valid one? */\n \tfor (i = 0; i < ARRAY_SIZE(valid_atom); i++) {\n \t\tint len = strlen(valid_atom[i].name);\n@@ -709,11 +733,14 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \treturn at;\n }\n \n-static void quote_formatting(struct strbuf *s, const char *str, int quote_style)\n+static void quote_formatting(struct strbuf *s, const char *str, size_t len, int quote_style)\n {\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n-\t\tstrbuf_addstr(s, str);\n+\t\tif (len != -1)\n+\t\t\tstrbuf_add(s, str, len);\n+\t\telse\n+\t\t\tstrbuf_addstr(s, str);\n \t\tbreak;\n \tcase QUOTE_SHELL:\n \t\tsq_quote_buf(s, str);\n@@ -740,9 +767,12 @@ static int append_atom(struct atom_value *v, struct ref_formatting_state *state,\n \t * encountered.\n \t */\n \tif (!state->stack->prev)\n-\t\tquote_formatting(&state->stack->output, v->s, state->quote_style);\n+\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n \telse\n-\t\tstrbuf_addstr(&state->stack->output, v->s);\n+\t\tif (v->s_size != ATOM_SIZE_UNSPECIFIED)\n+\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n+\t\telse\n+\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n \treturn 0;\n }\n \n@@ -842,21 +872,23 @@ static int if_atom_handler(struct atom_value *atomv, struct ref_formatting_state\n \treturn 0;\n }\n \n-static int is_empty(const char *s)\n+static int is_empty(struct strbuf *buf)\n {\n-\twhile (*s != '\\0') {\n-\t\tif (!isspace(*s))\n-\t\t\treturn 0;\n-\t\ts++;\n-\t}\n-\treturn 1;\n-}\n+\tconst char *cur = buf->buf;\n+\tconst char *end = buf->buf + buf->len;\n+\n+\twhile (cur != end && (isspace(*cur)))\n+\t\tcur++;\n+\n+\treturn cur == end;\n+ }\n \n static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t\t struct strbuf *err)\n {\n \tstruct ref_formatting_stack *cur = state->stack;\n \tstruct if_then_else *if_then_else = NULL;\n+\tsize_t str_len = 0;\n \n \tif (cur->at_end == if_then_else_handler)\n \t\tif_then_else = (struct if_then_else *)cur->at_end_data;\n@@ -867,18 +899,22 @@ static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_sta\n \tif (if_then_else->else_atom_seen)\n \t\treturn strbuf_addf_ret(err, -1, _(\"format: %%(then) atom used after %%(else)\"));\n \tif_then_else->then_atom_seen = 1;\n+\tif (if_then_else->str)\n+\t\tstr_len = strlen(if_then_else->str);\n \t/*\n \t * If the 'equals' or 'notequals' attribute is used then\n \t * perform the required comparison. If not, only non-empty\n \t * strings satisfy the 'if' condition.\n \t */\n \tif (if_then_else->cmp_status == COMPARE_EQUAL) {\n-\t\tif (!strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len == cur->output.len &&\n+\t\t !memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n \t} else if (if_then_else->cmp_status == COMPARE_UNEQUAL) {\n-\t\tif (strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len != cur->output.len ||\n+\t\t memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n-\t} else if (cur->output.len && !is_empty(cur->output.buf))\n+\t} else if (cur->output.len && !is_empty(&cur->output))\n \t\tif_then_else->condition_satisfied = 1;\n \tstrbuf_reset(&cur->output);\n \treturn 0;\n@@ -924,7 +960,7 @@ static int end_atom_handler(struct atom_value *atomv, struct ref_formatting_stat\n \t * only on the topmost supporting atom.\n \t */\n \tif (!current->prev->prev) {\n-\t\tquote_formatting(&s, current->output.buf, state->quote_style);\n+\t\tquote_formatting(&s, current->output.buf, current->output.len, state->quote_style);\n \t\tstrbuf_swap(¤t->output, &s);\n \t}\n \tstrbuf_release(&s);\n@@ -974,6 +1010,10 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\t\t\tdie(_(\"--format=%.*s cannot be used with\"\n+\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\n@@ -1367,12 +1407,25 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp\n \t\tstruct used_atom *atom = &used_atom[i];\n \t\tconst char *name = atom->name;\n \t\tstruct atom_value *v = &val[i];\n+\t\tenum atom_type atom_type = atom->atom_type;\n \n \t\tif (!!deref != (*name == '*'))\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n \n+\t\tif (atom_type == ATOM_RAW) {\n+\t\t\tunsigned long buf_size = data->size;\n+\n+\t\t\tif (atom->u.raw_data.option == RAW_BARE) {\n+\t\t\t\tv->s = xmemdupz(buf, buf_size);\n+\t\t\t\tv->s_size = buf_size;\n+\t\t\t} else if (atom->u.raw_data.option == RAW_LENGTH) {\n+\t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX, (uintmax_t)buf_size);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif ((data->type != OBJ_TAG &&\n \t\t data->type != OBJ_COMMIT) ||\n \t\t (strcmp(name, \"body\") &&\n@@ -1460,9 +1513,11 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"Eh? Object of type %d?\", obj->type);\n@@ -1766,6 +1821,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n \n+\t\tv->s_size = ATOM_SIZE_UNSPECIFIED;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n@@ -2369,6 +2425,19 @@ static int compare_detached_head(struct ref_array_item *a, struct ref_array_item\n \treturn 0;\n }\n \n+static int memcasecmp(const void *vs1, const void *vs2, size_t n)\n+{\n+\tconst char *s1 = vs1, *s2 = vs2;\n+\tconst char *end = s1 + n;\n+\n+\tfor (; s1 < end; s1++, s2++) {\n+\t\tint diff = tolower(*s1) - tolower(*s2);\n+\t\tif (diff)\n+\t\t\treturn diff;\n+\t}\n+\treturn 0;\n+}\n+\n static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b)\n {\n \tstruct atom_value *va, *vb;\n@@ -2389,10 +2458,29 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n \t} else if (s->sort_flags & REF_SORTING_VERSION) {\n \t\tcmp = versioncmp(va->s, vb->s);\n \t} else if (cmp_type == FIELD_STR) {\n-\t\tint (*cmp_fn)(const char *, const char *);\n-\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n-\t\t\t? strcasecmp : strcmp;\n-\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\tif (va->s_size == ATOM_SIZE_UNSPECIFIED && vb->s_size == ATOM_SIZE_UNSPECIFIED) {\n+\t\t\tint (*cmp_fn)(const char *, const char *);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? strcasecmp : strcmp;\n+\t\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\t} else {\n+\t\t\tsize_t a_size = va->s_size == ATOM_SIZE_UNSPECIFIED ?\n+\t\t\t\t\tstrlen(va->s) : va->s_size;\n+\t\t\tsize_t b_size = vb->s_size == ATOM_SIZE_UNSPECIFIED ?\n+\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n+\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? memcasecmp : memcmp;\n+\n+\t\t\tcmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n+\t\t\t\t a_size : b_size);\n+\t\t\tif (!cmp) {\n+\t\t\t\tif (a_size > b_size)\n+\t\t\t\t\tcmp = 1;\n+\t\t\t\telse if (a_size < b_size)\n+\t\t\t\t\tcmp = -1;\n+\t\t\t}\n+\t\t}\n \t} else {\n \t\tif (va->value < vb->value)\n \t\t\tcmp = -1;\n@@ -2491,7 +2579,7 @@ int format_ref_array_item(struct ref_array_item *info,\n \t\tappend_literal(cp, sp, &state);\n \t}\n \tif (format->need_color_reset_at_eol) {\n-\t\tstruct atom_value resetv;\n+\t\tstruct atom_value resetv = ATOM_VALUE_INIT;\n \t\tresetv.s = GIT_COLOR_RESET;\n \t\tif (append_atom(&resetv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 9e0214076b4..18554f62d94 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -130,6 +130,8 @@ test_atom head parent:short=10 ''\n test_atom head numparent 0\n test_atom head object ''\n test_atom head type ''\n+test_atom head raw \"$(git cat-file commit refs/heads/main)\n+\"\n test_atom head '*objectname' ''\n test_atom head '*objecttype' ''\n test_atom head author 'A U Thor 1151968724 +0200'\n@@ -221,6 +223,15 @@ test_atom tag contents 'Tagging at 1151968727\n '\n test_atom tag HEAD ' '\n \n+test_expect_success 'basic atom: refs/tags/testtag *raw' '\n+\tgit cat-file commit refs/tags/testtag^{} >expected &&\n+\tgit for-each-ref --format=\"%(*raw)\" refs/tags/testtag >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'Check invalid atoms names are errors' '\n \ttest_must_fail git for-each-ref --format=\"%(INVALID)\" refs/heads\n '\n@@ -686,6 +697,15 @@ test_atom refs/tags/signed-empty contents:body ''\n test_atom refs/tags/signed-empty contents:signature \"$sig\"\n test_atom refs/tags/signed-empty contents \"$sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-empty raw' '\n+\tgit cat-file tag refs/tags/signed-empty >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-empty >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-short subject 'subject line'\n test_atom refs/tags/signed-short subject:sanitize 'subject-line'\n test_atom refs/tags/signed-short contents:subject 'subject line'\n@@ -695,6 +715,15 @@ test_atom refs/tags/signed-short contents:signature \"$sig\"\n test_atom refs/tags/signed-short contents \"subject line\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-short raw' '\n+\tgit cat-file tag refs/tags/signed-short >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-short >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-long subject 'subject line'\n test_atom refs/tags/signed-long subject:sanitize 'subject-line'\n test_atom refs/tags/signed-long contents:subject 'subject line'\n@@ -708,6 +737,15 @@ test_atom refs/tags/signed-long contents \"subject line\n body contents\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-long raw' '\n+\tgit cat-file tag refs/tags/signed-long >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-long >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'set up refs pointing to tree and blob' '\n \tgit update-ref refs/mytrees/first refs/heads/main^{tree} &&\n \tgit update-ref refs/myblobs/first refs/heads/main:one\n@@ -720,6 +758,16 @@ test_atom refs/mytrees/first contents:body \"\"\n test_atom refs/mytrees/first contents:signature \"\"\n test_atom refs/mytrees/first contents \"\"\n \n+test_expect_success 'basic atom: refs/mytrees/first raw' '\n+\tgit cat-file tree refs/mytrees/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/mytrees/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_atom refs/myblobs/first subject \"\"\n test_atom refs/myblobs/first contents:subject \"\"\n test_atom refs/myblobs/first body \"\"\n@@ -727,6 +775,174 @@ test_atom refs/myblobs/first contents:body \"\"\n test_atom refs/myblobs/first contents:signature \"\"\n test_atom refs/myblobs/first contents \"\"\n \n+test_expect_success 'basic atom: refs/myblobs/first raw' '\n+\tgit cat-file blob refs/myblobs/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/myblobs/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'set up refs pointing to binary blob' '\n+\tprintf \"a\\0b\\0c\" >blob1 &&\n+\tprintf \"a\\0c\\0b\" >blob2 &&\n+\tprintf \"\\0a\\0b\\0c\" >blob3 &&\n+\tprintf \"abc\" >blob4 &&\n+\tprintf \"\\0 \\0 \\0 \" >blob5 &&\n+\tprintf \"\\0 \\0a\\0 \" >blob6 &&\n+\tprintf \" \" >blob7 &&\n+\t>blob8 &&\n+\tobj=$(git hash-object -w blob1) &&\n+\tgit update-ref refs/myblobs/blob1 \"$obj\" &&\n+\tobj=$(git hash-object -w blob2) &&\n+\tgit update-ref refs/myblobs/blob2 \"$obj\" &&\n+\tobj=$(git hash-object -w blob3) &&\n+\tgit update-ref refs/myblobs/blob3 \"$obj\" &&\n+\tobj=$(git hash-object -w blob4) &&\n+\tgit update-ref refs/myblobs/blob4 \"$obj\" &&\n+\tobj=$(git hash-object -w blob5) &&\n+\tgit update-ref refs/myblobs/blob5 \"$obj\" &&\n+\tobj=$(git hash-object -w blob6) &&\n+\tgit update-ref refs/myblobs/blob6 \"$obj\" &&\n+\tobj=$(git hash-object -w blob7) &&\n+\tgit update-ref refs/myblobs/blob7 \"$obj\" &&\n+\tobj=$(git hash-object -w blob8) &&\n+\tgit update-ref refs/myblobs/blob8 \"$obj\"\n+'\n+\n+test_expect_success 'Verify sorts with raw' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob7\n+\trefs/mytrees/first\n+\trefs/myblobs/first\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob4\n+\trefs/heads/main\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'Verify sorts with raw:size' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\trefs/myblobs/blob7\n+\trefs/heads/main\n+\trefs/myblobs/blob4\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/mytrees/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw:size \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:equals)' '\n+\tcat >expected <<-EOF &&\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\trefs/myblobs/blob4\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:equals=abc)%(raw)%(then)%(refname)%(else)not equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:notequals)' '\n+\tcat >expected <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/heads/main\n+\trefs/heads/newtag\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\tequals\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob7\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:notequals=abc)%(raw)%(then)%(refname)%(else)equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'empty raw refs with %(if)' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob1 not empty\n+\trefs/myblobs/blob2 not empty\n+\trefs/myblobs/blob3 not empty\n+\trefs/myblobs/blob4 not empty\n+\trefs/myblobs/blob5 not empty\n+\trefs/myblobs/blob6 not empty\n+\trefs/myblobs/blob7 empty\n+\trefs/myblobs/blob8 empty\n+\trefs/myblobs/first not empty\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname) %(if)%(raw)%(then)not empty%(else)empty%(end)\" \\\n+\t\trefs/myblobs/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success '%(raw) with --python must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --python\n+'\n+\n+test_expect_success '%(raw) with --tcl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n+'\n+\n+test_expect_success '%(raw) with --perl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+'\n+\n+test_expect_success '%(raw) with --shell must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --shell\n+'\n+\n+test_expect_success '%(raw) with --shell and --sort=raw must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --sort=raw --shell\n+'\n+\n+test_expect_success '%(raw:size) with --shell' '\n+\tgit for-each-ref --format=\"%(raw:size)\" | while read line\n+\tdo\n+\t\techo \"'\\''$line'\\''\" >>expect\n+\tdone &&\n+\tgit for-each-ref --format=\"%(raw:size)\" --shell >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for-each-ref --format compare with cat-file --batch' '\n+\tgit rev-parse refs/mytrees/first | git cat-file --batch >expected &&\n+\tgit for-each-ref --format=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'set up multiple-sort tags' '\n \tfor when in 100000 200000\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"431116","messageId":"ac5d98647dad31ebe165daa7cbf84f2b7e5fbe80.1627136062.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v3.git.1627136061.gitgitgadget@gmail.com","subject":"[PATCH v3 3/5] [GSOC] ref-filter: --format=%(raw) re-support --perl","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-24T14:14:19Z","receivedAt":"2021-07-24T14:14:30Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause the perl language can handle binary data correctly,\nadd the function perl_quote_buf_with_len(), which can specify\nthe length of the data and prevent the data from being truncated\nat '\\0' to help `--format=\"%(raw)\"` re-support `--perl`.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 4 ++--\n quote.c | 17 +++++++++++++++++\n quote.h | 1 +\n ref-filter.c | 15 +++++++++++----\n t/t6300-for-each-ref.sh | 19 +++++++++++++++++--\n 5 files changed, 48 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex cbb6f87d13f..6da899c6296 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -241,8 +241,8 @@ raw:size::\n \tThe raw data size of the object.\n \n Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n-`--perl` because such language may not support arbitrary binary data in their\n-string variable type.\n+because such language may not support arbitrary binary data in their string\n+variable type.\n \n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\ndiff --git a/quote.c b/quote.c\nindex 8a3a5e39eb1..26719d21d1e 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src)\n \tstrbuf_addch(sb, sq);\n }\n \n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len)\n+{\n+\tconst char sq = '\\'';\n+\tconst char bq = '\\\\';\n+\tconst char *c = src;\n+\tconst char *end = src + len;\n+\n+\tstrbuf_addch(sb, sq);\n+\twhile (c != end) {\n+\t\tif (*c == sq || *c == bq)\n+\t\t\tstrbuf_addch(sb, bq);\n+\t\tstrbuf_addch(sb, *c);\n+\t\tc++;\n+\t}\n+\tstrbuf_addch(sb, sq);\n+}\n+\n void python_quote_buf(struct strbuf *sb, const char *src)\n {\n \tconst char sq = '\\'';\ndiff --git a/quote.h b/quote.h\nindex 768cc6338e2..0fe69e264b0 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -94,6 +94,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne\n \n /* quoting as a string literal for other languages */\n void perl_quote_buf(struct strbuf *sb, const char *src);\n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len);\n void python_quote_buf(struct strbuf *sb, const char *src);\n void tcl_quote_buf(struct strbuf *sb, const char *src);\n void basic_regex_quote_buf(struct strbuf *sb, const char *src);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 01545618101..597354c941d 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -746,7 +746,10 @@ static void quote_formatting(struct strbuf *s, const char *str, size_t len, int\n \t\tsq_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PERL:\n-\t\tperl_quote_buf(s, str);\n+\t\tif (len != -1)\n+\t\t\tperl_quote_buf_with_len(s, str, len);\n+\t\telse\n+\t\t\tperl_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PYTHON:\n \t\tpython_quote_buf(s, str);\n@@ -1010,10 +1013,14 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n-\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n-\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\n+\t\tif ((format->quote_style == QUOTE_PYTHON ||\n+\t\t format->quote_style == QUOTE_SHELL ||\n+\t\t format->quote_style == QUOTE_TCL) &&\n+\t\t used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n \t\t\tdie(_(\"--format=%.*s cannot be used with\"\n-\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n+\t\t\t \"--python, --shell, --tcl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 18554f62d94..3d15d0a5360 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' '\n \ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n '\n \n-test_expect_success '%(raw) with --perl must fail' '\n-\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+test_expect_success '%(raw) with --perl' '\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob1 --perl | perl >actual &&\n+\tcmp blob1 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob3 --perl | perl >actual &&\n+\tcmp blob3 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob8 --perl | perl >actual &&\n+\tcmp blob8 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/first --perl | perl >actual &&\n+\tcmp one actual &&\n+\tgit cat-file tree refs/mytrees/first > expected &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/mytrees/first --perl | perl >actual &&\n+\tcmp expected actual\n '\n \n test_expect_success '%(raw) with --shell must fail' '\n-- \ngitgitgadget\n\n"},{"id":"431117","messageId":"0722f546df981081f69d6ba5c30f34f5033aea63.1627136062.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v3.git.1627136061.gitgitgadget@gmail.com","subject":"[PATCH v3 4/5] [GSOC] ref-filter: use non-const ref_format in *_atom_parser()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-24T14:14:20Z","receivedAt":"2021-07-24T14:14:34Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse non-const ref_format in *_atom_parser(), which can help us\nmodify the members of ref_format in *_atom_parser().\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/tag.c | 2 +-\n ref-filter.c | 44 ++++++++++++++++++++++----------------------\n ref-filter.h | 4 ++--\n 3 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 82fcfc09824..452558ec957 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref,\n \t\t const struct object_id *oid, void *cb_data)\n {\n \tint flags;\n-\tconst struct ref_format *format = cb_data;\n+\tstruct ref_format *format = cb_data;\n \tflags = GPG_VERIFY_VERBOSE;\n \n \tif (format->format)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 597354c941d..7cef839aaf1 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -226,7 +226,7 @@ static int strbuf_addf_ret(struct strbuf *sb, int ret, const char *fmt, ...)\n \treturn ret;\n }\n \n-static int color_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int color_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *color_value, struct strbuf *err)\n {\n \tif (!color_value)\n@@ -264,7 +264,7 @@ static int refname_atom_parser_internal(struct refname_atom *atom, const char *a\n \treturn 0;\n }\n \n-static int remote_ref_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int remote_ref_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct string_list params = STRING_LIST_INIT_DUP;\n@@ -311,7 +311,7 @@ static int remote_ref_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objecttype_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -323,7 +323,7 @@ static int objecttype_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objectsize_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -343,7 +343,7 @@ static int objectsize_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int deltabase_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int deltabase_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -355,7 +355,7 @@ static int deltabase_atom_parser(const struct ref_format *format, struct used_at\n \treturn 0;\n }\n \n-static int body_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int body_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -364,7 +364,7 @@ static int body_atom_parser(const struct ref_format *format, struct used_atom *a\n \treturn 0;\n }\n \n-static int subject_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int subject_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -376,7 +376,7 @@ static int subject_atom_parser(const struct ref_format *format, struct used_atom\n \treturn 0;\n }\n \n-static int trailers_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int trailers_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tatom->u.contents.trailer_opts.no_divider = 1;\n@@ -402,7 +402,7 @@ static int trailers_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int contents_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int contents_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -430,7 +430,7 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int raw_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -442,7 +442,7 @@ static int raw_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int oid_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -461,7 +461,7 @@ static int oid_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int person_email_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int person_email_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -475,7 +475,7 @@ static int person_email_atom_parser(const struct ref_format *format, struct used\n \treturn 0;\n }\n \n-static int refname_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int refname_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \treturn refname_atom_parser_internal(&atom->u.refname, arg, atom->name, err);\n@@ -492,7 +492,7 @@ static align_type parse_align_position(const char *s)\n \treturn -1;\n }\n \n-static int align_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int align_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct align *align = &atom->u.align;\n@@ -544,7 +544,7 @@ static int align_atom_parser(const struct ref_format *format, struct used_atom *\n \treturn 0;\n }\n \n-static int if_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -559,7 +559,7 @@ static int if_atom_parser(const struct ref_format *format, struct used_atom *ato\n \treturn 0;\n }\n \n-static int head_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n \tatom->u.head = resolve_refdup(\"HEAD\", RESOLVE_REF_READING, NULL, NULL);\n@@ -570,7 +570,7 @@ static struct {\n \tconst char *name;\n \tinfo_source source;\n \tcmp_type cmp_type;\n-\tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n+\tint (*parser)(struct ref_format *format, struct used_atom *atom,\n \t\t const char *arg, struct strbuf *err);\n } valid_atom[] = {\n \t[ATOM_REFNAME] = { \"refname\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -653,7 +653,7 @@ struct atom_value {\n /*\n * Used to parse format string and sort specifiers\n */\n-static int parse_ref_filter_atom(const struct ref_format *format,\n+static int parse_ref_filter_atom(struct ref_format *format,\n \t\t\t\t const char *atom, const char *ep,\n \t\t\t\t struct strbuf *err)\n {\n@@ -2557,9 +2557,9 @@ static void append_literal(const char *cp, const char *ep, struct ref_formatting\n }\n \n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n-\t\t\t struct strbuf *final_buf,\n-\t\t\t struct strbuf *error_buf)\n+\t\t\t struct ref_format *format,\n+\t\t\t struct strbuf *final_buf,\n+\t\t\t struct strbuf *error_buf)\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n@@ -2603,7 +2603,7 @@ int format_ref_array_item(struct ref_array_item *info,\n }\n \n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format)\n+\t\t struct ref_format *format)\n {\n \tstruct ref_array_item *ref_item;\n \tstruct strbuf output = STRBUF_INIT;\ndiff --git a/ref-filter.h b/ref-filter.h\nindex baf72a71896..74fb423fc89 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);\n void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting, unsigned int mask, int on);\n /* Based on the given format and quote_style, fill the strbuf */\n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n+\t\t\t struct ref_format *format,\n \t\t\t struct strbuf *final_buf,\n \t\t\t struct strbuf *error_buf);\n /* Parse a single sort specifier and add it to the list */\n@@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void);\n * name must be a fully qualified refname.\n */\n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format);\n+\t\t struct ref_format *format);\n \n /*\n * Push a single ref onto the array; this can be used to construct your own\n-- \ngitgitgadget\n\n"},{"id":"431118","messageId":"2300b0463f2fca76970ae7debcffa505909b8c4d.1627136062.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v3.git.1627136061.gitgitgadget@gmail.com","subject":"[PATCH v3 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-24T14:14:21Z","receivedAt":"2021-07-24T14:14:36Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\n%(rest) is a atom used for cat-file batch mode, which can split\nthe input lines at the first whitespace boundary, all characters\nbefore that whitespace are considered to be the object name;\ncharacters after that first run of whitespace (i.e., the \"rest\"\nof the line) are output in place of the %(rest) atom.\n\nIn order to let \"cat-file --batch=%(rest)\" use the ref-filter\ninterface, add %(rest) atom for ref-filter.\n\nIntroduce the reject_atom() to reject the atom %(rest) for\n\"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n\nSuggected-by: Jacob Keller \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 25 +++++++++++++++++++++++++\n ref-filter.h | 5 ++++-\n t/t3203-branch-output.sh | 4 ++++\n t/t6300-for-each-ref.sh | 4 ++++\n t/t7004-tag.sh | 4 ++++\n t/t7030-verify-tag.sh | 4 ++++\n 6 files changed, 45 insertions(+), 1 deletion(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 7cef839aaf1..85e9e657fe7 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -157,6 +157,7 @@ enum atom_type {\n \tATOM_IF,\n \tATOM_THEN,\n \tATOM_ELSE,\n+\tATOM_REST,\n };\n \n /*\n@@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"%%(rest) does not take arguments\"));\n+\tformat->use_rest = 1;\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n@@ -615,6 +625,7 @@ static struct {\n \t[ATOM_IF] = { \"if\", SOURCE_NONE, FIELD_STR, if_atom_parser },\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n+\t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -993,6 +1004,11 @@ static const char *find_next(const char *cp)\n \treturn NULL;\n }\n \n+static int reject_atom(enum atom_type atom_type)\n+{\n+\treturn atom_type == ATOM_REST;\n+}\n+\n /*\n * Make sure the format string is well formed, and parse out\n * the used atoms.\n@@ -1013,6 +1029,8 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (reject_atom(used_atom[at].atom_type))\n+\t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n \t\t format->quote_style == QUOTE_SHELL ||\n@@ -1932,6 +1950,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_REST) {\n+\t\t\tif (ref->rest)\n+\t\t\t\tv->s = xstrdup(ref->rest);\n+\t\t\telse\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2149,6 +2173,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n+\tref->rest = NULL;\n \n \treturn ref;\n }\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 74fb423fc89..c15dee8d6b9 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -38,6 +38,7 @@ struct ref_sorting {\n \n struct ref_array_item {\n \tstruct object_id objectname;\n+\tconst char *rest;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n@@ -76,14 +77,16 @@ struct ref_format {\n \t * verify_ref_format() afterwards to finalize.\n \t */\n \tconst char *format;\n+\tconst char *rest;\n \tint quote_style;\n+\tint use_rest;\n \tint use_color;\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n };\n \n-#define REF_FORMAT_INIT { NULL, 0, -1 }\n+#define REF_FORMAT_INIT { .use_color = -1 }\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\ndiff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh\nindex 5325b9f67a0..6e94c6db7b5 100755\n--- a/t/t3203-branch-output.sh\n+++ b/t/t3203-branch-output.sh\n@@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git branch with --format=%(rest) must fail' '\n+\ttest_must_fail git branch --format=\"%(rest)\" >actual\n+'\n+\n test_expect_success 'worktree colors correct' '\n \tcat >expect <<-EOF &&\n \t* (HEAD detached from fromtag)\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 3d15d0a5360..0d2e062f791 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' '\n \ttest_cmp expect actual.clean\n '\n \n+test_expect_success 'basic atom: rest must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(rest)\" refs/heads/main\n+'\n+\n test_expect_success 'trailer parsing not fooled by --- line' '\n \tgit commit --allow-empty -F - <<-\\EOF &&\n \tthis is the subject\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 2f72c5c6883..082be85dffc 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git tag -l with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git tag -l --format=\"%(rest)\" \"v1*\"\n+'\n+\n test_expect_success \"set up color tests\" '\n \techo \"v1.0\" >expect.color &&\n \techo \"v1.0\" >expect.bare &&\ndiff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh\nindex 3cefde9602b..10faa645157 100755\n--- a/t/t7030-verify-tag.sh\n+++ b/t/t7030-verify-tag.sh\n@@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success GPG 'verifying tag with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git verify-tag --format=\"%(rest)\" \"fourth-signed\"\n+'\n+\n test_expect_success GPG 'verifying a forged tag with --format should fail silently' '\n \ttest_must_fail git verify-tag --format=\"tagname : %(tag)\" $(cat forged1.tag) >actual-forged &&\n \ttest_must_be_empty actual-forged\n-- \ngitgitgadget\n"},{"id":"431120","messageId":"xmqqy29vk3sp.fsf@gitster.g","threadId":"56142","inReplyTo":"CAOLTT8SggCXkajPG3om+6zhM_K8fyAb2qTBDj40JJa1pszshzg@mail.gmail.com","subject":"Re: [PATCH v2 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-24T17:41:26Z","receivedAt":"2021-07-24T17:41:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"ZheNing Hu writes:\n\n>> It may make sense to\n>>\n>> * Turn atom_value.s_size field into ssize_t instead of size_t\n>>\n>\n> Will the conversion of size_t and ssize_t break -Wsign-conversion?\n> Although there is a lot of code in Git that has broken it, but I am not\n> sure if it is wise to use ssize_t here.\n>\n>> * Rewrite (v->s_size != -1) comparison to (v->s_size < 0)\n>>\n>\n> For size_t, this scheme is wrong.\n>\n>>\n>> Optionally we could introduce #define ATOM_SIZE_UNSPECIFIED (-1) and\n>> use it to initialize .s_size in ATOM_VALUE_INIT, but if we are just\n>> going to test \"is it negative, then it is not given\", then it probably\n>> is not needed.\n>>\n>\n> It seems that this is the only method left. Although I think\n> ATOM_SIZE_UNSPECIFIED\n> is not very useful becasue we already have ATOM_VALUE_INIT.\n\nSorry, but I think you misread what I wrote.\n\nThese three were not offered as \"you can do one of these three, pick\none you like\" choices. I meant to say \"I think it makes sense to do\nall these three things, but the last one is optional, doing only the\nfirst two may be good enough\". As the second one requires that the\nfirst is done, of course, doing only the second one would not make\nsense.\n\nAlso, you seem to have missed the distinction between _INIT and\n_UNSPECIFIED. You can initialize something to (1) a reasonable\ndefault value, or (2) unusable value that you can detect at runtime\nand notice that it was not set. If you called something to\nFOO_INIT, your readers cannot tell which one it is, but if you call\nit FOO_UNSPECIFIED, it is clear it is the latter case.\n\nIn many places, we do use ssize_t for \"normally this is size, but we\ncan express exception cases (like errors) by storing negative value\"\nin our codebase (grep for it), and I think the member in question is\nprime candidate for such use.\n\n"},{"id":"431128","messageId":"CA+P7+xrasxoSnbxNBVhwoOjejFw8UFH_Xufpts7-fGByknynHQ@mail.gmail.com","threadId":"56142","inReplyTo":"d77268bb1882a6cfbea740053baac5546ca5c5f1.1627136062.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2021-07-25T08:22:02Z","receivedAt":"2021-07-25T08:22:18Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Sat, Jul 24, 2021 at 7:14 AM ZheNing Hu via GitGitGadget\n wrote:\n>\n> From: ZheNing Hu \n>\n> Only tag and commit objects use `grab_sub_body_contents()` to grab\n> object contents in the current codebase. We want to teach the\n> function to also handle blobs and trees to get their raw data,\n> without parsing a blob (whose contents looks like a commit or a tag)\n> incorrectly as a commit or a tag. So it's needed to pass a\n> `struct expand_data *data` instread of only `void *buf` to both\n> `grab_sub_body_contents()` and `grab_values()` to be able to check\n> the object type.\n>\n> Skip the block of code that is specific to handling commits and tags\n> early when the given object is of a wrong type to help later\n> addition to handle other types of objects in this function.\n>\n> Mentored-by: Christian Couder \n> Mentored-by: Hariom Verma \n> Helped-by: Junio C Hamano \n> Signed-off-by: ZheNing Hu \n> ---\n\nNice. The addition of changing the type made sense and is spelled out nicely.\n\nReviewed-by: Jacob Keller \n\n> ref-filter.c | 24 +++++++++++++++---------\n> 1 file changed, 15 insertions(+), 9 deletions(-)\n>\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 4db0e40ff4c..5cee6512fba 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n> }\n>\n> /* See grab_values */\n> -static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n> +static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n> {\n> int i;\n> const char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n> size_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n> + void *buf = data->content;\n>\n> for (i = 0; i < used_atom_cnt; i++) {\n> struct used_atom *atom = &used_atom[i];\n> @@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n> continue;\n> if (deref)\n> name++;\n> - if (strcmp(name, \"body\") &&\n> - !starts_with(name, \"subject\") &&\n> - !starts_with(name, \"trailers\") &&\n> - !starts_with(name, \"contents\"))\n> +\n> + if ((data->type != OBJ_TAG &&\n> + data->type != OBJ_COMMIT) ||\n> + (strcmp(name, \"body\") &&\n> + !starts_with(name, \"subject\") &&\n> + !starts_with(name, \"trailers\") &&\n> + !starts_with(name, \"contents\")))\n> continue;\n> if (!subpos)\n> find_subpos(buf,\n> @@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n> * pointed at by the ref itself; otherwise it is the object the\n> * ref (which is a tag) refers to.\n> */\n> -static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n> +static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n> {\n> + void *buf = data->content;\n> +\n> switch (obj->type) {\n> case OBJ_TAG:\n> grab_tag_values(val, deref, obj);\n> - grab_sub_body_contents(val, deref, buf);\n> + grab_sub_body_contents(val, deref, data);\n> grab_person(\"tagger\", val, deref, buf);\n> break;\n> case OBJ_COMMIT:\n> grab_commit_values(val, deref, obj);\n> - grab_sub_body_contents(val, deref, buf);\n> + grab_sub_body_contents(val, deref, data);\n> grab_person(\"author\", val, deref, buf);\n> grab_person(\"committer\", val, deref, buf);\n> break;\n> @@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n> return strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n> oid_to_hex(&oi->oid), ref->refname);\n> }\n> - grab_values(ref->value, deref, *obj, oi->content);\n> + grab_values(ref->value, deref, *obj, oi);\n> }\n>\n> grab_common_values(ref->value, deref, oi);\n> --\n> gitgitgadget\n>\n"},{"id":"431129","messageId":"CA+P7+xqYMt1TvDY-GO--Aw5n9L7_06e-TExszcsS5oWWtFUxsA@mail.gmail.com","threadId":"56142","inReplyTo":"ac5d98647dad31ebe165daa7cbf84f2b7e5fbe80.1627136062.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/5] [GSOC] ref-filter: --format=%(raw) re-support --perl","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2021-07-25T08:27:48Z","receivedAt":"2021-07-25T08:28:16Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Sat, Jul 24, 2021 at 7:14 AM ZheNing Hu via GitGitGadget\n wrote:\n>\n> From: ZheNing Hu \n>\n> Because the perl language can handle binary data correctly,\n> add the function perl_quote_buf_with_len(), which can specify\n> the length of the data and prevent the data from being truncated\n> at '\\0' to help `--format=\"%(raw)\"` re-support `--perl`.\n>\n> Helped-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: ZheNing Hu \n> ---\n\nI presume that we used to support raw with perl before in cat-file? It\ndid read a bit weird to see \"Re-support\" in the title, because the\nprevious patch which introduced the raw atom when in ref-filter code\nnever supported perl. Still, I think it's fairly clear either way and\nthat change wouldn't deserve a reroll on its own.\n\nMakes sense. Reviewed-by: Jacob Keller \n\n> Documentation/git-for-each-ref.txt | 4 ++--\n> quote.c | 17 +++++++++++++++++\n> quote.h | 1 +\n> ref-filter.c | 15 +++++++++++----\n> t/t6300-for-each-ref.sh | 19 +++++++++++++++++--\n> 5 files changed, 48 insertions(+), 8 deletions(-)\n>\n> diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\n> index cbb6f87d13f..6da899c6296 100644\n> --- a/Documentation/git-for-each-ref.txt\n> +++ b/Documentation/git-for-each-ref.txt\n> @@ -241,8 +241,8 @@ raw:size::\n> The raw data size of the object.\n>\n> Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n> -`--perl` because such language may not support arbitrary binary data in their\n> -string variable type.\n> +because such language may not support arbitrary binary data in their string\n> +variable type.\n>\n> The message in a commit or a tag object is `contents`, from which\n> `contents:` can be used to extract various parts out of:\n> diff --git a/quote.c b/quote.c\n> index 8a3a5e39eb1..26719d21d1e 100644\n> --- a/quote.c\n> +++ b/quote.c\n> @@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src)\n> strbuf_addch(sb, sq);\n> }\n>\n> +void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len)\n> +{\n> + const char sq = '\\'';\n> + const char bq = '\\\\';\n> + const char *c = src;\n> + const char *end = src + len;\n> +\n> + strbuf_addch(sb, sq);\n> + while (c != end) {\n> + if (*c == sq || *c == bq)\n> + strbuf_addch(sb, bq);\n> + strbuf_addch(sb, *c);\n> + c++;\n> + }\n> + strbuf_addch(sb, sq);\n> +}\n> +\n> void python_quote_buf(struct strbuf *sb, const char *src)\n> {\n> const char sq = '\\'';\n> diff --git a/quote.h b/quote.h\n> index 768cc6338e2..0fe69e264b0 100644\n> --- a/quote.h\n> +++ b/quote.h\n> @@ -94,6 +94,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne\n>\n> /* quoting as a string literal for other languages */\n> void perl_quote_buf(struct strbuf *sb, const char *src);\n> +void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len);\n> void python_quote_buf(struct strbuf *sb, const char *src);\n> void tcl_quote_buf(struct strbuf *sb, const char *src);\n> void basic_regex_quote_buf(struct strbuf *sb, const char *src);\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 01545618101..597354c941d 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -746,7 +746,10 @@ static void quote_formatting(struct strbuf *s, const char *str, size_t len, int\n> sq_quote_buf(s, str);\n> break;\n> case QUOTE_PERL:\n> - perl_quote_buf(s, str);\n> + if (len != -1)\n> + perl_quote_buf_with_len(s, str, len);\n> + else\n> + perl_quote_buf(s, str);\n> break;\n> case QUOTE_PYTHON:\n> python_quote_buf(s, str);\n> @@ -1010,10 +1013,14 @@ int verify_ref_format(struct ref_format *format)\n> at = parse_ref_filter_atom(format, sp + 2, ep, &err);\n> if (at < 0)\n> die(\"%s\", err.buf);\n> - if (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n> - used_atom[at].u.raw_data.option == RAW_BARE)\n> +\n> + if ((format->quote_style == QUOTE_PYTHON ||\n> + format->quote_style == QUOTE_SHELL ||\n> + format->quote_style == QUOTE_TCL) &&\n> + used_atom[at].atom_type == ATOM_RAW &&\n> + used_atom[at].u.raw_data.option == RAW_BARE)\n> die(_(\"--format=%.*s cannot be used with\"\n> - \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n> + \"--python, --shell, --tcl\"), (int)(ep - sp - 2), sp + 2);\n> cp = ep + 1;\n>\n\nRight, so now we have to check all of the format quote styles instead\nof just checking that its set to any of them, since we want to\ncontinue in the case of perl.\n\nOk.\n\n> if (skip_prefix(used_atom[at].name, \"color:\", &color))\n> diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\n> index 18554f62d94..3d15d0a5360 100755\n> --- a/t/t6300-for-each-ref.sh\n> +++ b/t/t6300-for-each-ref.sh\n> @@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' '\n> test_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n> '\n>\n> -test_expect_success '%(raw) with --perl must fail' '\n> - test_must_fail git for-each-ref --format=\"%(raw)\" --perl\n> +test_expect_success '%(raw) with --perl' '\n> + git for-each-ref --format=\"\\$name= %(raw);\n> +print \\\"\\$name\\\"\" refs/myblobs/blob1 --perl | perl >actual &&\n> + cmp blob1 actual &&\n> + git for-each-ref --format=\"\\$name= %(raw);\n> +print \\\"\\$name\\\"\" refs/myblobs/blob3 --perl | perl >actual &&\n> + cmp blob3 actual &&\n> + git for-each-ref --format=\"\\$name= %(raw);\n> +print \\\"\\$name\\\"\" refs/myblobs/blob8 --perl | perl >actual &&\n> + cmp blob8 actual &&\n> + git for-each-ref --format=\"\\$name= %(raw);\n> +print \\\"\\$name\\\"\" refs/myblobs/first --perl | perl >actual &&\n> + cmp one actual &&\n> + git cat-file tree refs/mytrees/first > expected &&\n> + git for-each-ref --format=\"\\$name= %(raw);\n> +print \\\"\\$name\\\"\" refs/mytrees/first --perl | perl >actual &&\n> + cmp expected actual\n> '\n>\n> test_expect_success '%(raw) with --shell must fail' '\n> --\n> gitgitgadget\n>\n"},{"id":"431130","messageId":"CA+P7+xpfWA_e3rn2vfWBPixFJop-zomj9_N99n9ufe1MxV-1KA@mail.gmail.com","threadId":"56142","inReplyTo":"2300b0463f2fca76970ae7debcffa505909b8c4d.1627136062.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2021-07-25T08:29:52Z","receivedAt":"2021-07-25T08:30:07Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Sat, Jul 24, 2021 at 7:14 AM ZheNing Hu via GitGitGadget\n wrote:\n>\n> From: ZheNing Hu \n>\n> %(rest) is a atom used for cat-file batch mode, which can split\n> the input lines at the first whitespace boundary, all characters\n> before that whitespace are considered to be the object name;\n> characters after that first run of whitespace (i.e., the \"rest\"\n> of the line) are output in place of the %(rest) atom.\n>\n> In order to let \"cat-file --batch=%(rest)\" use the ref-filter\n> interface, add %(rest) atom for ref-filter.\n>\n> Introduce the reject_atom() to reject the atom %(rest) for\n> \"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n>\n> Suggected-by: Jacob Keller \n> Mentored-by: Christian Couder \n> Mentored-by: Hariom Verma \n> Signed-off-by: ZheNing Hu \n> ---\n\nThanks for the improved commit message here. That helps clarify what\nthis atom does and aids in understanding the implementation.\n\nReviewed-by: Jacob Keller \n\n> ref-filter.c | 25 +++++++++++++++++++++++++\n> ref-filter.h | 5 ++++-\n> t/t3203-branch-output.sh | 4 ++++\n> t/t6300-for-each-ref.sh | 4 ++++\n> t/t7004-tag.sh | 4 ++++\n> t/t7030-verify-tag.sh | 4 ++++\n> 6 files changed, 45 insertions(+), 1 deletion(-)\n>\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 7cef839aaf1..85e9e657fe7 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -157,6 +157,7 @@ enum atom_type {\n> ATOM_IF,\n> ATOM_THEN,\n> ATOM_ELSE,\n> + ATOM_REST,\n> };\n>\n> /*\n> @@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n> return 0;\n> }\n>\n> +static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n> + const char *arg, struct strbuf *err)\n> +{\n> + if (arg)\n> + return strbuf_addf_ret(err, -1, _(\"%%(rest) does not take arguments\"));\n> + format->use_rest = 1;\n> + return 0;\n> +}\n> +\n> static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n> const char *arg, struct strbuf *unused_err)\n> {\n> @@ -615,6 +625,7 @@ static struct {\n> [ATOM_IF] = { \"if\", SOURCE_NONE, FIELD_STR, if_atom_parser },\n> [ATOM_THEN] = { \"then\", SOURCE_NONE },\n> [ATOM_ELSE] = { \"else\", SOURCE_NONE },\n> + [ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n> /*\n> * Please update $__git_ref_fieldlist in git-completion.bash\n> * when you add new atoms\n> @@ -993,6 +1004,11 @@ static const char *find_next(const char *cp)\n> return NULL;\n> }\n>\n> +static int reject_atom(enum atom_type atom_type)\n> +{\n> + return atom_type == ATOM_REST;\n> +}\n> +\n> /*\n> * Make sure the format string is well formed, and parse out\n> * the used atoms.\n> @@ -1013,6 +1029,8 @@ int verify_ref_format(struct ref_format *format)\n> at = parse_ref_filter_atom(format, sp + 2, ep, &err);\n> if (at < 0)\n> die(\"%s\", err.buf);\n> + if (reject_atom(used_atom[at].atom_type))\n> + die(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n>\n> if ((format->quote_style == QUOTE_PYTHON ||\n> format->quote_style == QUOTE_SHELL ||\n> @@ -1932,6 +1950,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n> v->handler = else_atom_handler;\n> v->s = xstrdup(\"\");\n> continue;\n> + } else if (atom_type == ATOM_REST) {\n> + if (ref->rest)\n> + v->s = xstrdup(ref->rest);\n> + else\n> + v->s = xstrdup(\"\");\n> + continue;\n> } else\n> continue;\n>\n> @@ -2149,6 +2173,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n>\n> FLEX_ALLOC_STR(ref, refname, refname);\n> oidcpy(&ref->objectname, oid);\n> + ref->rest = NULL;\n>\n> return ref;\n> }\n> diff --git a/ref-filter.h b/ref-filter.h\n> index 74fb423fc89..c15dee8d6b9 100644\n> --- a/ref-filter.h\n> +++ b/ref-filter.h\n> @@ -38,6 +38,7 @@ struct ref_sorting {\n>\n> struct ref_array_item {\n> struct object_id objectname;\n> + const char *rest;\n> int flag;\n> unsigned int kind;\n> const char *symref;\n> @@ -76,14 +77,16 @@ struct ref_format {\n> * verify_ref_format() afterwards to finalize.\n> */\n> const char *format;\n> + const char *rest;\n> int quote_style;\n> + int use_rest;\n> int use_color;\n>\n> /* Internal state to ref-filter */\n> int need_color_reset_at_eol;\n> };\n>\n> -#define REF_FORMAT_INIT { NULL, 0, -1 }\n> +#define REF_FORMAT_INIT { .use_color = -1 }\n>\n> /* Macros for checking --merged and --no-merged options */\n> #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\n> diff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh\n> index 5325b9f67a0..6e94c6db7b5 100755\n> --- a/t/t3203-branch-output.sh\n> +++ b/t/t3203-branch-output.sh\n> @@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' '\n> test_cmp expect actual\n> '\n>\n> +test_expect_success 'git branch with --format=%(rest) must fail' '\n> + test_must_fail git branch --format=\"%(rest)\" >actual\n> +'\n> +\n> test_expect_success 'worktree colors correct' '\n> cat >expect <<-EOF &&\n> * (HEAD detached from fromtag)\n> diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\n> index 3d15d0a5360..0d2e062f791 100755\n> --- a/t/t6300-for-each-ref.sh\n> +++ b/t/t6300-for-each-ref.sh\n> @@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' '\n> test_cmp expect actual.clean\n> '\n>\n> +test_expect_success 'basic atom: rest must fail' '\n> + test_must_fail git for-each-ref --format=\"%(rest)\" refs/heads/main\n> +'\n> +\n> test_expect_success 'trailer parsing not fooled by --- line' '\n> git commit --allow-empty -F - <<-\\EOF &&\n> this is the subject\n> diff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\n> index 2f72c5c6883..082be85dffc 100755\n> --- a/t/t7004-tag.sh\n> +++ b/t/t7004-tag.sh\n> @@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' '\n> test_cmp expect actual\n> '\n>\n> +test_expect_success 'git tag -l with --format=\"%(rest)\" must fail' '\n> + test_must_fail git tag -l --format=\"%(rest)\" \"v1*\"\n> +'\n> +\n> test_expect_success \"set up color tests\" '\n> echo \"v1.0\" >expect.color &&\n> echo \"v1.0\" >expect.bare &&\n> diff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh\n> index 3cefde9602b..10faa645157 100755\n> --- a/t/t7030-verify-tag.sh\n> +++ b/t/t7030-verify-tag.sh\n> @@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' '\n> test_cmp expect actual\n> '\n>\n> +test_expect_success GPG 'verifying tag with --format=\"%(rest)\" must fail' '\n> + test_must_fail git verify-tag --format=\"%(rest)\" \"fourth-signed\"\n> +'\n> +\n> test_expect_success GPG 'verifying a forged tag with --format should fail silently' '\n> test_must_fail git verify-tag --format=\"tagname : %(tag)\" $(cat forged1.tag) >actual-forged &&\n> test_must_be_empty actual-forged\n> --\n> gitgitgadget\n"},{"id":"431133","messageId":"CAOLTT8Sm=qs0K84C=yJOL7Ct5GUeHYe=ODXiGpwCSx11vgEnwA@mail.gmail.com","threadId":"56142","inReplyTo":"xmqqy29vk3sp.fsf@gitster.g","subject":"Re: [PATCH v2 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-25T12:47:16Z","receivedAt":"2021-07-25T12:46:53Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年7月25日周日 上午1:41写道：\n>\n> ZheNing Hu writes:\n>\n> >> It may make sense to\n> >>\n> >> * Turn atom_value.s_size field into ssize_t instead of size_t\n> >>\n> >\n> > Will the conversion of size_t and ssize_t break -Wsign-conversion?\n> > Although there is a lot of code in Git that has broken it, but I am not\n> > sure if it is wise to use ssize_t here.\n> >\n> >> * Rewrite (v->s_size != -1) comparison to (v->s_size < 0)\n> >>\n> >\n> > For size_t, this scheme is wrong.\n> >\n> >>\n> >> Optionally we could introduce #define ATOM_SIZE_UNSPECIFIED (-1) and\n> >> use it to initialize .s_size in ATOM_VALUE_INIT, but if we are just\n> >> going to test \"is it negative, then it is not given\", then it probably\n> >> is not needed.\n> >>\n> >\n> > It seems that this is the only method left. Although I think\n> > ATOM_SIZE_UNSPECIFIED\n> > is not very useful becasue we already have ATOM_VALUE_INIT.\n>\n> Sorry, but I think you misread what I wrote.\n>\n> These three were not offered as \"you can do one of these three, pick\n> one you like\" choices. I meant to say \"I think it makes sense to do\n> all these three things, but the last one is optional, doing only the\n> first two may be good enough\". As the second one requires that the\n> first is done, of course, doing only the second one would not make\n> sense.\n>\n\nOK. I know it now, I will do all of them.\n\n> Also, you seem to have missed the distinction between _INIT and\n> _UNSPECIFIED. You can initialize something to (1) a reasonable\n> default value, or (2) unusable value that you can detect at runtime\n> and notice that it was not set. If you called something to\n> FOO_INIT, your readers cannot tell which one it is, but if you call\n> it FOO_UNSPECIFIED, it is clear it is the latter case.\n>\n\nThanks for clarification, I understand the difference between them now.\n\n> In many places, we do use ssize_t for \"normally this is size, but we\n> can express exception cases (like errors) by storing negative value\"\n> in our codebase (grep for it), and I think the member in question is\n> prime candidate for such use.\n>\n\nYeah, as abspath.c:137, `ssize_t len` used to distinguish situations\nif an error is\nreturned from strbuf_readlink().\n\n ssize_t len;\n ...\n len = strbuf_readlink(&symlink, resolved->buf,\n st.st_size);\n if (len < 0) {\n if (flags & REALPATH_DIE_ON_ERROR)\n die_errno(\"Invalid symlink '%s'\",\n resolved->buf);\n else\n goto error_out;\n }\n ...\n\nOr just read() and write(), they return is ssize_t too,\nwhich can return -1 and set errno when an exception occurs.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"431149","messageId":"CAOLTT8TR-nYePc65=gJ8DtbuXtOgLs+p52k80iGqLKH6rkMarQ@mail.gmail.com","threadId":"56142","inReplyTo":"CA+P7+xqYMt1TvDY-GO--Aw5n9L7_06e-TExszcsS5oWWtFUxsA@mail.gmail.com","subject":"Re: [PATCH v3 3/5] [GSOC] ref-filter: --format=%(raw) re-support --perl","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-25T13:18:08Z","receivedAt":"2021-07-25T13:17:41Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Jacob Keller 于2021年7月25日周日 下午4:27写道：\n>\n> On Sat, Jul 24, 2021 at 7:14 AM ZheNing Hu via GitGitGadget\n> wrote:\n> >\n> > From: ZheNing Hu \n> >\n> > Because the perl language can handle binary data correctly,\n> > add the function perl_quote_buf_with_len(), which can specify\n> > the length of the data and prevent the data from being truncated\n> > at '\\0' to help `--format=\"%(raw)\"` re-support `--perl`.\n> >\n> > Helped-by: Ævar Arnfjörð Bjarmason \n> > Signed-off-by: ZheNing Hu \n> > ---\n>\n> I presume that we used to support raw with perl before in cat-file? It\n> did read a bit weird to see \"Re-support\" in the title, because the\n> previous patch which introduced the raw atom when in ref-filter code\n> never supported perl. Still, I think it's fairly clear either way and\n> that change wouldn't deserve a reroll on its own.\n>\n\nNo, we haven't support it. It can be said that this commit is only split from\nthe previous commit. Therefore, \"re-support\" is indeed wrong. I will modify it.\n\n> Makes sense. Reviewed-by: Jacob Keller \n>\n> > @@ -1010,10 +1013,14 @@ int verify_ref_format(struct ref_format *format)\n> > at = parse_ref_filter_atom(format, sp + 2, ep, &err);\n> > if (at < 0)\n> > die(\"%s\", err.buf);\n> > - if (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n> > - used_atom[at].u.raw_data.option == RAW_BARE)\n> > +\n> > + if ((format->quote_style == QUOTE_PYTHON ||\n> > + format->quote_style == QUOTE_SHELL ||\n> > + format->quote_style == QUOTE_TCL) &&\n> > + used_atom[at].atom_type == ATOM_RAW &&\n> > + used_atom[at].u.raw_data.option == RAW_BARE)\n> > die(_(\"--format=%.*s cannot be used with\"\n> > - \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n> > + \"--python, --shell, --tcl\"), (int)(ep - sp - 2), sp + 2);\n> > cp = ep + 1;\n> >\n>\n> Right, so now we have to check all of the format quote styles instead\n> of just checking that its set to any of them, since we want to\n> continue in the case of perl.\n>\n> Ok.\n>\n\nThanks.\n--\nZheNing Hu\n"},{"id":"431157","messageId":"0e294ff1626ef2b7992e84e8d960f5058bd39bdc.1627270010.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v4.git.1627270010.gitgitgadget@gmail.com","subject":"[PATCH v4 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-26T03:26:46Z","receivedAt":"2021-07-26T03:26:57Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nOnly tag and commit objects use `grab_sub_body_contents()` to grab\nobject contents in the current codebase. We want to teach the\nfunction to also handle blobs and trees to get their raw data,\nwithout parsing a blob (whose contents looks like a commit or a tag)\nincorrectly as a commit or a tag. So it's needed to pass a\n`struct expand_data *data` instread of only `void *buf` to both\n`grab_sub_body_contents()` and `grab_values()` to be able to check\nthe object type.\n\nSkip the block of code that is specific to handling commits and tags\nearly when the given object is of a wrong type to help later\naddition to handle other types of objects in this function.\n\nReviewed-by: Jacob Keller \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Junio C Hamano \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4db0e40ff4c..5cee6512fba 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n+static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n {\n \tint i;\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n+\tvoid *buf = data->content;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n@@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (strcmp(name, \"body\") &&\n-\t\t !starts_with(name, \"subject\") &&\n-\t\t !starts_with(name, \"trailers\") &&\n-\t\t !starts_with(name, \"contents\"))\n+\n+\t\tif ((data->type != OBJ_TAG &&\n+\t\t data->type != OBJ_COMMIT) ||\n+\t\t (strcmp(name, \"body\") &&\n+\t\t !starts_with(name, \"subject\") &&\n+\t\t !starts_with(name, \"trailers\") &&\n+\t\t !starts_with(name, \"contents\")))\n \t\t\tcontinue;\n \t\tif (!subpos)\n \t\t\tfind_subpos(buf,\n@@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n+static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n {\n+\tvoid *buf = data->content;\n+\n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n@@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi->content);\n+\t\tgrab_values(ref->value, deref, *obj, oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n-- \ngitgitgadget\n\n"},{"id":"431158","messageId":"pull.1000.v4.git.1627270010.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v3.git.1627136061.gitgitgadget@gmail.com","subject":"[PATCH v4 0/5] [GSOC] ref-filter: add %(raw) and %(rest) atoms","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-26T03:26:45Z","receivedAt":"2021-07-26T03:26:59Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"This patch series is split from my main patch series zh/ref-filter-raw-data,\nto make things easier for reviewers to read. It's last version is here:\nhttps://lore.kernel.org/git/pull.993.v2.git.1626363626.gitgitgadget@gmail.com/\n\nThis patch series provided %(raw) and %(rest) for ref-filter, which will be\nused by zh/cat-file-reuse-ref-filter-logic later.\n\nv3--->v4:\n\n 1. Change the type of atom_value->s_size from size_t to ssize_t, and change\n the type of quote_formatting() parameters len from size_t to ssize_t at\n the same time. All of if (v->s_size == -1) change to if (v->s_size < 0).\n 2. Correct the commit subject of \"[GSOC] ref-filter: --format=%(raw)\n re-support --perl\" to \"[GSOC] ref-filter: --format=%(raw) support\n --perl\" and its commit messages.\n\nv3:\nhttps://lore.kernel.org/git/pull.1000.v3.git.1627136061.gitgitgadget@gmail.com/\n\nZheNing Hu (5):\n [GSOC] ref-filter: add obj-type check in grab contents\n [GSOC] ref-filter: add %(raw) atom\n [GSOC] ref-filter: --format=%(raw) support --perl\n [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n [GSOC] ref-filter: add %(rest) atom\n\n Documentation/git-for-each-ref.txt | 9 ++\n builtin/tag.c | 2 +-\n quote.c | 17 ++\n quote.h | 1 +\n ref-filter.c | 243 ++++++++++++++++++++++-------\n ref-filter.h | 9 +-\n t/t3203-branch-output.sh | 4 +\n t/t6300-for-each-ref.sh | 235 ++++++++++++++++++++++++++++\n t/t7004-tag.sh | 4 +\n t/t7030-verify-tag.sh | 4 +\n 10 files changed, 465 insertions(+), 63 deletions(-)\n\n\nbase-commit: daab8a564f8bbac55f70f8bf86c070e001a9b006\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1000%2Fadlternative%2Fref-filter-raw-data-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1000/adlternative/ref-filter-raw-data-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1000\n\nRange-diff vs v3:\n\n 1: d77268bb188 ! 1: 0e294ff1626 [GSOC] ref-filter: add obj-type check in grab contents\n @@ Commit message\n early when the given object is of a wrong type to help later\n addition to handle other types of objects in this function.\n \n + Reviewed-by: Jacob Keller \n Mentored-by: Christian Couder \n Mentored-by: Hariom Verma \n Helped-by: Junio C Hamano \n 2: 0ffa549da6e ! 2: 96d79a3d16b [GSOC] ref-filter: add %(raw) atom\n @@ Commit message\n variable in such languages, these may not support arbitrary binary data\n in their string variable type.\n \n + Reviewed-by: Jacob Keller \n Mentored-by: Christian Couder \n Mentored-by: Hariom Verma \n Helped-by: Bagas Sanjaya \n @@ ref-filter.c: struct ref_formatting_state {\n \n struct atom_value {\n \tconst char *s;\n -+\tsize_t s_size;\n ++\tssize_t s_size;\n \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t struct strbuf *err);\n \tuintmax_t value; /* used for sorting when not FIELD_STR */\n @@ ref-filter.c: static int parse_ref_filter_atom(const struct ref_format *format,\n }\n \n -static void quote_formatting(struct strbuf *s, const char *str, int quote_style)\n -+static void quote_formatting(struct strbuf *s, const char *str, size_t len, int quote_style)\n ++static void quote_formatting(struct strbuf *s, const char *str, ssize_t len, int quote_style)\n {\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n -\t\tstrbuf_addstr(s, str);\n -+\t\tif (len != -1)\n -+\t\t\tstrbuf_add(s, str, len);\n -+\t\telse\n ++\t\tif (len < 0)\n +\t\t\tstrbuf_addstr(s, str);\n ++\t\telse\n ++\t\t\tstrbuf_add(s, str, len);\n \t\tbreak;\n \tcase QUOTE_SHELL:\n \t\tsq_quote_buf(s, str);\n @@ ref-filter.c: static int append_atom(struct atom_value *v, struct ref_formatting\n \t */\n \tif (!state->stack->prev)\n -\t\tquote_formatting(&state->stack->output, v->s, state->quote_style);\n +-\telse\n +\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n - \telse\n --\t\tstrbuf_addstr(&state->stack->output, v->s);\n -+\t\tif (v->s_size != ATOM_SIZE_UNSPECIFIED)\n -+\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n -+\t\telse\n -+\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n ++\telse if (v->s_size < 0)\n + \t\tstrbuf_addstr(&state->stack->output, v->s);\n ++\telse\n ++\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n \treturn 0;\n }\n \n @@ ref-filter.c: static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array\n -\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n -\t\t\t? strcasecmp : strcmp;\n -\t\tcmp = cmp_fn(va->s, vb->s);\n -+\t\tif (va->s_size == ATOM_SIZE_UNSPECIFIED && vb->s_size == ATOM_SIZE_UNSPECIFIED) {\n ++\t\tif (va->s_size < 0 && vb->s_size < 0) {\n +\t\t\tint (*cmp_fn)(const char *, const char *);\n +\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n +\t\t\t\t? strcasecmp : strcmp;\n +\t\t\tcmp = cmp_fn(va->s, vb->s);\n +\t\t} else {\n -+\t\t\tsize_t a_size = va->s_size == ATOM_SIZE_UNSPECIFIED ?\n ++\t\t\tsize_t a_size = va->s_size < 0 ?\n +\t\t\t\t\tstrlen(va->s) : va->s_size;\n -+\t\t\tsize_t b_size = vb->s_size == ATOM_SIZE_UNSPECIFIED ?\n ++\t\t\tsize_t b_size = vb->s_size < 0 ?\n +\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n +\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n +\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n 3: ac5d98647da ! 3: d14fae52080 [GSOC] ref-filter: --format=%(raw) re-support --perl\n @@ Metadata\n Author: ZheNing Hu \n \n ## Commit message ##\n - [GSOC] ref-filter: --format=%(raw) re-support --perl\n + [GSOC] ref-filter: --format=%(raw) support --perl\n \n Because the perl language can handle binary data correctly,\n add the function perl_quote_buf_with_len(), which can specify\n the length of the data and prevent the data from being truncated\n - at '\\0' to help `--format=\"%(raw)\"` re-support `--perl`.\n + at '\\0' to help `--format=\"%(raw)\"` support `--perl`.\n \n + Reviewed-by: Jacob Keller \n Helped-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: ZheNing Hu \n \n @@ quote.h: char *quote_path(const char *in, const char *prefix, struct strbuf *out\n void basic_regex_quote_buf(struct strbuf *sb, const char *src);\n \n ## ref-filter.c ##\n -@@ ref-filter.c: static void quote_formatting(struct strbuf *s, const char *str, size_t len, int\n +@@ ref-filter.c: static void quote_formatting(struct strbuf *s, const char *str, ssize_t len, int\n \t\tsq_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PERL:\n -\t\tperl_quote_buf(s, str);\n -+\t\tif (len != -1)\n -+\t\t\tperl_quote_buf_with_len(s, str, len);\n -+\t\telse\n ++\t\tif (len < 0)\n +\t\t\tperl_quote_buf(s, str);\n ++\t\telse\n ++\t\t\tperl_quote_buf_with_len(s, str, len);\n \t\tbreak;\n \tcase QUOTE_PYTHON:\n \t\tpython_quote_buf(s, str);\n 4: 0722f546df9 = 4: 0aa398e8a8f [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n 5: 2300b0463f2 ! 5: 207cc512964 [GSOC] ref-filter: add %(rest) atom\n @@ Commit message\n Introduce the reject_atom() to reject the atom %(rest) for\n \"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n \n + Reviewed-by: Jacob Keller \n Suggected-by: Jacob Keller \n Mentored-by: Christian Couder \n Mentored-by: Hariom Verma \n\n-- \ngitgitgadget\n"},{"id":"431159","messageId":"96d79a3d16bcfeb18e05d95c76b93cdba9f4d6de.1627270010.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v4.git.1627270010.gitgitgadget@gmail.com","subject":"[PATCH v4 2/5] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-26T03:26:47Z","receivedAt":"2021-07-26T03:27:00Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd new formatting option `%(raw)`, which will print the raw\nobject data without any changes. It will help further to migrate\nall cat-file formatting logic from cat-file to ref-filter.\n\nThe raw data of blob, tree objects may contain '\\0', but most of\nthe logic in `ref-filter` depends on the output of the atom being\ntext (specifically, no embedded NULs in it).\n\nE.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()`\nadd the data to the buffer. The raw data of a tree object is\n`100644 one\\0...`, only the `100644 one` will be added to the buffer,\nwhich is incorrect.\n\nTherefore, we need to find a way to record the length of the\natom_value's member `s`. Although strbuf can already record the\nstring and its length, if we want to replace the type of atom_value's\nmember `s` with strbuf, many places in ref-filter that are filled\nwith dynamically allocated mermory in `v->s` are not easy to replace.\nAt the same time, we need to check if `v->s == NULL` in\npopulate_value(), and strbuf cannot easily distinguish NULL and empty\nstrings, but c-style \"const char *\" can do it. So add a new member in\n`struct atom_value`: `s_size`, which can record raw object size, it\ncan help us add raw object data to the buffer or compare two buffers\nwhich contain raw object data.\n\nNote that `--format=%(raw)` cannot be used with `--python`, `--shell`,\n`--tcl`, and `--perl` because if the binary raw data is passed to a\nvariable in such languages, these may not support arbitrary binary data\nin their string variable type.\n\nReviewed-by: Jacob Keller \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Bagas Sanjaya \nHelped-by: Ævar Arnfjörð Bjarmason \nHelped-by: Felipe Contreras \nHelped-by: Phillip Wood \nHelped-by: Junio C Hamano \nBased-on-patch-by: Olga Telezhnaya \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 9 ++\n ref-filter.c | 143 +++++++++++++++----\n t/t6300-for-each-ref.sh | 216 +++++++++++++++++++++++++++++\n 3 files changed, 340 insertions(+), 28 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 2ae2478de70..cbb6f87d13f 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`,\n without angle brackets, and `:localpart` to get the part before the `@` symbol\n out of the trimmed email.\n \n+The raw data in an object is `raw`.\n+\n+raw:size::\n+\tThe raw data size of the object.\n+\n+Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n+`--perl` because such language may not support arbitrary binary data in their\n+string variable type.\n+\n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5cee6512fba..0d5eb91ed54 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -144,6 +144,7 @@ enum atom_type {\n \tATOM_BODY,\n \tATOM_TRAILERS,\n \tATOM_CONTENTS,\n+\tATOM_RAW,\n \tATOM_UPSTREAM,\n \tATOM_PUSH,\n \tATOM_SYMREF,\n@@ -189,6 +190,9 @@ static struct used_atom {\n \t\t\tstruct process_trailer_options trailer_opts;\n \t\t\tunsigned int nlines;\n \t\t} contents;\n+\t\tstruct {\n+\t\t\tenum { RAW_BARE, RAW_LENGTH } option;\n+\t\t} raw_data;\n \t\tstruct {\n \t\t\tcmp_status cmp_status;\n \t\t\tconst char *str;\n@@ -426,6 +430,18 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n+static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\tconst char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\tatom->u.raw_data.option = RAW_BARE;\n+\telse if (!strcmp(arg, \"size\"))\n+\t\tatom->u.raw_data.option = RAW_LENGTH;\n+\telse\n+\t\treturn strbuf_addf_ret(err, -1, _(\"unrecognized %%(raw) argument: %s\"), arg);\n+\treturn 0;\n+}\n+\n static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -586,6 +602,7 @@ static struct {\n \t[ATOM_BODY] = { \"body\", SOURCE_OBJ, FIELD_STR, body_atom_parser },\n \t[ATOM_TRAILERS] = { \"trailers\", SOURCE_OBJ, FIELD_STR, trailers_atom_parser },\n \t[ATOM_CONTENTS] = { \"contents\", SOURCE_OBJ, FIELD_STR, contents_atom_parser },\n+\t[ATOM_RAW] = { \"raw\", SOURCE_OBJ, FIELD_STR, raw_atom_parser },\n \t[ATOM_UPSTREAM] = { \"upstream\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_PUSH] = { \"push\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_SYMREF] = { \"symref\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -620,12 +637,19 @@ struct ref_formatting_state {\n \n struct atom_value {\n \tconst char *s;\n+\tssize_t s_size;\n \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t struct strbuf *err);\n \tuintmax_t value; /* used for sorting when not FIELD_STR */\n \tstruct used_atom *atom;\n };\n \n+#define ATOM_SIZE_UNSPECIFIED (-1)\n+\n+#define ATOM_VALUE_INIT { \\\n+\t.s_size = ATOM_SIZE_UNSPECIFIED \\\n+}\n+\n /*\n * Used to parse format string and sort specifiers\n */\n@@ -644,13 +668,6 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \t\treturn strbuf_addf_ret(err, -1, _(\"malformed field name: %.*s\"),\n \t\t\t\t (int)(ep-atom), atom);\n \n-\t/* Do we have the atom already used elsewhere? */\n-\tfor (i = 0; i < used_atom_cnt; i++) {\n-\t\tint len = strlen(used_atom[i].name);\n-\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n-\t\t\treturn i;\n-\t}\n-\n \t/*\n \t * If the atom name has a colon, strip it and everything after\n \t * it off - it specifies the format for this entry, and\n@@ -660,6 +677,13 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \targ = memchr(sp, ':', ep - sp);\n \tatom_len = (arg ? arg : ep) - sp;\n \n+\t/* Do we have the atom already used elsewhere? */\n+\tfor (i = 0; i < used_atom_cnt; i++) {\n+\t\tint len = strlen(used_atom[i].name);\n+\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n+\t\t\treturn i;\n+\t}\n+\n \t/* Is the atom a valid one? */\n \tfor (i = 0; i < ARRAY_SIZE(valid_atom); i++) {\n \t\tint len = strlen(valid_atom[i].name);\n@@ -709,11 +733,14 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \treturn at;\n }\n \n-static void quote_formatting(struct strbuf *s, const char *str, int quote_style)\n+static void quote_formatting(struct strbuf *s, const char *str, ssize_t len, int quote_style)\n {\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n-\t\tstrbuf_addstr(s, str);\n+\t\tif (len < 0)\n+\t\t\tstrbuf_addstr(s, str);\n+\t\telse\n+\t\t\tstrbuf_add(s, str, len);\n \t\tbreak;\n \tcase QUOTE_SHELL:\n \t\tsq_quote_buf(s, str);\n@@ -740,9 +767,11 @@ static int append_atom(struct atom_value *v, struct ref_formatting_state *state,\n \t * encountered.\n \t */\n \tif (!state->stack->prev)\n-\t\tquote_formatting(&state->stack->output, v->s, state->quote_style);\n-\telse\n+\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n+\telse if (v->s_size < 0)\n \t\tstrbuf_addstr(&state->stack->output, v->s);\n+\telse\n+\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n \treturn 0;\n }\n \n@@ -842,21 +871,23 @@ static int if_atom_handler(struct atom_value *atomv, struct ref_formatting_state\n \treturn 0;\n }\n \n-static int is_empty(const char *s)\n+static int is_empty(struct strbuf *buf)\n {\n-\twhile (*s != '\\0') {\n-\t\tif (!isspace(*s))\n-\t\t\treturn 0;\n-\t\ts++;\n-\t}\n-\treturn 1;\n-}\n+\tconst char *cur = buf->buf;\n+\tconst char *end = buf->buf + buf->len;\n+\n+\twhile (cur != end && (isspace(*cur)))\n+\t\tcur++;\n+\n+\treturn cur == end;\n+ }\n \n static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t\t struct strbuf *err)\n {\n \tstruct ref_formatting_stack *cur = state->stack;\n \tstruct if_then_else *if_then_else = NULL;\n+\tsize_t str_len = 0;\n \n \tif (cur->at_end == if_then_else_handler)\n \t\tif_then_else = (struct if_then_else *)cur->at_end_data;\n@@ -867,18 +898,22 @@ static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_sta\n \tif (if_then_else->else_atom_seen)\n \t\treturn strbuf_addf_ret(err, -1, _(\"format: %%(then) atom used after %%(else)\"));\n \tif_then_else->then_atom_seen = 1;\n+\tif (if_then_else->str)\n+\t\tstr_len = strlen(if_then_else->str);\n \t/*\n \t * If the 'equals' or 'notequals' attribute is used then\n \t * perform the required comparison. If not, only non-empty\n \t * strings satisfy the 'if' condition.\n \t */\n \tif (if_then_else->cmp_status == COMPARE_EQUAL) {\n-\t\tif (!strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len == cur->output.len &&\n+\t\t !memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n \t} else if (if_then_else->cmp_status == COMPARE_UNEQUAL) {\n-\t\tif (strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len != cur->output.len ||\n+\t\t memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n-\t} else if (cur->output.len && !is_empty(cur->output.buf))\n+\t} else if (cur->output.len && !is_empty(&cur->output))\n \t\tif_then_else->condition_satisfied = 1;\n \tstrbuf_reset(&cur->output);\n \treturn 0;\n@@ -924,7 +959,7 @@ static int end_atom_handler(struct atom_value *atomv, struct ref_formatting_stat\n \t * only on the topmost supporting atom.\n \t */\n \tif (!current->prev->prev) {\n-\t\tquote_formatting(&s, current->output.buf, state->quote_style);\n+\t\tquote_formatting(&s, current->output.buf, current->output.len, state->quote_style);\n \t\tstrbuf_swap(¤t->output, &s);\n \t}\n \tstrbuf_release(&s);\n@@ -974,6 +1009,10 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\t\t\tdie(_(\"--format=%.*s cannot be used with\"\n+\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\n@@ -1367,12 +1406,25 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp\n \t\tstruct used_atom *atom = &used_atom[i];\n \t\tconst char *name = atom->name;\n \t\tstruct atom_value *v = &val[i];\n+\t\tenum atom_type atom_type = atom->atom_type;\n \n \t\tif (!!deref != (*name == '*'))\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n \n+\t\tif (atom_type == ATOM_RAW) {\n+\t\t\tunsigned long buf_size = data->size;\n+\n+\t\t\tif (atom->u.raw_data.option == RAW_BARE) {\n+\t\t\t\tv->s = xmemdupz(buf, buf_size);\n+\t\t\t\tv->s_size = buf_size;\n+\t\t\t} else if (atom->u.raw_data.option == RAW_LENGTH) {\n+\t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX, (uintmax_t)buf_size);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif ((data->type != OBJ_TAG &&\n \t\t data->type != OBJ_COMMIT) ||\n \t\t (strcmp(name, \"body\") &&\n@@ -1460,9 +1512,11 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"Eh? Object of type %d?\", obj->type);\n@@ -1766,6 +1820,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n \n+\t\tv->s_size = ATOM_SIZE_UNSPECIFIED;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n@@ -2369,6 +2424,19 @@ static int compare_detached_head(struct ref_array_item *a, struct ref_array_item\n \treturn 0;\n }\n \n+static int memcasecmp(const void *vs1, const void *vs2, size_t n)\n+{\n+\tconst char *s1 = vs1, *s2 = vs2;\n+\tconst char *end = s1 + n;\n+\n+\tfor (; s1 < end; s1++, s2++) {\n+\t\tint diff = tolower(*s1) - tolower(*s2);\n+\t\tif (diff)\n+\t\t\treturn diff;\n+\t}\n+\treturn 0;\n+}\n+\n static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b)\n {\n \tstruct atom_value *va, *vb;\n@@ -2389,10 +2457,29 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n \t} else if (s->sort_flags & REF_SORTING_VERSION) {\n \t\tcmp = versioncmp(va->s, vb->s);\n \t} else if (cmp_type == FIELD_STR) {\n-\t\tint (*cmp_fn)(const char *, const char *);\n-\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n-\t\t\t? strcasecmp : strcmp;\n-\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\tif (va->s_size < 0 && vb->s_size < 0) {\n+\t\t\tint (*cmp_fn)(const char *, const char *);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? strcasecmp : strcmp;\n+\t\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\t} else {\n+\t\t\tsize_t a_size = va->s_size < 0 ?\n+\t\t\t\t\tstrlen(va->s) : va->s_size;\n+\t\t\tsize_t b_size = vb->s_size < 0 ?\n+\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n+\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? memcasecmp : memcmp;\n+\n+\t\t\tcmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n+\t\t\t\t a_size : b_size);\n+\t\t\tif (!cmp) {\n+\t\t\t\tif (a_size > b_size)\n+\t\t\t\t\tcmp = 1;\n+\t\t\t\telse if (a_size < b_size)\n+\t\t\t\t\tcmp = -1;\n+\t\t\t}\n+\t\t}\n \t} else {\n \t\tif (va->value < vb->value)\n \t\t\tcmp = -1;\n@@ -2491,7 +2578,7 @@ int format_ref_array_item(struct ref_array_item *info,\n \t\tappend_literal(cp, sp, &state);\n \t}\n \tif (format->need_color_reset_at_eol) {\n-\t\tstruct atom_value resetv;\n+\t\tstruct atom_value resetv = ATOM_VALUE_INIT;\n \t\tresetv.s = GIT_COLOR_RESET;\n \t\tif (append_atom(&resetv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 9e0214076b4..18554f62d94 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -130,6 +130,8 @@ test_atom head parent:short=10 ''\n test_atom head numparent 0\n test_atom head object ''\n test_atom head type ''\n+test_atom head raw \"$(git cat-file commit refs/heads/main)\n+\"\n test_atom head '*objectname' ''\n test_atom head '*objecttype' ''\n test_atom head author 'A U Thor 1151968724 +0200'\n@@ -221,6 +223,15 @@ test_atom tag contents 'Tagging at 1151968727\n '\n test_atom tag HEAD ' '\n \n+test_expect_success 'basic atom: refs/tags/testtag *raw' '\n+\tgit cat-file commit refs/tags/testtag^{} >expected &&\n+\tgit for-each-ref --format=\"%(*raw)\" refs/tags/testtag >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'Check invalid atoms names are errors' '\n \ttest_must_fail git for-each-ref --format=\"%(INVALID)\" refs/heads\n '\n@@ -686,6 +697,15 @@ test_atom refs/tags/signed-empty contents:body ''\n test_atom refs/tags/signed-empty contents:signature \"$sig\"\n test_atom refs/tags/signed-empty contents \"$sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-empty raw' '\n+\tgit cat-file tag refs/tags/signed-empty >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-empty >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-short subject 'subject line'\n test_atom refs/tags/signed-short subject:sanitize 'subject-line'\n test_atom refs/tags/signed-short contents:subject 'subject line'\n@@ -695,6 +715,15 @@ test_atom refs/tags/signed-short contents:signature \"$sig\"\n test_atom refs/tags/signed-short contents \"subject line\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-short raw' '\n+\tgit cat-file tag refs/tags/signed-short >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-short >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-long subject 'subject line'\n test_atom refs/tags/signed-long subject:sanitize 'subject-line'\n test_atom refs/tags/signed-long contents:subject 'subject line'\n@@ -708,6 +737,15 @@ test_atom refs/tags/signed-long contents \"subject line\n body contents\n $sig\"\n \n+test_expect_success GPG 'basic atom: refs/tags/signed-long raw' '\n+\tgit cat-file tag refs/tags/signed-long >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-long >actual &&\n+\tsanitize_pgp expected.clean &&\n+\techo >>expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'set up refs pointing to tree and blob' '\n \tgit update-ref refs/mytrees/first refs/heads/main^{tree} &&\n \tgit update-ref refs/myblobs/first refs/heads/main:one\n@@ -720,6 +758,16 @@ test_atom refs/mytrees/first contents:body \"\"\n test_atom refs/mytrees/first contents:signature \"\"\n test_atom refs/mytrees/first contents \"\"\n \n+test_expect_success 'basic atom: refs/mytrees/first raw' '\n+\tgit cat-file tree refs/mytrees/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/mytrees/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_atom refs/myblobs/first subject \"\"\n test_atom refs/myblobs/first contents:subject \"\"\n test_atom refs/myblobs/first body \"\"\n@@ -727,6 +775,174 @@ test_atom refs/myblobs/first contents:body \"\"\n test_atom refs/myblobs/first contents:signature \"\"\n test_atom refs/myblobs/first contents \"\"\n \n+test_expect_success 'basic atom: refs/myblobs/first raw' '\n+\tgit cat-file blob refs/myblobs/first >expected &&\n+\techo >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/myblobs/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'set up refs pointing to binary blob' '\n+\tprintf \"a\\0b\\0c\" >blob1 &&\n+\tprintf \"a\\0c\\0b\" >blob2 &&\n+\tprintf \"\\0a\\0b\\0c\" >blob3 &&\n+\tprintf \"abc\" >blob4 &&\n+\tprintf \"\\0 \\0 \\0 \" >blob5 &&\n+\tprintf \"\\0 \\0a\\0 \" >blob6 &&\n+\tprintf \" \" >blob7 &&\n+\t>blob8 &&\n+\tobj=$(git hash-object -w blob1) &&\n+\tgit update-ref refs/myblobs/blob1 \"$obj\" &&\n+\tobj=$(git hash-object -w blob2) &&\n+\tgit update-ref refs/myblobs/blob2 \"$obj\" &&\n+\tobj=$(git hash-object -w blob3) &&\n+\tgit update-ref refs/myblobs/blob3 \"$obj\" &&\n+\tobj=$(git hash-object -w blob4) &&\n+\tgit update-ref refs/myblobs/blob4 \"$obj\" &&\n+\tobj=$(git hash-object -w blob5) &&\n+\tgit update-ref refs/myblobs/blob5 \"$obj\" &&\n+\tobj=$(git hash-object -w blob6) &&\n+\tgit update-ref refs/myblobs/blob6 \"$obj\" &&\n+\tobj=$(git hash-object -w blob7) &&\n+\tgit update-ref refs/myblobs/blob7 \"$obj\" &&\n+\tobj=$(git hash-object -w blob8) &&\n+\tgit update-ref refs/myblobs/blob8 \"$obj\"\n+'\n+\n+test_expect_success 'Verify sorts with raw' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob7\n+\trefs/mytrees/first\n+\trefs/myblobs/first\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob4\n+\trefs/heads/main\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'Verify sorts with raw:size' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\trefs/myblobs/blob7\n+\trefs/heads/main\n+\trefs/myblobs/blob4\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/mytrees/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw:size \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:equals)' '\n+\tcat >expected <<-EOF &&\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\trefs/myblobs/blob4\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:equals=abc)%(raw)%(then)%(refname)%(else)not equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:notequals)' '\n+\tcat >expected <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/heads/main\n+\trefs/heads/newtag\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\tequals\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob7\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:notequals=abc)%(raw)%(then)%(refname)%(else)equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'empty raw refs with %(if)' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob1 not empty\n+\trefs/myblobs/blob2 not empty\n+\trefs/myblobs/blob3 not empty\n+\trefs/myblobs/blob4 not empty\n+\trefs/myblobs/blob5 not empty\n+\trefs/myblobs/blob6 not empty\n+\trefs/myblobs/blob7 empty\n+\trefs/myblobs/blob8 empty\n+\trefs/myblobs/first not empty\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname) %(if)%(raw)%(then)not empty%(else)empty%(end)\" \\\n+\t\trefs/myblobs/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success '%(raw) with --python must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --python\n+'\n+\n+test_expect_success '%(raw) with --tcl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n+'\n+\n+test_expect_success '%(raw) with --perl must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+'\n+\n+test_expect_success '%(raw) with --shell must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --shell\n+'\n+\n+test_expect_success '%(raw) with --shell and --sort=raw must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --sort=raw --shell\n+'\n+\n+test_expect_success '%(raw:size) with --shell' '\n+\tgit for-each-ref --format=\"%(raw:size)\" | while read line\n+\tdo\n+\t\techo \"'\\''$line'\\''\" >>expect\n+\tdone &&\n+\tgit for-each-ref --format=\"%(raw:size)\" --shell >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for-each-ref --format compare with cat-file --batch' '\n+\tgit rev-parse refs/mytrees/first | git cat-file --batch >expected &&\n+\tgit for-each-ref --format=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'set up multiple-sort tags' '\n \tfor when in 100000 200000\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"431160","messageId":"d14fae5208013836d8015cf1be19314c61a65d1e.1627270010.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v4.git.1627270010.gitgitgadget@gmail.com","subject":"[PATCH v4 3/5] [GSOC] ref-filter: --format=%(raw) support --perl","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-26T03:26:48Z","receivedAt":"2021-07-26T03:27:02Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nBecause the perl language can handle binary data correctly,\nadd the function perl_quote_buf_with_len(), which can specify\nthe length of the data and prevent the data from being truncated\nat '\\0' to help `--format=\"%(raw)\"` support `--perl`.\n\nReviewed-by: Jacob Keller \nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 4 ++--\n quote.c | 17 +++++++++++++++++\n quote.h | 1 +\n ref-filter.c | 15 +++++++++++----\n t/t6300-for-each-ref.sh | 19 +++++++++++++++++--\n 5 files changed, 48 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex cbb6f87d13f..6da899c6296 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -241,8 +241,8 @@ raw:size::\n \tThe raw data size of the object.\n \n Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n-`--perl` because such language may not support arbitrary binary data in their\n-string variable type.\n+because such language may not support arbitrary binary data in their string\n+variable type.\n \n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\ndiff --git a/quote.c b/quote.c\nindex 8a3a5e39eb1..26719d21d1e 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src)\n \tstrbuf_addch(sb, sq);\n }\n \n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len)\n+{\n+\tconst char sq = '\\'';\n+\tconst char bq = '\\\\';\n+\tconst char *c = src;\n+\tconst char *end = src + len;\n+\n+\tstrbuf_addch(sb, sq);\n+\twhile (c != end) {\n+\t\tif (*c == sq || *c == bq)\n+\t\t\tstrbuf_addch(sb, bq);\n+\t\tstrbuf_addch(sb, *c);\n+\t\tc++;\n+\t}\n+\tstrbuf_addch(sb, sq);\n+}\n+\n void python_quote_buf(struct strbuf *sb, const char *src)\n {\n \tconst char sq = '\\'';\ndiff --git a/quote.h b/quote.h\nindex 768cc6338e2..0fe69e264b0 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -94,6 +94,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne\n \n /* quoting as a string literal for other languages */\n void perl_quote_buf(struct strbuf *sb, const char *src);\n+void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len);\n void python_quote_buf(struct strbuf *sb, const char *src);\n void tcl_quote_buf(struct strbuf *sb, const char *src);\n void basic_regex_quote_buf(struct strbuf *sb, const char *src);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 0d5eb91ed54..ce6c2115016 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -746,7 +746,10 @@ static void quote_formatting(struct strbuf *s, const char *str, ssize_t len, int\n \t\tsq_quote_buf(s, str);\n \t\tbreak;\n \tcase QUOTE_PERL:\n-\t\tperl_quote_buf(s, str);\n+\t\tif (len < 0)\n+\t\t\tperl_quote_buf(s, str);\n+\t\telse\n+\t\t\tperl_quote_buf_with_len(s, str, len);\n \t\tbreak;\n \tcase QUOTE_PYTHON:\n \t\tpython_quote_buf(s, str);\n@@ -1009,10 +1012,14 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n-\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n-\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\n+\t\tif ((format->quote_style == QUOTE_PYTHON ||\n+\t\t format->quote_style == QUOTE_SHELL ||\n+\t\t format->quote_style == QUOTE_TCL) &&\n+\t\t used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n \t\t\tdie(_(\"--format=%.*s cannot be used with\"\n-\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n+\t\t\t \"--python, --shell, --tcl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 18554f62d94..3d15d0a5360 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' '\n \ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n '\n \n-test_expect_success '%(raw) with --perl must fail' '\n-\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+test_expect_success '%(raw) with --perl' '\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob1 --perl | perl >actual &&\n+\tcmp blob1 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob3 --perl | perl >actual &&\n+\tcmp blob3 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/blob8 --perl | perl >actual &&\n+\tcmp blob8 actual &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/myblobs/first --perl | perl >actual &&\n+\tcmp one actual &&\n+\tgit cat-file tree refs/mytrees/first > expected &&\n+\tgit for-each-ref --format=\"\\$name= %(raw);\n+print \\\"\\$name\\\"\" refs/mytrees/first --perl | perl >actual &&\n+\tcmp expected actual\n '\n \n test_expect_success '%(raw) with --shell must fail' '\n-- \ngitgitgadget\n\n"},{"id":"431161","messageId":"0aa398e8a8f7e87d2af05a12f6a008486efae92d.1627270010.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v4.git.1627270010.gitgitgadget@gmail.com","subject":"[PATCH v4 4/5] [GSOC] ref-filter: use non-const ref_format in *_atom_parser()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-26T03:26:49Z","receivedAt":"2021-07-26T03:27:07Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse non-const ref_format in *_atom_parser(), which can help us\nmodify the members of ref_format in *_atom_parser().\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n builtin/tag.c | 2 +-\n ref-filter.c | 44 ++++++++++++++++++++++----------------------\n ref-filter.h | 4 ++--\n 3 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 82fcfc09824..452558ec957 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref,\n \t\t const struct object_id *oid, void *cb_data)\n {\n \tint flags;\n-\tconst struct ref_format *format = cb_data;\n+\tstruct ref_format *format = cb_data;\n \tflags = GPG_VERIFY_VERBOSE;\n \n \tif (format->format)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex ce6c2115016..f944160c5ce 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -226,7 +226,7 @@ static int strbuf_addf_ret(struct strbuf *sb, int ret, const char *fmt, ...)\n \treturn ret;\n }\n \n-static int color_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int color_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *color_value, struct strbuf *err)\n {\n \tif (!color_value)\n@@ -264,7 +264,7 @@ static int refname_atom_parser_internal(struct refname_atom *atom, const char *a\n \treturn 0;\n }\n \n-static int remote_ref_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int remote_ref_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct string_list params = STRING_LIST_INIT_DUP;\n@@ -311,7 +311,7 @@ static int remote_ref_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objecttype_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -323,7 +323,7 @@ static int objecttype_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objectsize_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -343,7 +343,7 @@ static int objectsize_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int deltabase_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int deltabase_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -355,7 +355,7 @@ static int deltabase_atom_parser(const struct ref_format *format, struct used_at\n \treturn 0;\n }\n \n-static int body_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int body_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -364,7 +364,7 @@ static int body_atom_parser(const struct ref_format *format, struct used_atom *a\n \treturn 0;\n }\n \n-static int subject_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int subject_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -376,7 +376,7 @@ static int subject_atom_parser(const struct ref_format *format, struct used_atom\n \treturn 0;\n }\n \n-static int trailers_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int trailers_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tatom->u.contents.trailer_opts.no_divider = 1;\n@@ -402,7 +402,7 @@ static int trailers_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int contents_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int contents_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -430,7 +430,7 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int raw_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -442,7 +442,7 @@ static int raw_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int oid_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -461,7 +461,7 @@ static int oid_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int person_email_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int person_email_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -475,7 +475,7 @@ static int person_email_atom_parser(const struct ref_format *format, struct used\n \treturn 0;\n }\n \n-static int refname_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int refname_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \treturn refname_atom_parser_internal(&atom->u.refname, arg, atom->name, err);\n@@ -492,7 +492,7 @@ static align_type parse_align_position(const char *s)\n \treturn -1;\n }\n \n-static int align_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int align_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct align *align = &atom->u.align;\n@@ -544,7 +544,7 @@ static int align_atom_parser(const struct ref_format *format, struct used_atom *\n \treturn 0;\n }\n \n-static int if_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -559,7 +559,7 @@ static int if_atom_parser(const struct ref_format *format, struct used_atom *ato\n \treturn 0;\n }\n \n-static int head_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n \tatom->u.head = resolve_refdup(\"HEAD\", RESOLVE_REF_READING, NULL, NULL);\n@@ -570,7 +570,7 @@ static struct {\n \tconst char *name;\n \tinfo_source source;\n \tcmp_type cmp_type;\n-\tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n+\tint (*parser)(struct ref_format *format, struct used_atom *atom,\n \t\t const char *arg, struct strbuf *err);\n } valid_atom[] = {\n \t[ATOM_REFNAME] = { \"refname\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -653,7 +653,7 @@ struct atom_value {\n /*\n * Used to parse format string and sort specifiers\n */\n-static int parse_ref_filter_atom(const struct ref_format *format,\n+static int parse_ref_filter_atom(struct ref_format *format,\n \t\t\t\t const char *atom, const char *ep,\n \t\t\t\t struct strbuf *err)\n {\n@@ -2556,9 +2556,9 @@ static void append_literal(const char *cp, const char *ep, struct ref_formatting\n }\n \n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n-\t\t\t struct strbuf *final_buf,\n-\t\t\t struct strbuf *error_buf)\n+\t\t\t struct ref_format *format,\n+\t\t\t struct strbuf *final_buf,\n+\t\t\t struct strbuf *error_buf)\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n@@ -2602,7 +2602,7 @@ int format_ref_array_item(struct ref_array_item *info,\n }\n \n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format)\n+\t\t struct ref_format *format)\n {\n \tstruct ref_array_item *ref_item;\n \tstruct strbuf output = STRBUF_INIT;\ndiff --git a/ref-filter.h b/ref-filter.h\nindex baf72a71896..74fb423fc89 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);\n void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting, unsigned int mask, int on);\n /* Based on the given format and quote_style, fill the strbuf */\n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n+\t\t\t struct ref_format *format,\n \t\t\t struct strbuf *final_buf,\n \t\t\t struct strbuf *error_buf);\n /* Parse a single sort specifier and add it to the list */\n@@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void);\n * name must be a fully qualified refname.\n */\n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format);\n+\t\t struct ref_format *format);\n \n /*\n * Push a single ref onto the array; this can be used to construct your own\n-- \ngitgitgadget\n\n"},{"id":"431162","messageId":"207cc5129649e767036d8467ea7c884c3f664cc7.1627270010.git.gitgitgadget@gmail.com","threadId":"56142","inReplyTo":"pull.1000.v4.git.1627270010.gitgitgadget@gmail.com","subject":"[PATCH v4 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-26T03:26:50Z","receivedAt":"2021-07-26T03:27:10Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\n%(rest) is a atom used for cat-file batch mode, which can split\nthe input lines at the first whitespace boundary, all characters\nbefore that whitespace are considered to be the object name;\ncharacters after that first run of whitespace (i.e., the \"rest\"\nof the line) are output in place of the %(rest) atom.\n\nIn order to let \"cat-file --batch=%(rest)\" use the ref-filter\ninterface, add %(rest) atom for ref-filter.\n\nIntroduce the reject_atom() to reject the atom %(rest) for\n\"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n\nReviewed-by: Jacob Keller \nSuggected-by: Jacob Keller \nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 25 +++++++++++++++++++++++++\n ref-filter.h | 5 ++++-\n t/t3203-branch-output.sh | 4 ++++\n t/t6300-for-each-ref.sh | 4 ++++\n t/t7004-tag.sh | 4 ++++\n t/t7030-verify-tag.sh | 4 ++++\n 6 files changed, 45 insertions(+), 1 deletion(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex f944160c5ce..2cacee9e64a 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -157,6 +157,7 @@ enum atom_type {\n \tATOM_IF,\n \tATOM_THEN,\n \tATOM_ELSE,\n+\tATOM_REST,\n };\n \n /*\n@@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"%%(rest) does not take arguments\"));\n+\tformat->use_rest = 1;\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n@@ -615,6 +625,7 @@ static struct {\n \t[ATOM_IF] = { \"if\", SOURCE_NONE, FIELD_STR, if_atom_parser },\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n+\t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -992,6 +1003,11 @@ static const char *find_next(const char *cp)\n \treturn NULL;\n }\n \n+static int reject_atom(enum atom_type atom_type)\n+{\n+\treturn atom_type == ATOM_REST;\n+}\n+\n /*\n * Make sure the format string is well formed, and parse out\n * the used atoms.\n@@ -1012,6 +1028,8 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (reject_atom(used_atom[at].atom_type))\n+\t\t\tdie(_(\"this command reject atom %%(%.*s)\"), (int)(ep - sp - 2), sp + 2);\n \n \t\tif ((format->quote_style == QUOTE_PYTHON ||\n \t\t format->quote_style == QUOTE_SHELL ||\n@@ -1931,6 +1949,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_REST) {\n+\t\t\tif (ref->rest)\n+\t\t\t\tv->s = xstrdup(ref->rest);\n+\t\t\telse\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2148,6 +2172,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n+\tref->rest = NULL;\n \n \treturn ref;\n }\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 74fb423fc89..c15dee8d6b9 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -38,6 +38,7 @@ struct ref_sorting {\n \n struct ref_array_item {\n \tstruct object_id objectname;\n+\tconst char *rest;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n@@ -76,14 +77,16 @@ struct ref_format {\n \t * verify_ref_format() afterwards to finalize.\n \t */\n \tconst char *format;\n+\tconst char *rest;\n \tint quote_style;\n+\tint use_rest;\n \tint use_color;\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n };\n \n-#define REF_FORMAT_INIT { NULL, 0, -1 }\n+#define REF_FORMAT_INIT { .use_color = -1 }\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\ndiff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh\nindex 5325b9f67a0..6e94c6db7b5 100755\n--- a/t/t3203-branch-output.sh\n+++ b/t/t3203-branch-output.sh\n@@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git branch with --format=%(rest) must fail' '\n+\ttest_must_fail git branch --format=\"%(rest)\" >actual\n+'\n+\n test_expect_success 'worktree colors correct' '\n \tcat >expect <<-EOF &&\n \t* (HEAD detached from fromtag)\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 3d15d0a5360..0d2e062f791 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' '\n \ttest_cmp expect actual.clean\n '\n \n+test_expect_success 'basic atom: rest must fail' '\n+\ttest_must_fail git for-each-ref --format=\"%(rest)\" refs/heads/main\n+'\n+\n test_expect_success 'trailer parsing not fooled by --- line' '\n \tgit commit --allow-empty -F - <<-\\EOF &&\n \tthis is the subject\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 2f72c5c6883..082be85dffc 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git tag -l with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git tag -l --format=\"%(rest)\" \"v1*\"\n+'\n+\n test_expect_success \"set up color tests\" '\n \techo \"v1.0\" >expect.color &&\n \techo \"v1.0\" >expect.bare &&\ndiff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh\nindex 3cefde9602b..10faa645157 100755\n--- a/t/t7030-verify-tag.sh\n+++ b/t/t7030-verify-tag.sh\n@@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success GPG 'verifying tag with --format=\"%(rest)\" must fail' '\n+\ttest_must_fail git verify-tag --format=\"%(rest)\" \"fourth-signed\"\n+'\n+\n test_expect_success GPG 'verifying a forged tag with --format should fail silently' '\n \ttest_must_fail git verify-tag --format=\"tagname : %(tag)\" $(cat forged1.tag) >actual-forged &&\n \ttest_must_be_empty actual-forged\n-- \ngitgitgadget\n"},{"id":"431202","messageId":"xmqqtukhj7x9.fsf@gitster.g","threadId":"56142","inReplyTo":"CA+P7+xpfWA_e3rn2vfWBPixFJop-zomj9_N99n9ufe1MxV-1KA@mail.gmail.com","subject":"Re: [PATCH v3 5/5] [GSOC] ref-filter: add %(rest) atom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-26T17:34:26Z","receivedAt":"2021-07-26T17:34:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jacob Keller writes:\n\n> On Sat, Jul 24, 2021 at 7:14 AM ZheNing Hu via GitGitGadget\n> wrote:\n>>\n>> From: ZheNing Hu \n>>\n>> %(rest) is a atom used for cat-file batch mode, which can split\n>> the input lines at the first whitespace boundary, all characters\n>> before that whitespace are considered to be the object name;\n>> characters after that first run of whitespace (i.e., the \"rest\"\n>> of the line) are output in place of the %(rest) atom.\n>>\n>> In order to let \"cat-file --batch=%(rest)\" use the ref-filter\n>> interface, add %(rest) atom for ref-filter.\n>>\n>> Introduce the reject_atom() to reject the atom %(rest) for\n>> \"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n>>\n>> Suggected-by: Jacob Keller \n>> Mentored-by: Christian Couder \n>> Mentored-by: Hariom Verma \n>> Signed-off-by: ZheNing Hu \n>> ---\n>\n> Thanks for the improved commit message here. That helps clarify what\n> this atom does and aids in understanding the implementation.\n>\n> Reviewed-by: Jacob Keller \n\nThanks, both.\n"},{"id":"431223","messageId":"xmqqv94wj38w.fsf@gitster.g","threadId":"56142","inReplyTo":"0e294ff1626ef2b7992e84e8d960f5058bd39bdc.1627270010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-26T19:15:27Z","receivedAt":"2021-07-26T19:15:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"ZheNing Hu via GitGitGadget\" writes:\n\n> From: ZheNing Hu \n>\n> Only tag and commit objects use `grab_sub_body_contents()` to grab\n> object contents in the current codebase. We want to teach the\n> function to also handle blobs and trees to get their raw data,\n> without parsing a blob (whose contents looks like a commit or a tag)\n> incorrectly as a commit or a tag. So it's needed to pass a\n> `struct expand_data *data` instread of only `void *buf` to both\n> `grab_sub_body_contents()` and `grab_values()` to be able to check\n> the object type.\n>\n> Skip the block of code that is specific to handling commits and tags\n> early when the given object is of a wrong type to help later\n> addition to handle other types of objects in this function.\n>\n> Reviewed-by: Jacob Keller \n> Mentored-by: Christian Couder \n> Mentored-by: Hariom Verma \n> Helped-by: Junio C Hamano \n> Signed-off-by: ZheNing Hu \n\nIt is not a huge deal, but are these trailers given in a sensible\norder? The order of event happened were that help and mentoring\nwere given, your finished version of the patch came after that to be\nsigned off by you, and then a reviewer gave you a reviewed-by, no?\n"},{"id":"431281","messageId":"CAOLTT8QHKjubYowdCERs3ujcCsB7HNAK6+703rbSHRBKcR_nsg@mail.gmail.com","threadId":"56142","inReplyTo":"xmqqv94wj38w.fsf@gitster.g","subject":"Re: [PATCH v4 1/5] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-27T01:41:05Z","receivedAt":"2021-07-27T01:40:36Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年7月27日周二 上午3:15写道：\n>\n> \"ZheNing Hu via GitGitGadget\" writes:\n>\n> > From: ZheNing Hu \n> >\n> > Only tag and commit objects use `grab_sub_body_contents()` to grab\n> > object contents in the current codebase. We want to teach the\n> > function to also handle blobs and trees to get their raw data,\n> > without parsing a blob (whose contents looks like a commit or a tag)\n> > incorrectly as a commit or a tag. So it's needed to pass a\n> > `struct expand_data *data` instread of only `void *buf` to both\n> > `grab_sub_body_contents()` and `grab_values()` to be able to check\n> > the object type.\n> >\n> > Skip the block of code that is specific to handling commits and tags\n> > early when the given object is of a wrong type to help later\n> > addition to handle other types of objects in this function.\n> >\n> > Reviewed-by: Jacob Keller \n> > Mentored-by: Christian Couder \n> > Mentored-by: Hariom Verma \n> > Helped-by: Junio C Hamano \n> > Signed-off-by: ZheNing Hu \n>\n> It is not a huge deal, but are these trailers given in a sensible\n> order? The order of event happened were that help and mentoring\n> were given, your finished version of the patch came after that to be\n> signed off by you, and then a reviewer gave you a reviewed-by, no?\n\nIndeed so.\n\nThanks.\n--\nZheNing Hu\n"}]}