{"thread":{"id":"55843","subject":"[PATCH 2/6] [GSOC] ref-filter: add %(raw) atom","startedAt":"2021-06-05T09:13:42Z","lastAt":"2021-06-09T12:47:40Z","messageCount":27,"participants":["ZheNing Hu via GitGitGadget","Bagas Sanjaya","ZheNing Hu","Hariom verma","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"426475","messageId":"0efed9435b59098f3ad928acd46c3c7e9f13677d.1622884415.git.gitgitgadget@gmail.com","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"[PATCH 2/6] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-05T09:13:30Z","receivedAt":"2021-06-05T09:13:42Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nAdd new formatting option `%(raw)`, which will print the raw\nobject data without any changes. It will help further to migrate\nall cat-file formatting logic from cat-file to ref-filter.\n\nThe raw data of blob, tree objects may contain '\\0', but most of\nthe logic in `ref-filter` depands on the output of the atom being\ntext (specifically, no embedded NULs in it).\n\nE.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()`\nadd the data to the buffer. The raw data of a tree object is\n`100644 one\\0...`, only the `100644 one` will be added to the buffer,\nwhich is incorrect.\n\nTherefore, add a new member in `struct atom_value`: `s_size`, which\ncan record raw object size, it can help us add raw object data to\nthe buffer or compare two buffers which contain raw object data.\n\nBeyond, `--format=%(raw)` cannot be used with `--python`, `--shell`,\n`--tcl`, `--perl` because if our binary raw data is passed to a variable\nin the host language, the host language may not support arbitrary binary\ndata in the variables of its string type.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Felipe Contreras \nHelped-by: Phillip Wood \nHelped-by: Junio C Hamano \nBased-on-patch-by: Olga Telezhnaya \nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 9 ++\n ref-filter.c | 140 +++++++++++++++----\n t/t6300-for-each-ref.sh | 207 +++++++++++++++++++++++++++++\n 3 files changed, 328 insertions(+), 28 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 2ae2478de706..8f8d8cd1e04f 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`,\n without angle brackets, and `:localpart` to get the part before the `@` symbol\n out of the trimmed email.\n \n+The raw data in a object is `raw`.\n+\n+raw:size::\n+\tThe raw data size of the object.\n+\n+Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n+`--perl` because the host language may not support arbitrary binary data in the\n+variables of its string type.\n+\n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5cee6512fbaf..46aec291de62 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -144,6 +144,7 @@ enum atom_type {\n \tATOM_BODY,\n \tATOM_TRAILERS,\n \tATOM_CONTENTS,\n+\tATOM_RAW,\n \tATOM_UPSTREAM,\n \tATOM_PUSH,\n \tATOM_SYMREF,\n@@ -189,6 +190,9 @@ static struct used_atom {\n \t\t\tstruct process_trailer_options trailer_opts;\n \t\t\tunsigned int nlines;\n \t\t} contents;\n+\t\tstruct {\n+\t\t\tenum { RAW_BARE, RAW_LENGTH } option;\n+\t\t} raw_data;\n \t\tstruct {\n \t\t\tcmp_status cmp_status;\n \t\t\tconst char *str;\n@@ -426,6 +430,18 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n+static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\tconst char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\tatom->u.raw_data.option = RAW_BARE;\n+\telse if (!strcmp(arg, \"size\"))\n+\t\tatom->u.raw_data.option = RAW_LENGTH;\n+\telse\n+\t\treturn strbuf_addf_ret(err, -1, _(\"unrecognized %%(raw) argument: %s\"), arg);\n+\treturn 0;\n+}\n+\n static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -586,6 +602,7 @@ static struct {\n \t[ATOM_BODY] = { \"body\", SOURCE_OBJ, FIELD_STR, body_atom_parser },\n \t[ATOM_TRAILERS] = { \"trailers\", SOURCE_OBJ, FIELD_STR, trailers_atom_parser },\n \t[ATOM_CONTENTS] = { \"contents\", SOURCE_OBJ, FIELD_STR, contents_atom_parser },\n+\t[ATOM_RAW] = { \"raw\", SOURCE_OBJ, FIELD_STR, raw_atom_parser },\n \t[ATOM_UPSTREAM] = { \"upstream\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_PUSH] = { \"push\", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser },\n \t[ATOM_SYMREF] = { \"symref\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -620,12 +637,15 @@ struct ref_formatting_state {\n \n struct atom_value {\n \tconst char *s;\n+\tsize_t s_size;\n \tint (*handler)(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t struct strbuf *err);\n \tuintmax_t value; /* used for sorting when not FIELD_STR */\n \tstruct used_atom *atom;\n };\n \n+#define ATOM_VALUE_S_SIZE_INIT (-1)\n+\n /*\n * Used to parse format string and sort specifiers\n */\n@@ -644,13 +664,6 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \t\treturn strbuf_addf_ret(err, -1, _(\"malformed field name: %.*s\"),\n \t\t\t\t (int)(ep-atom), atom);\n \n-\t/* Do we have the atom already used elsewhere? */\n-\tfor (i = 0; i < used_atom_cnt; i++) {\n-\t\tint len = strlen(used_atom[i].name);\n-\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n-\t\t\treturn i;\n-\t}\n-\n \t/*\n \t * If the atom name has a colon, strip it and everything after\n \t * it off - it specifies the format for this entry, and\n@@ -660,6 +673,13 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \targ = memchr(sp, ':', ep - sp);\n \tatom_len = (arg ? arg : ep) - sp;\n \n+\t/* Do we have the atom already used elsewhere? */\n+\tfor (i = 0; i < used_atom_cnt; i++) {\n+\t\tint len = strlen(used_atom[i].name);\n+\t\tif (len == ep - atom && !memcmp(used_atom[i].name, atom, len))\n+\t\t\treturn i;\n+\t}\n+\n \t/* Is the atom a valid one? */\n \tfor (i = 0; i < ARRAY_SIZE(valid_atom); i++) {\n \t\tint len = strlen(valid_atom[i].name);\n@@ -709,11 +729,14 @@ static int parse_ref_filter_atom(const struct ref_format *format,\n \treturn at;\n }\n \n-static void quote_formatting(struct strbuf *s, const char *str, int quote_style)\n+static void quote_formatting(struct strbuf *s, const char *str, size_t len, int quote_style)\n {\n \tswitch (quote_style) {\n \tcase QUOTE_NONE:\n-\t\tstrbuf_addstr(s, str);\n+\t\tif (len != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tstrbuf_add(s, str, len);\n+\t\telse\n+\t\t\tstrbuf_addstr(s, str);\n \t\tbreak;\n \tcase QUOTE_SHELL:\n \t\tsq_quote_buf(s, str);\n@@ -740,9 +763,12 @@ static int append_atom(struct atom_value *v, struct ref_formatting_state *state,\n \t * encountered.\n \t */\n \tif (!state->stack->prev)\n-\t\tquote_formatting(&state->stack->output, v->s, state->quote_style);\n+\t\tquote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style);\n \telse\n-\t\tstrbuf_addstr(&state->stack->output, v->s);\n+\t\tif (v->s_size != ATOM_VALUE_S_SIZE_INIT)\n+\t\t\tstrbuf_add(&state->stack->output, v->s, v->s_size);\n+\t\telse\n+\t\t\tstrbuf_addstr(&state->stack->output, v->s);\n \treturn 0;\n }\n \n@@ -842,21 +868,23 @@ static int if_atom_handler(struct atom_value *atomv, struct ref_formatting_state\n \treturn 0;\n }\n \n-static int is_empty(const char *s)\n+static int is_empty(struct strbuf *buf)\n {\n-\twhile (*s != '\\0') {\n-\t\tif (!isspace(*s))\n-\t\t\treturn 0;\n-\t\ts++;\n-\t}\n-\treturn 1;\n-}\n+\tconst char *cur = buf->buf;\n+\tconst char *end = buf->buf + buf->len;\n+\n+\twhile (cur != end && (isspace(*cur)))\n+\t\tcur++;\n+\n+\treturn cur == end;\n+ }\n \n static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_state *state,\n \t\t\t struct strbuf *err)\n {\n \tstruct ref_formatting_stack *cur = state->stack;\n \tstruct if_then_else *if_then_else = NULL;\n+\tsize_t str_len = 0;\n \n \tif (cur->at_end == if_then_else_handler)\n \t\tif_then_else = (struct if_then_else *)cur->at_end_data;\n@@ -867,18 +895,22 @@ static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_sta\n \tif (if_then_else->else_atom_seen)\n \t\treturn strbuf_addf_ret(err, -1, _(\"format: %%(then) atom used after %%(else)\"));\n \tif_then_else->then_atom_seen = 1;\n+\tif (if_then_else->str)\n+\t\tstr_len = strlen(if_then_else->str);\n \t/*\n \t * If the 'equals' or 'notequals' attribute is used then\n \t * perform the required comparison. If not, only non-empty\n \t * strings satisfy the 'if' condition.\n \t */\n \tif (if_then_else->cmp_status == COMPARE_EQUAL) {\n-\t\tif (!strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len == cur->output.len &&\n+\t\t !memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n \t} else if (if_then_else->cmp_status == COMPARE_UNEQUAL) {\n-\t\tif (strcmp(if_then_else->str, cur->output.buf))\n+\t\tif (str_len != cur->output.len ||\n+\t\t memcmp(if_then_else->str, cur->output.buf, cur->output.len))\n \t\t\tif_then_else->condition_satisfied = 1;\n-\t} else if (cur->output.len && !is_empty(cur->output.buf))\n+\t} else if (cur->output.len && !is_empty(&cur->output))\n \t\tif_then_else->condition_satisfied = 1;\n \tstrbuf_reset(&cur->output);\n \treturn 0;\n@@ -924,7 +956,7 @@ static int end_atom_handler(struct atom_value *atomv, struct ref_formatting_stat\n \t * only on the topmost supporting atom.\n \t */\n \tif (!current->prev->prev) {\n-\t\tquote_formatting(&s, current->output.buf, state->quote_style);\n+\t\tquote_formatting(&s, current->output.buf, current->output.len, state->quote_style);\n \t\tstrbuf_swap(¤t->output, &s);\n \t}\n \tstrbuf_release(&s);\n@@ -974,6 +1006,10 @@ int verify_ref_format(struct ref_format *format)\n \t\tat = parse_ref_filter_atom(format, sp + 2, ep, &err);\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n+\t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n+\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\t\t\tdie(_(\"--format=%.*s cannot be used with\"\n+\t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n \n \t\tif (skip_prefix(used_atom[at].name, \"color:\", &color))\n@@ -1362,17 +1398,29 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n \tvoid *buf = data->content;\n+\tunsigned long buf_size = data->size;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n \t\tconst char *name = atom->name;\n \t\tstruct atom_value *v = &val[i];\n+\t\tenum atom_type atom_type = atom->atom_type;\n \n \t\tif (!!deref != (*name == '*'))\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n \n+\t\tif (atom_type == ATOM_RAW) {\n+\t\t\tif (atom->u.raw_data.option == RAW_BARE) {\n+\t\t\t\tv->s = xmemdupz(buf, buf_size);\n+\t\t\t\tv->s_size = buf_size;\n+\t\t\t} else if (atom->u.raw_data.option == RAW_LENGTH) {\n+\t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX, (uintmax_t)buf_size);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif ((data->type != OBJ_TAG &&\n \t\t data->type != OBJ_COMMIT) ||\n \t\t (strcmp(name, \"body\") &&\n@@ -1460,9 +1508,11 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"Eh? Object of type %d?\", obj->type);\n@@ -1765,7 +1815,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\tint deref = 0;\n \t\tconst char *refname;\n \t\tstruct branch *branch = NULL;\n-\n+\t\tv->s_size = ATOM_VALUE_S_SIZE_INIT;\n \t\tv->handler = append_atom;\n \t\tv->atom = atom;\n \n@@ -2369,6 +2419,19 @@ static int compare_detached_head(struct ref_array_item *a, struct ref_array_item\n \treturn 0;\n }\n \n+static int memcasecmp(const void *vs1, const void *vs2, size_t n)\n+{\n+\tconst char *s1 = vs1, *s2 = vs2;\n+\tconst char *end = s1 + n;\n+\n+\tfor (; s1 < end; s1++, s2++) {\n+\t\tint diff = tolower(*s1) - tolower(*s2);\n+\t\tif (diff)\n+\t\t\treturn diff;\n+\t}\n+\treturn 0;\n+}\n+\n static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b)\n {\n \tstruct atom_value *va, *vb;\n@@ -2389,10 +2452,30 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n \t} else if (s->sort_flags & REF_SORTING_VERSION) {\n \t\tcmp = versioncmp(va->s, vb->s);\n \t} else if (cmp_type == FIELD_STR) {\n-\t\tint (*cmp_fn)(const char *, const char *);\n-\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n-\t\t\t? strcasecmp : strcmp;\n-\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\tif (va->s_size == ATOM_VALUE_S_SIZE_INIT &&\n+\t\t vb->s_size == ATOM_VALUE_S_SIZE_INIT) {\n+\t\t\tint (*cmp_fn)(const char *, const char *);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? strcasecmp : strcmp;\n+\t\t\tcmp = cmp_fn(va->s, vb->s);\n+\t\t} else {\n+\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n+\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\t\t? memcasecmp : memcmp;\n+\t\t\tsize_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT ?\n+\t\t\t\t\tstrlen(va->s) : va->s_size;\n+\t\t\tsize_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT ?\n+\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n+\n+\t\t\tcmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n+\t\t\t\t a_size : b_size);\n+\t\t\tif (!cmp) {\n+\t\t\t\tif (a_size > b_size)\n+\t\t\t\t\tcmp = 1;\n+\t\t\t\telse if (a_size < b_size)\n+\t\t\t\t\tcmp = -1;\n+\t\t\t}\n+\t\t}\n \t} else {\n \t\tif (va->value < vb->value)\n \t\t\tcmp = -1;\n@@ -2492,6 +2575,7 @@ int format_ref_array_item(struct ref_array_item *info,\n \t}\n \tif (format->need_color_reset_at_eol) {\n \t\tstruct atom_value resetv;\n+\t\tresetv.s_size = ATOM_VALUE_S_SIZE_INIT;\n \t\tresetv.s = GIT_COLOR_RESET;\n \t\tif (append_atom(&resetv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 9e0214076b4d..5f66d933ace0 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -130,6 +130,8 @@ test_atom head parent:short=10 ''\n test_atom head numparent 0\n test_atom head object ''\n test_atom head type ''\n+test_atom head raw \"$(git cat-file commit refs/heads/main)\n+\"\n test_atom head '*objectname' ''\n test_atom head '*objecttype' ''\n test_atom head author 'A U Thor 1151968724 +0200'\n@@ -221,6 +223,15 @@ test_atom tag contents 'Tagging at 1151968727\n '\n test_atom tag HEAD ' '\n \n+test_expect_success 'basic atom: refs/tags/testtag *raw' '\n+\tgit cat-file commit refs/tags/testtag^{} >expected &&\n+\tgit for-each-ref --format=\"%(*raw)\" refs/tags/testtag >actual &&\n+\tsanitize_pgp expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\techo \"\" >>expected.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'Check invalid atoms names are errors' '\n \ttest_must_fail git for-each-ref --format=\"%(INVALID)\" refs/heads\n '\n@@ -686,6 +697,15 @@ test_atom refs/tags/signed-empty contents:body ''\n test_atom refs/tags/signed-empty contents:signature \"$sig\"\n test_atom refs/tags/signed-empty contents \"$sig\"\n \n+test_expect_success 'basic atom: refs/tags/signed-empty raw' '\n+\tgit cat-file tag refs/tags/signed-empty >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-empty >actual &&\n+\tsanitize_pgp expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\techo \"\" >>expected.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-short subject 'subject line'\n test_atom refs/tags/signed-short subject:sanitize 'subject-line'\n test_atom refs/tags/signed-short contents:subject 'subject line'\n@@ -695,6 +715,15 @@ test_atom refs/tags/signed-short contents:signature \"$sig\"\n test_atom refs/tags/signed-short contents \"subject line\n $sig\"\n \n+test_expect_success 'basic atom: refs/tags/signed-short raw' '\n+\tgit cat-file tag refs/tags/signed-short >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-short >actual &&\n+\tsanitize_pgp expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\techo \"\" >>expected.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_atom refs/tags/signed-long subject 'subject line'\n test_atom refs/tags/signed-long subject:sanitize 'subject-line'\n test_atom refs/tags/signed-long contents:subject 'subject line'\n@@ -708,6 +737,15 @@ test_atom refs/tags/signed-long contents \"subject line\n body contents\n $sig\"\n \n+test_expect_success 'basic atom: refs/tags/signed-long raw' '\n+\tgit cat-file tag refs/tags/signed-long >expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/tags/signed-long >actual &&\n+\tsanitize_pgp expected.clean &&\n+\tsanitize_pgp actual.clean &&\n+\techo \"\" >>expected.clean &&\n+\ttest_cmp expected.clean actual.clean\n+'\n+\n test_expect_success 'set up refs pointing to tree and blob' '\n \tgit update-ref refs/mytrees/first refs/heads/main^{tree} &&\n \tgit update-ref refs/myblobs/first refs/heads/main:one\n@@ -720,6 +758,16 @@ test_atom refs/mytrees/first contents:body \"\"\n test_atom refs/mytrees/first contents:signature \"\"\n test_atom refs/mytrees/first contents \"\"\n \n+test_expect_success 'basic atom: refs/mytrees/first raw' '\n+\tgit cat-file tree refs/mytrees/first >expected &&\n+\techo \"\" >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/mytrees/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_atom refs/myblobs/first subject \"\"\n test_atom refs/myblobs/first contents:subject \"\"\n test_atom refs/myblobs/first body \"\"\n@@ -727,6 +775,165 @@ test_atom refs/myblobs/first contents:body \"\"\n test_atom refs/myblobs/first contents:signature \"\"\n test_atom refs/myblobs/first contents \"\"\n \n+test_expect_success 'basic atom: refs/myblobs/first raw' '\n+\tgit cat-file blob refs/myblobs/first >expected &&\n+\techo \"\" >>expected &&\n+\tgit for-each-ref --format=\"%(raw)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual &&\n+\tgit cat-file -s refs/myblobs/first >expected &&\n+\tgit for-each-ref --format=\"%(raw:size)\" refs/myblobs/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'set up refs pointing to binary blob' '\n+\tprintf \"%b\" \"a\\0b\\0c\" >blob1 &&\n+\tprintf \"%b\" \"a\\0c\\0b\" >blob2 &&\n+\tprintf \"%b\" \"\\0a\\0b\\0c\" >blob3 &&\n+\tprintf \"%b\" \"abc\" >blob4 &&\n+\tprintf \"%b\" \"\\0 \\0 \\0 \" >blob5 &&\n+\tprintf \"%b\" \"\\0 \\0a\\0 \" >blob6 &&\n+\tprintf \"%b\" \" \" >blob7 &&\n+\t>blob8 &&\n+\tgit hash-object blob1 -w | xargs git update-ref refs/myblobs/blob1 &&\n+\tgit hash-object blob2 -w | xargs git update-ref refs/myblobs/blob2 &&\n+\tgit hash-object blob3 -w | xargs git update-ref refs/myblobs/blob3 &&\n+\tgit hash-object blob4 -w | xargs git update-ref refs/myblobs/blob4 &&\n+\tgit hash-object blob5 -w | xargs git update-ref refs/myblobs/blob5 &&\n+\tgit hash-object blob6 -w | xargs git update-ref refs/myblobs/blob6 &&\n+\tgit hash-object blob7 -w | xargs git update-ref refs/myblobs/blob7 &&\n+\tgit hash-object blob8 -w | xargs git update-ref refs/myblobs/blob8\n+'\n+\n+test_expect_success 'Verify sorts with raw' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob7\n+\trefs/mytrees/first\n+\trefs/myblobs/first\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob4\n+\trefs/heads/main\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'Verify sorts with raw:size' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\trefs/myblobs/blob7\n+\trefs/heads/main\n+\trefs/myblobs/blob4\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/mytrees/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --sort=raw:size \\\n+\t\trefs/heads/main refs/myblobs/ refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'validate raw atom with %(if:equals)' '\n+\tcat >expected <<-EOF &&\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\trefs/myblobs/blob4\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tnot equals\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:equals=abc)%(raw)%(then)%(refname)%(else)not equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+test_expect_success 'validate raw atom with %(if:notequals)' '\n+\tcat >expected <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/heads/main\n+\trefs/heads/newtag\n+\trefs/myblobs/blob1\n+\trefs/myblobs/blob2\n+\trefs/myblobs/blob3\n+\tequals\n+\trefs/myblobs/blob5\n+\trefs/myblobs/blob6\n+\trefs/myblobs/blob7\n+\trefs/myblobs/blob8\n+\trefs/myblobs/first\n+\tEOF\n+\tgit for-each-ref --format=\"%(if:notequals=abc)%(raw)%(then)%(refname)%(else)equals%(end)\" \\\n+\t\trefs/myblobs/ refs/heads/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'empty raw refs with %(if)' '\n+\tcat >expected <<-EOF &&\n+\trefs/myblobs/blob1 not empty\n+\trefs/myblobs/blob2 not empty\n+\trefs/myblobs/blob3 not empty\n+\trefs/myblobs/blob4 not empty\n+\trefs/myblobs/blob5 not empty\n+\trefs/myblobs/blob6 not empty\n+\trefs/myblobs/blob7 empty\n+\trefs/myblobs/blob8 empty\n+\trefs/myblobs/first not empty\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname) %(if)%(raw)%(then)not empty%(else)empty%(end)\" \\\n+\t\trefs/myblobs/ >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success '%(raw) with --python must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --python\n+'\n+\n+test_expect_success '%(raw) with --tcl must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --tcl\n+'\n+\n+test_expect_success '%(raw) with --perl must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --perl\n+'\n+\n+test_expect_success '%(raw) with --shell must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --shell\n+'\n+\n+test_expect_success '%(raw) with --shell and --sort=raw must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw)\" --sort=raw --shell\n+'\n+\n+test_expect_success '%(raw:size) with --shell' '\n+\tgit for-each-ref --format=\"%(raw:size)\" | while read line\n+\tdo\n+\t\techo \"'\\''$line'\\''\" >>expect\n+\tdone &&\n+\tgit for-each-ref --format=\"%(raw:size)\" --shell >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for-each-ref --format compare with cat-file --batch' '\n+\tgit rev-parse refs/mytrees/first | git cat-file --batch >expected &&\n+\tgit for-each-ref --format=\"%(objectname) %(objecttype) %(objectsize)\n+%(raw)\" refs/mytrees/first >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'set up multiple-sort tags' '\n \tfor when in 100000 200000\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"426476","messageId":"48d256db5c349c1fa0615bb60d74039c78a831fd.1622884415.git.gitgitgadget@gmail.com","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"[PATCH 1/6] [GSOC] ref-filter: add obj-type check in grab contents","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-05T09:13:29Z","receivedAt":"2021-06-05T09:13:55Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nOnly tag and commit objects use `grab_sub_body_contents()` to grab\nobject contents in the current codebase. We want to teach the\nfunction to also handle blobs and trees to get their raw data,\nwithout parsing a blob (whose contents looks like a commit or a tag)\nincorrectly as a commit or a tag.\n\nSkip the block of code that is specific to handling commits and tags\nearly when the given object is of a wrong type to help later\naddition to handle other types of objects in this function.\n\nMentored-by: Christian Couder \nMentored-by: Hariom Verma \nHelped-by: Junio C Hamano \nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4db0e40ff4c6..5cee6512fbaf 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n+static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n {\n \tint i;\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n+\tvoid *buf = data->content;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n@@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)\n \t\t\tcontinue;\n \t\tif (deref)\n \t\t\tname++;\n-\t\tif (strcmp(name, \"body\") &&\n-\t\t !starts_with(name, \"subject\") &&\n-\t\t !starts_with(name, \"trailers\") &&\n-\t\t !starts_with(name, \"contents\"))\n+\n+\t\tif ((data->type != OBJ_TAG &&\n+\t\t data->type != OBJ_COMMIT) ||\n+\t\t (strcmp(name, \"body\") &&\n+\t\t !starts_with(name, \"subject\") &&\n+\t\t !starts_with(name, \"trailers\") &&\n+\t\t !starts_with(name, \"contents\")))\n \t\t\tcontinue;\n \t\tif (!subpos)\n \t\t\tfind_subpos(buf,\n@@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf)\n+static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n {\n+\tvoid *buf = data->content;\n+\n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, buf);\n+\t\tgrab_sub_body_contents(val, deref, data);\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n@@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi->content);\n+\t\tgrab_values(ref->value, deref, *obj, oi);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n-- \ngitgitgadget\n\n"},{"id":"426477","messageId":"a50304454e178a453ede5a217ec8aea7ca68e0e8.1622884415.git.gitgitgadget@gmail.com","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"[PATCH 3/6] [GSOC] ref-filter: use non-const ref_format in *_atom_parser()","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-05T09:13:31Z","receivedAt":"2021-06-05T09:13:57Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nUse non-const ref_format in *_atom_parser(), which can help us\nmodify the members of ref_format in *_atom_parser().\n\nSigned-off-by: ZheNing Hu \n---\n builtin/tag.c | 2 +-\n ref-filter.c | 44 ++++++++++++++++++++++----------------------\n ref-filter.h | 4 ++--\n 3 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 82fcfc098242..452558ec9575 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref,\n \t\t const struct object_id *oid, void *cb_data)\n {\n \tint flags;\n-\tconst struct ref_format *format = cb_data;\n+\tstruct ref_format *format = cb_data;\n \tflags = GPG_VERIFY_VERBOSE;\n \n \tif (format->format)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 46aec291de62..608e38aa4160 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -226,7 +226,7 @@ static int strbuf_addf_ret(struct strbuf *sb, int ret, const char *fmt, ...)\n \treturn ret;\n }\n \n-static int color_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int color_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *color_value, struct strbuf *err)\n {\n \tif (!color_value)\n@@ -264,7 +264,7 @@ static int refname_atom_parser_internal(struct refname_atom *atom, const char *a\n \treturn 0;\n }\n \n-static int remote_ref_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int remote_ref_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct string_list params = STRING_LIST_INIT_DUP;\n@@ -311,7 +311,7 @@ static int remote_ref_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objecttype_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -323,7 +323,7 @@ static int objecttype_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int objectsize_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -343,7 +343,7 @@ static int objectsize_atom_parser(const struct ref_format *format, struct used_a\n \treturn 0;\n }\n \n-static int deltabase_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int deltabase_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -355,7 +355,7 @@ static int deltabase_atom_parser(const struct ref_format *format, struct used_at\n \treturn 0;\n }\n \n-static int body_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int body_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (arg)\n@@ -364,7 +364,7 @@ static int body_atom_parser(const struct ref_format *format, struct used_atom *a\n \treturn 0;\n }\n \n-static int subject_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int subject_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -376,7 +376,7 @@ static int subject_atom_parser(const struct ref_format *format, struct used_atom\n \treturn 0;\n }\n \n-static int trailers_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int trailers_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tatom->u.contents.trailer_opts.no_divider = 1;\n@@ -402,7 +402,7 @@ static int trailers_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int contents_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int contents_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -430,7 +430,7 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n-static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int raw_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -442,7 +442,7 @@ static int raw_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int oid_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -461,7 +461,7 @@ static int oid_atom_parser(const struct ref_format *format, struct used_atom *at\n \treturn 0;\n }\n \n-static int person_email_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int person_email_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg)\n@@ -475,7 +475,7 @@ static int person_email_atom_parser(const struct ref_format *format, struct used\n \treturn 0;\n }\n \n-static int refname_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int refname_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \treturn refname_atom_parser_internal(&atom->u.refname, arg, atom->name, err);\n@@ -492,7 +492,7 @@ static align_type parse_align_position(const char *s)\n \treturn -1;\n }\n \n-static int align_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int align_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tstruct align *align = &atom->u.align;\n@@ -544,7 +544,7 @@ static int align_atom_parser(const struct ref_format *format, struct used_atom *\n \treturn 0;\n }\n \n-static int if_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n \tif (!arg) {\n@@ -559,7 +559,7 @@ static int if_atom_parser(const struct ref_format *format, struct used_atom *ato\n \treturn 0;\n }\n \n-static int head_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n \tatom->u.head = resolve_refdup(\"HEAD\", RESOLVE_REF_READING, NULL, NULL);\n@@ -570,7 +570,7 @@ static struct {\n \tconst char *name;\n \tinfo_source source;\n \tcmp_type cmp_type;\n-\tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n+\tint (*parser)(struct ref_format *format, struct used_atom *atom,\n \t\t const char *arg, struct strbuf *err);\n } valid_atom[] = {\n \t[ATOM_REFNAME] = { \"refname\", SOURCE_NONE, FIELD_STR, refname_atom_parser },\n@@ -649,7 +649,7 @@ struct atom_value {\n /*\n * Used to parse format string and sort specifiers\n */\n-static int parse_ref_filter_atom(const struct ref_format *format,\n+static int parse_ref_filter_atom(struct ref_format *format,\n \t\t\t\t const char *atom, const char *ep,\n \t\t\t\t struct strbuf *err)\n {\n@@ -2545,9 +2545,9 @@ static void append_literal(const char *cp, const char *ep, struct ref_formatting\n }\n \n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n-\t\t\t struct strbuf *final_buf,\n-\t\t\t struct strbuf *error_buf)\n+\t\t\t struct ref_format *format,\n+\t\t\t struct strbuf *final_buf,\n+\t\t\t struct strbuf *error_buf)\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n@@ -2592,7 +2592,7 @@ int format_ref_array_item(struct ref_array_item *info,\n }\n \n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format)\n+\t\t struct ref_format *format)\n {\n \tstruct ref_array_item *ref_item;\n \tstruct strbuf output = STRBUF_INIT;\ndiff --git a/ref-filter.h b/ref-filter.h\nindex baf72a718965..74fb423fc89f 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);\n void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting, unsigned int mask, int on);\n /* Based on the given format and quote_style, fill the strbuf */\n int format_ref_array_item(struct ref_array_item *info,\n-\t\t\t const struct ref_format *format,\n+\t\t\t struct ref_format *format,\n \t\t\t struct strbuf *final_buf,\n \t\t\t struct strbuf *error_buf);\n /* Parse a single sort specifier and add it to the list */\n@@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void);\n * name must be a fully qualified refname.\n */\n void pretty_print_ref(const char *name, const struct object_id *oid,\n-\t\t const struct ref_format *format);\n+\t\t struct ref_format *format);\n \n /*\n * Push a single ref onto the array; this can be used to construct your own\n-- \ngitgitgadget\n\n"},{"id":"426478","messageId":"1b74d4cb85ae197f1e1ab648f6db2fe516a42436.1622884415.git.gitgitgadget@gmail.com","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"[PATCH 5/6] [GSOC] ref-filter: teach grab_sub_body_contents() return value and err","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-05T09:13:33Z","receivedAt":"2021-06-05T09:13:58Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nTeach grab_sub_body_contents() return value and err can help us\nreport more useful error messages when when add %(raw:textconv)\nand %(raw:filter) later.\n\nSigned-off-by: ZheNing Hu \n---\n ref-filter.c | 43 ++++++++++++++++++++++++++++---------------\n 1 file changed, 28 insertions(+), 15 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 695f6f55e3e3..a859a94aa8e0 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1403,7 +1403,8 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data)\n+static int grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data,\n+\t\t\t\t struct strbuf *err)\n {\n \tint i;\n \tconst char *subpos = NULL, *bodypos = NULL, *sigpos = NULL;\n@@ -1478,6 +1479,7 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp\n \n \t}\n \tfree((void *)sigpos);\n+\treturn 0;\n }\n \n /*\n@@ -1501,33 +1503,39 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data)\n+static int grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data, struct strbuf *err)\n {\n \tvoid *buf = data->content;\n+\tint ret = 0;\n \n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, data);\n+\t\tif ((ret = grab_sub_body_contents(val, deref, data, err)))\n+\t\t\treturn ret;\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tgrab_sub_body_contents(val, deref, data);\n+\t\tif ((ret = grab_sub_body_contents(val, deref, data, err)))\n+\t\t\treturn ret;\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n-\t\tgrab_sub_body_contents(val, deref, data);\n+\t\tif ((ret = grab_sub_body_contents(val, deref, data, err)))\n+\t\t\treturn ret;\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n-\t\tgrab_sub_body_contents(val, deref, data);\n+\t\tif ((ret = grab_sub_body_contents(val, deref, data, err)))\n+\t\t\treturn ret;\n \t\tbreak;\n \tdefault:\n \t\tdie(\"Eh? Object of type %d?\", obj->type);\n \t}\n+\treturn ret;\n }\n \n static inline char *copy_advance(char *dst, const char *src)\n@@ -1725,6 +1733,8 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n {\n \t/* parse_object_buffer() will set eaten to 0 if free() will be needed */\n \tint eaten = 1;\n+\tint ret = 0;\n+\n \tif (oi->info.contentp) {\n \t\t/* We need to know that to use parse_object_buffer properly */\n \t\toi->info.sizep = &oi->size;\n@@ -1745,13 +1755,13 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tgrab_values(ref->value, deref, *obj, oi);\n+\t\tret = grab_values(ref->value, deref, *obj, oi, err);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\n \tif (!eaten)\n \t\tfree(oi->content);\n-\treturn 0;\n+\treturn ret;\n }\n \n static void populate_worktree_map(struct hashmap *map, struct worktree **worktrees)\n@@ -1805,7 +1815,7 @@ static char *get_worktree_path(const struct used_atom *atom, const struct ref_ar\n static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n {\n \tstruct object *obj;\n-\tint i;\n+\tint i, ret = 0;\n \tstruct object_info empty = OBJECT_INFO_INIT;\n \n \tCALLOC_ARRAY(ref->value, used_atom_cnt);\n@@ -1961,8 +1971,8 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \n \n \toi.oid = ref->objectname;\n-\tif (get_object(ref, 0, &obj, &oi, err))\n-\t\treturn -1;\n+\tif ((ret = get_object(ref, 0, &obj, &oi, err)))\n+\t\treturn ret;\n \n \t/*\n \t * If there is no atom that wants to know about tagged\n@@ -1993,9 +2003,11 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n static int get_ref_atom_value(struct ref_array_item *ref, int atom,\n \t\t\t struct atom_value **v, struct strbuf *err)\n {\n+\tint ret = 0;\n+\n \tif (!ref->value) {\n-\t\tif (populate_value(ref, err))\n-\t\t\treturn -1;\n+\t\tif ((ret = populate_value(ref, err)))\n+\t\t\treturn ret;\n \t\tfill_missing_values(ref->value);\n \t}\n \t*v = &ref->value[atom];\n@@ -2568,6 +2580,7 @@ int format_ref_array_item(struct ref_array_item *info,\n \t\t\t struct strbuf *error_buf)\n {\n \tconst char *cp, *sp, *ep;\n+\tint ret = 0;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n \n \tstate.quote_style = format->quote_style;\n@@ -2581,10 +2594,10 @@ int format_ref_array_item(struct ref_array_item *info,\n \t\tif (cp < sp)\n \t\t\tappend_literal(cp, sp, &state);\n \t\tpos = parse_ref_filter_atom(format, sp + 2, ep, error_buf);\n-\t\tif (pos < 0 || get_ref_atom_value(info, pos, &atomv, error_buf) ||\n+\t\tif (pos < 0 || ((ret = get_ref_atom_value(info, pos, &atomv, error_buf))) ||\n \t\t atomv->handler(atomv, &state, error_buf)) {\n \t\t\tpop_stack_element(&state.stack);\n-\t\t\treturn -1;\n+\t\t\treturn ret ? ret : -1;\n \t\t}\n \t}\n \tif (*cp) {\n-- \ngitgitgadget\n\n"},{"id":"426479","messageId":"ccdd18ad508824aa206a02c479229d0ede69522d.1622884415.git.gitgitgadget@gmail.com","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"[PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-05T09:13:32Z","receivedAt":"2021-06-05T09:13:59Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nIn order to let \"cat-file --batch=%(rest)\" use the ref-filter\ninterface, add %(rest) atom for ref-filter and --rest option\nfor \"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n`--rest` specify a string to replace %(rest) placeholders of\nthe --format option.\n\nSigned-off-by: ZheNing Hu \n---\n Documentation/git-branch.txt | 6 +++++-\n Documentation/git-for-each-ref.txt | 10 +++++++++-\n Documentation/git-tag.txt | 4 ++++\n Documentation/git-verify-tag.txt | 6 +++++-\n builtin/branch.c | 5 +++++\n builtin/for-each-ref.c | 6 ++++++\n builtin/tag.c | 6 ++++++\n builtin/verify-tag.c | 1 +\n ref-filter.c | 21 +++++++++++++++++++++\n ref-filter.h | 5 ++++-\n t/t3203-branch-output.sh | 14 ++++++++++++++\n t/t6300-for-each-ref.sh | 24 ++++++++++++++++++++++++\n t/t7004-tag.sh | 10 ++++++++++\n t/t7030-verify-tag.sh | 8 ++++++++\n 14 files changed, 122 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-branch.txt b/Documentation/git-branch.txt\nindex 94dc9a54f2d7..29fa579e3d8a 100644\n--- a/Documentation/git-branch.txt\n+++ b/Documentation/git-branch.txt\n@@ -15,7 +15,7 @@ SYNOPSIS\n \t[--contains []] [--no-contains []]\n \t[--points-at ] [--format=]\n \t[(-r | --remotes) | (-a | --all)]\n-\t[--list] [...]\n+\t[--list] [...] [--rest=]\n 'git branch' [--track | --no-track] [-f] []\n 'git branch' (--set-upstream-to= | -u ) []\n 'git branch' --unset-upstream []\n@@ -298,6 +298,10 @@ start-point is either a local or remote-tracking branch.\n \tand the object it points at. The format is the same as\n \tthat of linkgit:git-for-each-ref[1].\n \n+--rest=::\n+\tIf given, the `%(rest)` placeholders in the `--format` option\n+\twill be replaced.\n+\n CONFIGURATION\n -------------\n `pager.branch` is only respected when listing branches, i.e., when\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 8f8d8cd1e04f..e85b7b19a530 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n \t\t [(--sort=)...] [--format=] [...]\n-\t\t [--points-at=]\n+\t\t [--points-at=] [--rest=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n \n@@ -74,6 +74,10 @@ OPTIONS\n --points-at=::\n \tOnly list refs which points at the given object.\n \n+--rest=::\n+\tIf given, the `%(rest)` placeholders in the `--format` option\n+\twill be replaced by its contents.\n+\n --merged[=]::\n \tOnly list refs whose tips are reachable from the\n \tspecified commit (HEAD if not specified).\n@@ -235,6 +239,10 @@ and `date` to extract the named component. For email fields (`authoremail`,\n without angle brackets, and `:localpart` to get the part before the `@` symbol\n out of the trimmed email.\n \n+rest::\n+\tThe placeholder for `--rest` specified string. If no `--rest`, nothing\n+\tis printed.\n+\n The raw data in a object is `raw`.\n \n raw:size::\ndiff --git a/Documentation/git-tag.txt b/Documentation/git-tag.txt\nindex 31a97a1b6c5b..3bf6ae7216de 100644\n--- a/Documentation/git-tag.txt\n+++ b/Documentation/git-tag.txt\n@@ -200,6 +200,10 @@ This option is only applicable when listing tags without annotation lines.\n \tthat of linkgit:git-for-each-ref[1]. When unspecified,\n \tdefaults to `%(refname:strip=2)`.\n \n+--rest=::\n+\tIf given, the `%(rest)` placeholders in the `--format` option\n+\twill be replaced.\n+\n ::\n \tThe name of the tag to create, delete, or describe.\n \tThe new tag name must pass all checks defined by\ndiff --git a/Documentation/git-verify-tag.txt b/Documentation/git-verify-tag.txt\nindex 0b8075dad965..7ba4a6941ab1 100644\n--- a/Documentation/git-verify-tag.txt\n+++ b/Documentation/git-verify-tag.txt\n@@ -8,7 +8,7 @@ git-verify-tag - Check the GPG signature of tags\n SYNOPSIS\n --------\n [verse]\n-'git verify-tag' [--format=] ...\n+'git verify-tag' [--format=] [--rest=] ...\n \n DESCRIPTION\n -----------\n@@ -27,6 +27,10 @@ OPTIONS\n ...::\n \tSHA-1 identifiers of Git tag objects.\n \n+--rest=::\n+\tIf given, the `%(rest)` placeholders in the `--format` option\n+\twill be replaced.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex b23b1d1752af..b03a4c49c1d9 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -439,6 +439,10 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \tif (verify_ref_format(format))\n \t\tdie(_(\"unable to parse format string\"));\n \n+\tif (format->use_rest)\n+\t\tfor (i = 0; i < array.nr; i++)\n+\t\t\tarray.items[i]->rest = format->rest;\n+\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\n@@ -670,6 +674,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"print only branches of the object\"), parse_opt_object_name),\n \t\tOPT_BOOL('i', \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n+\t\tOPT_STRING(0, \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 89cb6307d46f..fac7777fd2c0 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -37,6 +37,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \t\tOPT_GROUP(\"\"),\n \t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only matched refs\")),\n+\t\tOPT_STRING( 0 , \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_SORT(sorting_tail),\n@@ -78,6 +79,11 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tfilter.name_patterns = argv;\n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL | FILTER_REFS_INCLUDE_BROKEN);\n+\n+\tif (format.use_rest)\n+\t\tfor (i = 0; i < array.nr; i++)\n+\t\t\tarray.items[i]->rest = format.rest;\n+\n \tref_array_sort(sorting, &array);\n \n \tif (!maxcount || array.nr < maxcount)\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 452558ec9575..9e52b3eacb16 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -63,6 +63,11 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\tdie(_(\"unable to parse format string\"));\n \tfilter->with_commit_tag_algo = 1;\n \tfilter_refs(&array, filter, FILTER_REFS_TAGS);\n+\n+\tif (format->use_rest)\n+\t\tfor (i = 0; i < array.nr; i++)\n+\t\t\tarray.items[i]->rest = format->rest;\n+\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\n@@ -481,6 +486,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"),\n \t\t\t N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n+\t\tOPT_STRING(0, \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n \t\tOPT_BOOL('i', \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n \t\tOPT_END()\n \t};\ndiff --git a/builtin/verify-tag.c b/builtin/verify-tag.c\nindex f45136a06ba7..00be4899678d 100644\n--- a/builtin/verify-tag.c\n+++ b/builtin/verify-tag.c\n@@ -36,6 +36,7 @@ int cmd_verify_tag(int argc, const char **argv, const char *prefix)\n \t\tOPT__VERBOSE(&verbose, N_(\"print tag contents\")),\n \t\tOPT_BIT(0, \"raw\", &flags, N_(\"print raw gpg status output\"), GPG_VERIFY_RAW),\n \t\tOPT_STRING(0, \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n+\t\tOPT_STRING(0, \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 608e38aa4160..695f6f55e3e3 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -157,6 +157,7 @@ enum atom_type {\n \tATOM_IF,\n \tATOM_THEN,\n \tATOM_ELSE,\n+\tATOM_REST,\n };\n \n /*\n@@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"%%(rest) does not take arguments\"));\n+\tformat->use_rest = 1;\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *unused_err)\n {\n@@ -615,6 +625,7 @@ static struct {\n \t[ATOM_IF] = { \"if\", SOURCE_NONE, FIELD_STR, if_atom_parser },\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n+\t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -1919,6 +1930,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tv->handler = else_atom_handler;\n \t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_REST) {\n+\t\t\tif (ref->rest)\n+\t\t\t\tv->s = xstrdup(ref->rest);\n+\t\t\telse\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2136,6 +2153,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \n \tFLEX_ALLOC_STR(ref, refname, refname);\n \toidcpy(&ref->objectname, oid);\n+\tref->rest = NULL;\n \n \treturn ref;\n }\n@@ -2600,6 +2618,9 @@ void pretty_print_ref(const char *name, const struct object_id *oid,\n \n \tref_item = new_ref_array_item(name, oid);\n \tref_item->kind = ref_kind_from_refname(name);\n+\tif (format->use_rest)\n+\t\tref_item->rest = format->rest;\n+\n \tif (format_ref_array_item(ref_item, format, &output, &err))\n \t\tdie(\"%s\", err.buf);\n \tfwrite(output.buf, 1, output.len, stdout);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 74fb423fc89f..9dc07476a584 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -38,6 +38,7 @@ struct ref_sorting {\n \n struct ref_array_item {\n \tstruct object_id objectname;\n+\tconst char *rest;\n \tint flag;\n \tunsigned int kind;\n \tconst char *symref;\n@@ -76,14 +77,16 @@ struct ref_format {\n \t * verify_ref_format() afterwards to finalize.\n \t */\n \tconst char *format;\n+\tconst char *rest;\n \tint quote_style;\n+\tint use_rest;\n \tint use_color;\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n };\n \n-#define REF_FORMAT_INIT { NULL, 0, -1 }\n+#define REF_FORMAT_INIT { NULL, NULL, 0, 0, -1 }\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\ndiff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh\nindex 5325b9f67a00..dc6bf2557088 100755\n--- a/t/t3203-branch-output.sh\n+++ b/t/t3203-branch-output.sh\n@@ -340,6 +340,20 @@ test_expect_success 'git branch --format option' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git branch with --format and --rest option' '\n+\tcat >expect <<-\\EOF &&\n+\t@Refname is (HEAD detached from fromtag)\n+\t@Refname is refs/heads/ambiguous\n+\t@Refname is refs/heads/branch-one\n+\t@Refname is refs/heads/branch-two\n+\t@Refname is refs/heads/main\n+\t@Refname is refs/heads/ref-to-branch\n+\t@Refname is refs/heads/ref-to-remote\n+\tEOF\n+\tgit branch --rest=\"@\" --format=\"%(rest)Refname is %(refname)\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'worktree colors correct' '\n \tcat >expect <<-EOF &&\n \t* (HEAD detached from fromtag)\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 5f66d933ace0..e908b2ca0522 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1187,6 +1187,30 @@ test_expect_success 'basic atom: head contents:trailers' '\n \ttest_cmp expect actual.clean\n '\n \n+test_expect_success 'bacic atom: rest with --rest' '\n+\tgit for-each-ref --format=\"###refname=%(refname)\n+###oid=%(objectname)\n+###type=%(objecttype)\n+###size=%(objectsize)\" refs/heads/main refs/tags/subject-body >expect &&\n+\tgit for-each-ref --rest=\"###\" --format=\"%(rest)refname=%(refname)\n+%(rest)oid=%(objectname)\n+%(rest)type=%(objecttype)\n+%(rest)size=%(objectsize)\" refs/heads/main refs/tags/subject-body >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'bacic atom: *rest with --rest' '\n+\tgit for-each-ref --format=\"###refname=%(refname)\n+###oid=%(objectname)\n+###type=%(objecttype)\n+###size=%(objectsize)\" refs/heads/main refs/tags/subject-body >expect &&\n+\tgit for-each-ref --rest=\"###\" --format=\"%(*rest)refname=%(refname)\n+%(rest)oid=%(objectname)\n+%(rest)type=%(objecttype)\n+%(rest)size=%(objectsize)\" refs/heads/main refs/tags/subject-body >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'trailer parsing not fooled by --- line' '\n \tgit commit --allow-empty -F - <<-\\EOF &&\n \tthis is the subject\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 2f72c5c6883e..93c4366ff52d 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1998,6 +1998,16 @@ test_expect_success '--format should list tags as per format given' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'tag -l with --format and --rest' '\n+\tcat >expect <<-\\EOF &&\n+\t#refname : refs/tags/v1.0\n+\t#refname : refs/tags/v1.0.1\n+\t#refname : refs/tags/v1.1.3\n+\tEOF\n+\tgit tag -l --rest=\"#\" --format=\"%(rest)refname : %(refname)\" \"v1*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success \"set up color tests\" '\n \techo \"v1.0\" >expect.color &&\n \techo \"v1.0\" >expect.bare &&\ndiff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh\nindex 3cefde9602bf..008df00c32d6 100755\n--- a/t/t7030-verify-tag.sh\n+++ b/t/t7030-verify-tag.sh\n@@ -194,6 +194,14 @@ test_expect_success GPG 'verifying tag with --format' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success GPG 'verifying tag with --format and --rest' '\n+\tcat >expect <<-\\EOF &&\n+\t#tagname : fourth-signed\n+\tEOF\n+\tgit verify-tag --rest=\"#\" --format=\"%(rest)tagname : %(tag)\" \"fourth-signed\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success GPG 'verifying a forged tag with --format should fail silently' '\n \ttest_must_fail git verify-tag --format=\"tagname : %(tag)\" $(cat forged1.tag) >actual-forged &&\n \ttest_must_be_empty actual-forged\n-- \ngitgitgadget\n\n"},{"id":"426480","messageId":"6170eb2fcac18d51a51c7f5f3890d004549812c9.1622884415.git.gitgitgadget@gmail.com","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"[PATCH 6/6] [GSOC] ref-filter: add %(raw:textconv) and %(raw:filters)","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-05T09:13:34Z","receivedAt":"2021-06-05T09:14:01Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"From: ZheNing Hu \n\nIn order to let `git cat-file --batch --filter` and\n`git cat-file --batch --textconv` use ref-filter\ninterface, `%(raw:textconv)` and `%(raw:filters)` are\nadded to ref-filter.\n\n`--rest` contents is used as the `` for them.\n\n`%(raw:textconv)` can show the object' contents as transformed\nby a textconv filter.\n\n`%(raw:filters)` can show the content as converted by the filters\nconfigured in the current working tree for the given ``\n(i.e. smudge filters, end-of-line conversion, etc).\n\nIn addition, they cannot be used with `--python`, `--tcl`,\n`--shell`, `--perl`.\n\nSigned-off-by: ZheNing Hu \n---\n Documentation/git-for-each-ref.txt | 18 ++++++--\n builtin/for-each-ref.c | 11 ++++-\n ref-filter.c | 68 ++++++++++++++++++++++++------\n t/t6300-for-each-ref.sh | 63 +++++++++++++++++++++++++++\n 4 files changed, 141 insertions(+), 19 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex e85b7b19a530..e49b8861f474 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -76,7 +76,8 @@ OPTIONS\n \n --rest=::\n \tIf given, the `%(rest)` placeholders in the `--format` option\n-\twill be replaced by its contents.\n+\twill be replaced by its contents. At the same time, its contents\n+\tis used as the `` for `%(raw:textconv)` and `%(raw:filter)`.\n \n --merged[=]::\n \tOnly list refs whose tips are reachable from the\n@@ -248,9 +249,18 @@ The raw data in a object is `raw`.\n raw:size::\n \tThe raw data size of the object.\n \n-Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`,\n-`--perl` because the host language may not support arbitrary binary data in the\n-variables of its string type.\n+raw:textconv::\n+\tShow the content as transformed by a textconv filter. It must be used\n+\twith `--rest`.\n+\n+raw:filters::\n+\tShow the content as converted by the filters configured in\n+\tthe current working tree for the given `` (i.e. smudge filters,\n+\tend-of-line conversion, etc). It must be used with `--rest`.\n+\n+Note that `--format=%(raw)`, `--format=%(raw:textconv)`, `--format=%(raw:filter)`\n+can not be used with `--python`, `--shell`, `--tcl`, `--perl` because the host\n+language may not support arbitrary binary data in the variables of its string type.\n \n The message in a commit or a tag object is `contents`, from which\n `contents:` can be used to extract various parts out of:\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex fac7777fd2c0..4be5ddacbac1 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -5,6 +5,7 @@\n #include \"object.h\"\n #include \"parse-options.h\"\n #include \"ref-filter.h\"\n+#include \"userdiff.h\"\n \n static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n@@ -14,6 +15,14 @@ static char const * const for_each_ref_usage[] = {\n \tNULL\n };\n \n+static int git_for_each_ref_config(const char *var, const char *value, void *cb)\n+{\n+\tif (userdiff_config(var, value) < 0)\n+\t\treturn -1;\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -57,7 +66,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tformat.format = \"%(objectname) %(objecttype)\\t%(refname)\";\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(git_for_each_ref_config, NULL);\n \n \tparse_options(argc, argv, prefix, opts, for_each_ref_usage, 0);\n \tif (maxcount < 0) {\ndiff --git a/ref-filter.c b/ref-filter.c\nindex a859a94aa8e0..ae684924b363 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1,3 +1,4 @@\n+#define USE_THE_INDEX_COMPATIBILITY_MACROS\n #include \"builtin.h\"\n #include \"cache.h\"\n #include \"parse-options.h\"\n@@ -192,7 +193,7 @@ static struct used_atom {\n \t\t\tunsigned int nlines;\n \t\t} contents;\n \t\tstruct {\n-\t\t\tenum { RAW_BARE, RAW_LENGTH } option;\n+\t\t\tenum { RAW_BARE, RAW_FILTERS, RAW_LENGTH, RAW_TEXT_CONV } option;\n \t\t} raw_data;\n \t\tstruct {\n \t\t\tcmp_status cmp_status;\n@@ -434,12 +435,19 @@ static int contents_atom_parser(struct ref_format *format, struct used_atom *ato\n static int raw_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t\tconst char *arg, struct strbuf *err)\n {\n-\tif (!arg)\n+\tif (!arg) {\n \t\tatom->u.raw_data.option = RAW_BARE;\n-\telse if (!strcmp(arg, \"size\"))\n+\t} else if (!strcmp(arg, \"size\")) {\n \t\tatom->u.raw_data.option = RAW_LENGTH;\n-\telse\n+\t} else if (!strcmp(arg, \"filters\")) {\n+\t\tatom->u.raw_data.option = RAW_FILTERS;\n+\t\tformat->use_rest = 1;\n+\t} else if (!strcmp(arg, \"textconv\")) {\n+\t\tatom->u.raw_data.option = RAW_TEXT_CONV;\n+\t\tformat->use_rest = 1;\n+\t} else {\n \t\treturn strbuf_addf_ret(err, -1, _(\"unrecognized %%(raw) argument: %s\"), arg);\n+\t}\n \treturn 0;\n }\n \n@@ -1018,7 +1026,9 @@ int verify_ref_format(struct ref_format *format)\n \t\tif (at < 0)\n \t\t\tdie(\"%s\", err.buf);\n \t\tif (format->quote_style && used_atom[at].atom_type == ATOM_RAW &&\n-\t\t used_atom[at].u.raw_data.option == RAW_BARE)\n+\t\t (used_atom[at].u.raw_data.option == RAW_BARE ||\n+\t\t used_atom[at].u.raw_data.option == RAW_FILTERS ||\n+\t\t used_atom[at].u.raw_data.option == RAW_TEXT_CONV))\n \t\t\tdie(_(\"--format=%.*s cannot be used with\"\n \t\t\t \"--python, --shell, --tcl, --perl\"), (int)(ep - sp - 2), sp + 2);\n \t\tcp = ep + 1;\n@@ -1403,7 +1413,7 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size\n }\n \n /* See grab_values */\n-static int grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data,\n+static int grab_sub_body_contents(struct ref_array_item *ref, int deref, struct expand_data *data,\n \t\t\t\t struct strbuf *err)\n {\n \tint i;\n@@ -1411,6 +1421,7 @@ static int grab_sub_body_contents(struct atom_value *val, int deref, struct expa\n \tsize_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0;\n \tvoid *buf = data->content;\n \tunsigned long buf_size = data->size;\n+\tstruct atom_value *val = ref->value;\n \n \tfor (i = 0; i < used_atom_cnt; i++) {\n \t\tstruct used_atom *atom = &used_atom[i];\n@@ -1425,12 +1436,39 @@ static int grab_sub_body_contents(struct atom_value *val, int deref, struct expa\n \n \t\tif (atom_type == ATOM_RAW) {\n \t\t\tif (atom->u.raw_data.option == RAW_BARE) {\n-\t\t\t\tv->s = xmemdupz(buf, buf_size);\n-\t\t\t\tv->s_size = buf_size;\n+\t\t\t\tgoto bare;\n \t\t\t} else if (atom->u.raw_data.option == RAW_LENGTH) {\n \t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX, (uintmax_t)buf_size);\n+\t\t\t} else if (atom->u.raw_data.option == RAW_FILTERS ||\n+\t\t\t\t atom->u.raw_data.option == RAW_TEXT_CONV) {\n+\t\t\t\tif (!ref->rest)\n+\t\t\t\t\treturn strbuf_addf_ret(err, -1, _(\"missing path for '%s'\"),\n+\t\t\t\t\t\t\t oid_to_hex(&data->oid));\n+\t\t\t\tif (data->type != OBJ_BLOB)\n+\t\t\t\t\tgoto bare;\n+\t\t\t\tif (atom->u.raw_data.option == RAW_FILTERS) {\n+\t\t\t\t\tstruct strbuf strbuf = STRBUF_INIT;\n+\t\t\t\t\tstruct checkout_metadata meta;\n+\n+\t\t\t\t\tinit_checkout_metadata(&meta, NULL, NULL, &data->oid);\n+\t\t\t\t\tif (convert_to_working_tree(&the_index, ref->rest, buf, data->size, &strbuf, &meta)) {\n+\t\t\t\t\t\tv->s_size = strbuf.len;\n+\t\t\t\t\t\tv->s = strbuf_detach(&strbuf, NULL);\n+\t\t\t\t\t} else {\n+\t\t\t\t\t\tgoto bare;\n+\t\t\t\t\t}\n+\t\t\t\t} else if (atom->u.raw_data.option == RAW_TEXT_CONV) {\n+\t\t\t\t\tif (!textconv_object(the_repository,\n+\t\t\t\t\t\t\tref->rest, 0100644, &data->oid,\n+\t\t\t\t\t\t\t1, (char **)(&v->s), &v->s_size))\n+\t\t\t\t\t\tgoto bare;\n+\t\t\t\t}\n \t\t\t}\n \t\t\tcontinue;\n+bare:\n+\t\t\tv->s = xmemdupz(buf, buf_size);\n+\t\t\tv->s_size = buf_size;\n+\t\t\tcontinue;\n \t\t}\n \n \t\tif ((data->type != OBJ_TAG &&\n@@ -1503,33 +1541,35 @@ static void fill_missing_values(struct atom_value *val)\n * pointed at by the ref itself; otherwise it is the object the\n * ref (which is a tag) refers to.\n */\n-static int grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data, struct strbuf *err)\n+static int grab_values(struct ref_array_item *ref, int deref, struct object *obj,\n+\t\t struct expand_data *data, struct strbuf *err)\n {\n \tvoid *buf = data->content;\n+\tstruct atom_value *val = ref->value;\n \tint ret = 0;\n \n \tswitch (obj->type) {\n \tcase OBJ_TAG:\n \t\tgrab_tag_values(val, deref, obj);\n-\t\tif ((ret = grab_sub_body_contents(val, deref, data, err)))\n+\t\tif ((ret = grab_sub_body_contents(ref, deref, data, err)))\n \t\t\treturn ret;\n \t\tgrab_person(\"tagger\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \t\tgrab_commit_values(val, deref, obj);\n-\t\tif ((ret = grab_sub_body_contents(val, deref, data, err)))\n+\t\tif ((ret = grab_sub_body_contents(ref, deref, data, err)))\n \t\t\treturn ret;\n \t\tgrab_person(\"author\", val, deref, buf);\n \t\tgrab_person(\"committer\", val, deref, buf);\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\t/* grab_tree_values(val, deref, obj, buf, sz); */\n-\t\tif ((ret = grab_sub_body_contents(val, deref, data, err)))\n+\t\tif ((ret = grab_sub_body_contents(ref, deref, data, err)))\n \t\t\treturn ret;\n \t\tbreak;\n \tcase OBJ_BLOB:\n \t\t/* grab_blob_values(val, deref, obj, buf, sz); */\n-\t\tif ((ret = grab_sub_body_contents(val, deref, data, err)))\n+\t\tif ((ret = grab_sub_body_contents(ref, deref, data, err)))\n \t\t\treturn ret;\n \t\tbreak;\n \tdefault:\n@@ -1755,7 +1795,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj\n \t\t\treturn strbuf_addf_ret(err, -1, _(\"parse_object_buffer failed on %s for %s\"),\n \t\t\t\t\t oid_to_hex(&oi->oid), ref->refname);\n \t\t}\n-\t\tret = grab_values(ref->value, deref, *obj, oi, err);\n+\t\tret = grab_values(ref, deref, *obj, oi, err);\n \t}\n \n \tgrab_common_values(ref->value, deref, oi);\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex e908b2ca0522..4ba2aa2dcd73 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1365,6 +1365,69 @@ test_expect_success 'for-each-ref --ignore-case works on multiple sort keys' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'ref-filter raw:textconv and raw:filter setup ' '\n+\techo \"*.txt eol=crlf diff=txt\" >.gitattributes &&\n+\techo \"hello\" | append_cr >world.txt &&\n+\tgit add .gitattributes world.txt &&\n+\ttest_tick &&\n+\tgit commit -m \"Initial commit\" &&\n+\tgit update-ref refs/myblobs/world_blob HEAD:world.txt\n+'\n+\n+test_expect_success 'basic atom raw:filters with --rest' '\n+\tsha1=$(git rev-parse -q --verify HEAD:world.txt) &&\n+\tgit for-each-ref --format=\"%(objectname) %(raw:filters)\" --rest=\"HEAD:world.txt\" \\\n+\t refs/myblobs/world_blob >actual &&\n+\tprintf \"%s hello\\r\\n\\n\" $sha1 >expect &&\n+\ttest_cmp expect actual &&\n+\tgit for-each-ref --format=\"%(objectname) %(raw:filters)\" --rest=\"HEAD:world.txt2\" \\\n+\t refs/myblobs/world_blob >actual &&\n+\tprintf \"%s hello\\n\\n\" $sha1 >expect &&\n+\ttest_cmp expect actual\n+\n+'\n+\n+test_expect_success 'basic atom raw:textconv with --rest' '\n+\tsha1=$(git rev-parse -q --verify HEAD:world.txt) &&\n+\tgit -c diff.txt.textconv=\"tr A-Za-z N-ZA-Mn-za-m <\" \\\n+\t for-each-ref --format=\"%(objectname) %(raw:textconv)\" --rest=\"HEAD:world.txt\" \\\n+\t refs/myblobs/world_blob >actual &&\n+\tprintf \"%s uryyb\\r\\n\\n\" $sha1 >expect &&\n+\ttest_cmp expect actual &&\n+\tgit for-each-ref --format=\"%(objectname) %(raw:textconv)\" --rest=\"HEAD:world.txt2\" \\\n+\t refs/myblobs/world_blob >actual &&\n+\tprintf \"%s hello\\n\\n\" $sha1 >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '%(raw:textconv) without --rest must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw:textconv)\"\n+'\n+\n+test_expect_success '%(raw:filters) without --rest must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw:filters)\"\n+'\n+\n+test_expect_success '%(raw:textconv) with --shell must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw:textconv)\" \\\n+\t\t\t --shell --rest=\"HEAD:world.txt\"\n+'\n+\n+test_expect_success '%(raw:filters) with --shell must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw:filters)\" \\\n+\t\t\t --shell --rest=\"HEAD:world.txt\"\n+'\n+\n+test_expect_success '%(raw:textconv) with --shell and --sort=raw:textconv must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw:textconv)\" \\\n+\t\t\t --sort=raw:textconv --shell --rest=\"HEAD:world.txt\"\n+'\n+\n+test_expect_success '%(raw:filters) with --shell and --sort=raw:filters must failed' '\n+\ttest_must_fail git for-each-ref --format=\"%(raw:filters)\" \\\n+\t\t\t --sort=raw:filters --shell --rest=\"HEAD:world.txt\"\n+'\n+\n test_expect_success 'for-each-ref reports broken tags' '\n \tgit tag -m \"good tag\" broken-tag-good HEAD &&\n \tgit cat-file tag broken-tag-good >good &&\n-- \ngitgitgadget\n"},{"id":"426481","messageId":"pull.972.git.1622884415.gitgitgadget@gmail.com","threadId":"55843","inReplyTo":null,"subject":"[PATCH 0/6] [GSOC][RFC] ref-filter: add %(raw:textconv) and %(raw:filters)","fromName":"ZheNing Hu via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-05T09:13:28Z","receivedAt":"2021-06-05T09:14:49Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"In order to let git cat-file --batch reuse ref-filter logic, This patch,\n%(rest), %(raw:textconv), %(raw:filters) atoms and --rest= option are\nadded to ref-filter.\n\n * %(rest) int the format will be replaced by the in --rest=.\n * the in --rest= can also be used as the for\n %(raw:textconv) and %(raw:filters).\n * %(raw:textconv) can show the object's contents as transformed by a\n textconv filter.\n * %(raw:filters) can show the content as converted by the filters\n configured in the current working tree for the given (i.e. smudge\n filters, end-of-line conversion, etc).\n\nThe current series is based on 0efed9435 ([GSOC] ref-filter: add %(raw)\natom)\nhttps://lore.kernel.org/git/pull.966.v2.git.1622808751.gitgitgadget@gmail.com/\nIf necessary, \"%(rest)\" part can be an independent patch later.\n\nZheNing Hu (6):\n [GSOC] ref-filter: add obj-type check in grab contents\n [GSOC] ref-filter: add %(raw) atom\n [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n [GSOC] ref-filter: add %(rest) atom and --rest option\n [GSOC] ref-filter: teach grab_sub_body_contents() return value and err\n [GSOC] ref-filter: add %(raw:textconv) and %(raw:filters)\n\n Documentation/git-branch.txt | 6 +-\n Documentation/git-for-each-ref.txt | 29 ++-\n Documentation/git-tag.txt | 4 +\n Documentation/git-verify-tag.txt | 6 +-\n builtin/branch.c | 5 +\n builtin/for-each-ref.c | 17 +-\n builtin/tag.c | 8 +-\n builtin/verify-tag.c | 1 +\n ref-filter.c | 296 ++++++++++++++++++++++-------\n ref-filter.h | 9 +-\n t/t3203-branch-output.sh | 14 ++\n t/t6300-for-each-ref.sh | 294 ++++++++++++++++++++++++++++\n t/t7004-tag.sh | 10 +\n t/t7030-verify-tag.sh | 8 +\n 14 files changed, 633 insertions(+), 74 deletions(-)\n\n\nbase-commit: 1197f1a46360d3ae96bd9c15908a3a6f8e562207\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-972%2Fadlternative%2Fref-filter-texconv-filters-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-972/adlternative/ref-filter-texconv-filters-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/972\n-- \ngitgitgadget\n"},{"id":"426482","messageId":"b699c1b7-5f4c-b52e-1d81-a569c4dc45bd@gmail.com","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/6] [GSOC][RFC] ref-filter: add %(raw:textconv) and %(raw:filters)","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-06-05T10:29:47Z","receivedAt":"2021-06-05T10:30:08Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"Hi,\n\nOn 05/06/21 16.13, ZheNing Hu via GitGitGadget wrote:\n> In order to let git cat-file --batch reuse ref-filter logic, This patch,\n> %(rest), %(raw:textconv), %(raw:filters) atoms and --rest= option are\n> added to ref-filter.\n> \n\nBetter say \"Add ... atoms and --rest= option to ref-filter, in \norder to let git cat-file reuse ref-filter logic.\"\n\n> * %(rest) int the format will be replaced by the in --rest=.\n> * the in --rest= can also be used as the for\n> %(raw:textconv) and %(raw:filters).\n\ns/int/in/\n\nDid you mean that %(rest) atom can also be used as for the latter?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"426489","messageId":"CAOLTT8TTbT3q_qNuhKeSoPr2gkLOupZxRSdMcJvHKbAS3bd5dw@mail.gmail.com","threadId":"55843","inReplyTo":"b699c1b7-5f4c-b52e-1d81-a569c4dc45bd@gmail.com","subject":"Re: [PATCH 0/6] [GSOC][RFC] ref-filter: add %(raw:textconv) and %(raw:filters)","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-05T13:19:07Z","receivedAt":"2021-06-05T13:19:27Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Bagas Sanjaya 于2021年6月5日周六 下午6:29写道：\n>\n> Hi,\n>\n> On 05/06/21 16.13, ZheNing Hu via GitGitGadget wrote:\n> > In order to let git cat-file --batch reuse ref-filter logic, This patch,\n> > %(rest), %(raw:textconv), %(raw:filters) atoms and --rest= option are\n> > added to ref-filter.\n> >\n>\n> Better say \"Add ... atoms and --rest= option to ref-filter, in\n> order to let git cat-file reuse ref-filter logic.\"\n>\n\nOK.\n\n> > * %(rest) int the format will be replaced by the in --rest=.\n> > * the in --rest= can also be used as the for\n> > %(raw:textconv) and %(raw:filters).\n>\n> s/int/in/\n>\n> Did you mean that %(rest) atom can also be used as for the latter?\n>\n\nNo. just the in `--rest=` will be treated as for\n%(raw:textconv)\nand %(raw:filters).\n\n> --\n> An old man doll... just what I always wanted! - Clara\n\nThanks.\n--\nZheNing Hu\n"},{"id":"426492","messageId":"CA+CkUQ9f8kN=S8dU_zt=-uG1pcK8cE9CuhJdqR9oMwcguZ9FLg@mail.gmail.com","threadId":"55843","inReplyTo":"ccdd18ad508824aa206a02c479229d0ede69522d.1622884415.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"Hariom verma","fromEmail":"hariom18599@gmail.com","sentAt":"2021-06-05T15:20:37Z","receivedAt":"2021-06-05T15:20:51Z","isPatch":true,"sender":{"key":"hariom18599@gmail.com","avatar":"https://avatars.githubusercontent.com/u/37576387?v=4"},"body":"Hi,\n\nOn Sat, Jun 5, 2021 at 2:43 PM ZheNing Hu via GitGitGadget\n wrote:\n>\n> From: ZheNing Hu \n>\n> --- a/builtin/branch.c\n> +++ b/builtin/branch.c\n> @@ -670,6 +674,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n> N_(\"print only branches of the object\"), parse_opt_object_name),\n> OPT_BOOL('i', \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n> OPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n> + OPT_STRING(0, \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n> OPT_END(),\n> };\n>\n\nAlthough it's not related to this patch. But I just noticed an unusual\nextra space(s) before the first argument of `OPT_STRING()`. (above the\nline you added)\n\n> --- a/builtin/for-each-ref.c\n> +++ b/builtin/for-each-ref.c\n> @@ -37,6 +37,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n>\n> OPT_GROUP(\"\"),\n> OPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only matched refs\")),\n> + OPT_STRING( 0 , \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n> OPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n> OPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n> OPT_REF_SORT(sorting_tail),\n\nHere too in `OPT_INTEGER()` and `OPT_INTEGER()`.\n\nAlso, I don't think these extra space(s) are intended. So you don't\nneed to imitate them.\n\n> --- a/builtin/tag.c\n> +++ b/builtin/tag.c\n> @@ -481,6 +486,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n> OPT_STRING( 0 , \"format\", &format.format, N_(\"format\"),\n> N_(\"format to use for the output\")),\n> OPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n> + OPT_STRING(0, \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n> OPT_BOOL('i', \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n> OPT_END()\n> };\n\nHere too in the first line.\n\nThanks,\nHariom\n"},{"id":"426522","messageId":"CAOLTT8RD1hmjiusDuf_O+UPBH0H9-2Y58LFd_UcvnZ2R5zL08w@mail.gmail.com","threadId":"55843","inReplyTo":"CA+CkUQ9f8kN=S8dU_zt=-uG1pcK8cE9CuhJdqR9oMwcguZ9FLg@mail.gmail.com","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-06T04:58:27Z","receivedAt":"2021-06-06T04:58:54Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Hi,\n\nHariom verma 于2021年6月5日周六 下午11:20写道：\n>\n> Hi,\n>\n> On Sat, Jun 5, 2021 at 2:43 PM ZheNing Hu via GitGitGadget\n> wrote:\n> >\n> > From: ZheNing Hu \n> >\n> > --- a/builtin/branch.c\n> > +++ b/builtin/branch.c\n> > @@ -670,6 +674,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n> > N_(\"print only branches of the object\"), parse_opt_object_name),\n> > OPT_BOOL('i', \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n> > OPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n> > + OPT_STRING(0, \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n> > OPT_END(),\n> > };\n> >\n>\n> Although it's not related to this patch. But I just noticed an unusual\n> extra space(s) before the first argument of `OPT_STRING()`. (above the\n> line you added)\n>\n\nYeah, I noticed it too.\n\n> > --- a/builtin/for-each-ref.c\n> > +++ b/builtin/for-each-ref.c\n> > @@ -37,6 +37,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> >\n> > OPT_GROUP(\"\"),\n> > OPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only matched refs\")),\n> > + OPT_STRING( 0 , \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n> > OPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n> > OPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n> > OPT_REF_SORT(sorting_tail),\n>\n> Here too in `OPT_INTEGER()` and `OPT_INTEGER()`.\n>\n> Also, I don't think these extra space(s) are intended. So you don't\n> need to imitate them.\n>\n\nMaybe... I find it's begin at c3428da8 (Make builtin-for-each-ref.c\nuse parse-opts.)\nThis is something from 2007. So It may be wrong to imitate its format.\nBut this format fix work may be can put in good-first-issue. :)\n\n> > --- a/builtin/tag.c\n> > +++ b/builtin/tag.c\n> > @@ -481,6 +486,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n> > OPT_STRING( 0 , \"format\", &format.format, N_(\"format\"),\n> > N_(\"format to use for the output\")),\n> > OPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n> > + OPT_STRING(0, \"rest\", &format.rest, N_(\"rest\"), N_(\"specify %(rest) contents\")),\n> > OPT_BOOL('i', \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n> > OPT_END()\n> > };\n>\n> Here too in the first line.\n>\n> Thanks,\n> Hariom\n\nThanks.\n--\nZheNing Hu\n"},{"id":"426557","messageId":"xmqq7dj6w7a6.fsf@gitster.g","threadId":"55843","inReplyTo":"ccdd18ad508824aa206a02c479229d0ede69522d.1622884415.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-07T05:52:33Z","receivedAt":"2021-06-07T05:52:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"ZheNing Hu via GitGitGadget\" writes:\n\n> From: ZheNing Hu \n>\n> In order to let \"cat-file --batch=%(rest)\" use the ref-filter\n> interface, add %(rest) atom for ref-filter and --rest option\n> for \"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n> `--rest` specify a string to replace %(rest) placeholders of\n> the --format option.\n\nI cannot think of a sane reason why we need to allow \"%(rest)\" in\nanithing but \"cat-file --batch\", where a natural source of %(rest)\nexists in its input stream (i.e. each input record begins with an\nobject name to be processed, and the rest of the record can become\n\"%(rest)\").\n\nThe \"cat-file --batch\" thing is much more understandable. You could\nfor example:\n\n git ls-files -s |\n sed -e 's/^[0-7]* \\([0-9a-f]*\\) [0-3]\t/\\1 /' |\n git cat-file --batch='%(objectname) %(objecttype) %(rest)'\n\nto massage output from \"ls-files -s\" like this\n\n 100644 c2f5fe385af1bbc161f6c010bdcf0048ab6671ed 0\t.cirrus.yml\n 100644 c592dda681fecfaa6bf64fb3f539eafaf4123ed8 0\t.clang-format\n 100644 f9d819623d832113014dd5d5366e8ee44ac9666a 0\t.editorconfig\n ...\n\ninto recods of \" \", and each output record will\nreplay the part from each corresponding input record.\n\nUnless for-each-ref family of commands read the list of refs that it\nshows from their standard input (they do not, and I do not think it\nmakes any sense to teach them to), there is no place to feed the\n\"rest\" information that is associated with each output record. The\nonly thing the commands taught about %(rest) by this patch can do is\nto parrot the same string into each and every output record. I am\nnot seeing what this new feature is attempting to give us.\n\nIf anything, I would imagine that it would be a very useful addition\nto teach the ref-filter machinery an ability to optionally error out\ndepending on the caller when the caller attempts to use certain\nplaceholder. Then, we can reject \"git branch --sort=rest\" sensibly,\ninstead of accepting \"git branch --sort=rest --rest=constant\", which\nis not technically wrong per-se, but smells like a total nonsense from\npractical usefulness's point of view.\n\n> -\t[--list] [...]\n> +\t[--list] [...] [--rest=]\n> 'git branch' [--track | --no-track] [-f] []\n> 'git branch' (--set-upstream-to= | -u ) []\n> 'git branch' --unset-upstream []\n> @@ -298,6 +298,10 @@ start-point is either a local or remote-tracking branch.\n> \tand the object it points at. The format is the same as\n> \tthat of linkgit:git-for-each-ref[1].\n> \n> +--rest=::\n> +\tIf given, the `%(rest)` placeholders in the `--format` option\n> +\twill be replaced.\n\nIf not given, what happens?\n"},{"id":"426558","messageId":"xmqq35tuw74h.fsf@gitster.g","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/6] [GSOC][RFC] ref-filter: add %(raw:textconv) and %(raw:filters)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-07T05:55:58Z","receivedAt":"2021-06-07T05:56:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"ZheNing Hu via GitGitGadget\" writes:\n\n> The current series is based on 0efed9435 ([GSOC] ref-filter: add %(raw)\n> atom)\n\nI do not have that commit object, but these six patches include the\ntwo commits that are %(raw) and %(raw:size), so I'll just discard\nthe old round that wasn't based on the atom-type stuff and queue\nthese six as a single series.\n\nAs I already said, I am not sure how %(rest) makes any sense outside\nthe context of \"cat-file --batch\"; I suspect it would make more sense\nto make it easier to arrange certain placeholders to error out when\nused in a context where they do not make sense (e.g. use of --rest\nin \"git branch --list\").\n\n"},{"id":"426599","messageId":"CAOLTT8S+5m+-XF-AcQi9t8njTvyDYzHt=BU+4OPcvTT27RP6dw@mail.gmail.com","threadId":"55843","inReplyTo":"xmqq7dj6w7a6.fsf@gitster.g","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-07T13:02:28Z","receivedAt":"2021-06-07T13:02:42Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年6月7日周一 下午1:52写道：\n>\n> \"ZheNing Hu via GitGitGadget\" writes:\n>\n> > From: ZheNing Hu \n> >\n> > In order to let \"cat-file --batch=%(rest)\" use the ref-filter\n> > interface, add %(rest) atom for ref-filter and --rest option\n> > for \"git for-each-ref\", \"git branch\", \"git tag\" and \"git verify-tag\".\n> > `--rest` specify a string to replace %(rest) placeholders of\n> > the --format option.\n>\n> I cannot think of a sane reason why we need to allow \"%(rest)\" in\n> anithing but \"cat-file --batch\", where a natural source of %(rest)\n> exists in its input stream (i.e. each input record begins with an\n> object name to be processed, and the rest of the record can become\n> \"%(rest)\").\n>\n\nFirst of all, although %(rest) is meaningless in ordinary circumstances,\nref-filter must learn %(rest), it is impossible for us to leave the parsing\nof %(rest) in cat-file.c alone.\n\nThen, `--rest` is a strategy that make %(rest) can use in `git for-each-ref`\nor `git branch -l`. As you said, it is just a boring placeholder used for string\nreplacement. We can make it output only empty content, If we really don’t\nneed `--rest`.\n\n> The \"cat-file --batch\" thing is much more understandable. You could\n> for example:\n>\n> git ls-files -s |\n> sed -e 's/^[0-7]* \\([0-9a-f]*\\) [0-3] /\\1 /' |\n> git cat-file --batch='%(objectname) %(objecttype) %(rest)'\n>\n\ns/[0-3] /[0-3]\\t/\n\n> to massage output from \"ls-files -s\" like this\n>\n> 100644 c2f5fe385af1bbc161f6c010bdcf0048ab6671ed 0 .cirrus.yml\n> 100644 c592dda681fecfaa6bf64fb3f539eafaf4123ed8 0 .clang-format\n> 100644 f9d819623d832113014dd5d5366e8ee44ac9666a 0 .editorconfig\n> ...\n>\n> into recods of \" \", and each output record will\n> replay the part from each corresponding input record.\n>\n\nYeah, the in the input will be treated as \"rest\".\n\n> Unless for-each-ref family of commands read the list of refs that it\n> shows from their standard input (they do not, and I do not think it\n> makes any sense to teach them to), there is no place to feed the\n> \"rest\" information that is associated with each output record. The\n> only thing the commands taught about %(rest) by this patch can do is\n> to parrot the same string into each and every output record. I am\n> not seeing what this new feature is attempting to give us.\n>\n\n\"parrot the same string\"? I think we should use an empty string here,\n\"parrot the same string\" more like what the \"git log --format\" family does.\n\n> If anything, I would imagine that it would be a very useful addition\n> to teach the ref-filter machinery an ability to optionally error out\n> depending on the caller when the caller attempts to use certain\n> placeholder. Then, we can reject \"git branch --sort=rest\" sensibly,\n> instead of accepting \"git branch --sort=rest --rest=constant\", which\n> is not technically wrong per-se, but smells like a total nonsense from\n> practical usefulness's point of view.\n>\n\nThis sounds like it might help `cat-file` to reject some useless atoms\nlike %(refname). So something like:\n\n$ git for-each-ref --format=\"%(objectname) %(objectsize)\"\n--refject-atoms=\"%(objectsize) %(objectname)\"\n\nwill fail.\n\n\"git for-each-ref\" family use hardcoded to reject %(rest).\nI can try to achieve this function.\n\n> > - [--list] [...]\n> > + [--list] [...] [--rest=]\n> > 'git branch' [--track | --no-track] [-f] []\n> > 'git branch' (--set-upstream-to= | -u ) []\n> > 'git branch' --unset-upstream []\n> > @@ -298,6 +298,10 @@ start-point is either a local or remote-tracking branch.\n> > and the object it points at. The format is the same as\n> > that of linkgit:git-for-each-ref[1].\n> >\n> > +--rest=::\n> > + If given, the `%(rest)` placeholders in the `--format` option\n> > + will be replaced.\n>\n> If not given, what happens?\n\nWill output an empty string.\n\nHope we can reach an agreement:\ndelete `--rest` and add `--reject-atoms`. ;-)\n\nThanks.\n--\nZheNing Hu\n"},{"id":"426601","messageId":"CAOLTT8SXj0bo2iqr4OAHHaSDCVbjAogAEvgFjupJBS-2N6dxHQ@mail.gmail.com","threadId":"55843","inReplyTo":"xmqq35tuw74h.fsf@gitster.g","subject":"Re: [PATCH 0/6] [GSOC][RFC] ref-filter: add %(raw:textconv) and %(raw:filters)","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-07T13:06:42Z","receivedAt":"2021-06-07T13:08:07Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年6月7日周一 下午1:56写道：\n>\n> \"ZheNing Hu via GitGitGadget\" writes:\n>\n> > The current series is based on 0efed9435 ([GSOC] ref-filter: add %(raw)\n> > atom)\n>\n> I do not have that commit object, but these six patches include the\n> two commits that are %(raw) and %(raw:size), so I'll just discard\n> the old round that wasn't based on the atom-type stuff and queue\n> these six as a single series.\n>\n\nWell, it is a commit that has not been sent to the mailing list. But it’s okay\nto treat the new 6 commits as a new patch series.\n\n> As I already said, I am not sure how %(rest) makes any sense outside\n> the context of \"cat-file --batch\"; I suspect it would make more sense\n> to make it easier to arrange certain placeholders to error out when\n> used in a context where they do not make sense (e.g. use of --rest\n> in \"git branch --list\").\n>\n\nI agree.\n\n--\nZheNing Hu\n"},{"id":"426603","messageId":"CAOLTT8QE7pafPmhnz-6=5zuyjg9n1FNbu_k6bA80jE1e5vYCmQ@mail.gmail.com","threadId":"55843","inReplyTo":"CAOLTT8S+5m+-XF-AcQi9t8njTvyDYzHt=BU+4OPcvTT27RP6dw@mail.gmail.com","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-07T13:18:38Z","receivedAt":"2021-06-07T13:19:05Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"ZheNing Hu 于2021年6月7日周一 下午9:02写道：\n>\n> Hope we can reach an agreement:\n> delete `--rest` and add `--reject-atoms`. ;-)\n>\n\nI forget one thing: %(raw:textconv) and %(raw:filters) can use\nthe value of \"--rest\" as their . But now if we want delete --rest,\nthey can not be used for \"for-each-ref\" family, Git will die with\n\"missing path for 'xxx'\".\n\n> Thanks.\n> --\n> ZheNing Hu\n"},{"id":"426700","messageId":"xmqqa6o1q6zz.fsf@gitster.g","threadId":"55843","inReplyTo":"0efed9435b59098f3ad928acd46c3c7e9f13677d.1622884415.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/6] [GSOC] ref-filter: add %(raw) atom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T05:07:28Z","receivedAt":"2021-06-08T05:07:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"ZheNing Hu via GitGitGadget\" writes:\n\n> static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b)\n> {\n> \tstruct atom_value *va, *vb;\n> @@ -2389,10 +2452,30 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n> \t} else if (s->sort_flags & REF_SORTING_VERSION) {\n> \t\tcmp = versioncmp(va->s, vb->s);\n> \t} else if (cmp_type == FIELD_STR) {\n> -\t\tint (*cmp_fn)(const char *, const char *);\n> -\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n> -\t\t\t? strcasecmp : strcmp;\n> -\t\tcmp = cmp_fn(va->s, vb->s);\n> +\t\tif (va->s_size == ATOM_VALUE_S_SIZE_INIT &&\n> +\t\t vb->s_size == ATOM_VALUE_S_SIZE_INIT) {\n> +\t\t\tint (*cmp_fn)(const char *, const char *);\n> +\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n> +\t\t\t\t? strcasecmp : strcmp;\n> +\t\t\tcmp = cmp_fn(va->s, vb->s);\n> +\t\t} else {\n> +\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n> +\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n> +\t\t\t\t? memcasecmp : memcmp;\n> +\t\t\tsize_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT ?\n> +\t\t\t\t\tstrlen(va->s) : va->s_size;\n> +\t\t\tsize_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT ?\n> +\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n\nThis breaks -Wdecl-after-stmt. A possible fix below.\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 46aec291de..648f9cabff 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2459,13 +2459,13 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n \t\t\t\t? strcasecmp : strcmp;\n \t\t\tcmp = cmp_fn(va->s, vb->s);\n \t\t} else {\n-\t\t\tint (*cmp_fn)(const void *, const void *, size_t);\n-\t\t\tcmp_fn = s->sort_flags & REF_SORTING_ICASE\n+\t\t\tsize_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT\n+\t\t\t\t\t? strlen(va->s) : va->s_size;\n+\t\t\tsize_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT\n+\t\t\t\t\t? strlen(vb->s) : vb->s_size;\n+\t\t\tint (*cmp_fn)(const void *, const void *, size_t) =\n+\t\t\t\ts->sort_flags & REF_SORTING_ICASE\n \t\t\t\t? memcasecmp : memcmp;\n-\t\t\tsize_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT ?\n-\t\t\t\t\tstrlen(va->s) : va->s_size;\n-\t\t\tsize_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT ?\n-\t\t\t\t\tstrlen(vb->s) : vb->s_size;\n \n \t\t\tcmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n \t\t\t\t a_size : b_size);\n"},{"id":"426710","messageId":"CAOLTT8QPMVueHMFCYP6YJ9_ODsKFxk2gyB1dO5ak=UFX-8Cm-A@mail.gmail.com","threadId":"55843","inReplyTo":"xmqqa6o1q6zz.fsf@gitster.g","subject":"Re: [PATCH 2/6] [GSOC] ref-filter: add %(raw) atom","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-08T06:10:48Z","receivedAt":"2021-06-08T06:11:02Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年6月8日周二 下午1:07写道：\n>\n> This breaks -Wdecl-after-stmt. A possible fix below.\n>\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 46aec291de..648f9cabff 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2459,13 +2459,13 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru\n> ? strcasecmp : strcmp;\n> cmp = cmp_fn(va->s, vb->s);\n> } else {\n> - int (*cmp_fn)(const void *, const void *, size_t);\n> - cmp_fn = s->sort_flags & REF_SORTING_ICASE\n> + size_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT\n> + ? strlen(va->s) : va->s_size;\n> + size_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT\n> + ? strlen(vb->s) : vb->s_size;\n> + int (*cmp_fn)(const void *, const void *, size_t) =\n> + s->sort_flags & REF_SORTING_ICASE\n> ? memcasecmp : memcmp;\n> - size_t a_size = va->s_size == ATOM_VALUE_S_SIZE_INIT ?\n> - strlen(va->s) : va->s_size;\n> - size_t b_size = vb->s_size == ATOM_VALUE_S_SIZE_INIT ?\n> - strlen(vb->s) : vb->s_size;\n>\n> cmp = cmp_fn(va->s, vb->s, b_size > a_size ?\n> a_size : b_size);\n\nYou are right.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"426712","messageId":"CAOLTT8TSue=Cx8xos20vnGSi3oOd8+=jTfTw2h82gXmmd4KyLg@mail.gmail.com","threadId":"55843","inReplyTo":"CAOLTT8QE7pafPmhnz-6=5zuyjg9n1FNbu_k6bA80jE1e5vYCmQ@mail.gmail.com","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-08T06:16:11Z","receivedAt":"2021-06-08T06:17:24Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"ZheNing Hu 于2021年6月7日周一 下午9:18写道：\n>\n> ZheNing Hu 于2021年6月7日周一 下午9:02写道：\n> >\n> > Hope we can reach an agreement:\n> > delete `--rest` and add `--reject-atoms`. ;-)\n> >\n>\n> I forget one thing: %(raw:textconv) and %(raw:filters) can use\n> the value of \"--rest\" as their . But now if we want delete --rest,\n> they can not be used for \"for-each-ref\" family, Git will die with\n> \"missing path for 'xxx'\".\n>\n\nIf we actually delete \"--rest\", we will have no way to test %(raw:textconv)\nand %(raw:filters)... So now I think we can keep --rest (or use\nanother name --path)\nand let \"git for-each-ref\" family reject %(rest) by default.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"426716","messageId":"xmqqtum8q2lm.fsf@gitster.g","threadId":"55843","inReplyTo":"pull.972.git.1622884415.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/6] [GSOC][RFC] ref-filter: add %(raw:textconv) and %(raw:filters)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T06:42:29Z","receivedAt":"2021-06-08T06:42:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"ZheNing Hu via GitGitGadget\" writes:\n\n> ZheNing Hu (6):\n> [GSOC] ref-filter: add obj-type check in grab contents\n> [GSOC] ref-filter: add %(raw) atom\n> [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n> [GSOC] ref-filter: add %(rest) atom and --rest option\n> [GSOC] ref-filter: teach grab_sub_body_contents() return value and err\n> [GSOC] ref-filter: add %(raw:textconv) and %(raw:filters)\n\nI haven't gotten around looking at anything after the %(rest) one,\nbut \n\nhttps://github.com/git/git/runs/2770688471?check_suite_focus=true\n\nseems to tell us that there is \"size_t *\" vs \"ulong *\" type\nconfusion, possibly around the textconv thing.\n\n\n"},{"id":"426720","messageId":"xmqqo8cgq28e.fsf@gitster.g","threadId":"55843","inReplyTo":"CAOLTT8S+5m+-XF-AcQi9t8njTvyDYzHt=BU+4OPcvTT27RP6dw@mail.gmail.com","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T06:50:25Z","receivedAt":"2021-06-08T06:50:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"ZheNing Hu writes:\n\n> First of all, although %(rest) is meaningless in ordinary circumstances,\n> ref-filter must learn %(rest), it is impossible for us to leave the parsing\n> of %(rest) in cat-file.c alone.\n\nOh, there is no question about that.\n\n> Then, `--rest` is a strategy that make %(rest) can use in `git for-each-ref`\n> or `git branch -l`.\n\nIf there is no need to expose %(rest) to the users who write\n--format for these two commands, it would be much better to detect\nattempted use of %(rest) and error out.\n\n> This sounds like it might help `cat-file` to reject some useless atoms\n> like %(refname).\n\nYes.\n\n> So something like:\n>\n> $ git for-each-ref --format=\"%(objectname) %(objectsize)\"\n> --refject-atoms=\"%(objectsize) %(objectname)\"\n>\n> will fail.\n\nI don't understand. Why do you even need to add --reject? Why\nwould any user would want to use it with for-each-ref?\n\nWithout any end-user input, %(rest) for for-each-ref would not make\nsense, and %(refname) for cat-file --batch would not make sense, I\nwould imagine, so there is no need to be able to tell --reject=rest\nto for-each-ref. It is not like giving --no-reject=rest to for-each-ref\nand make it interpolate to an empty string is a useful feature anyway,\nso I do not see a need for such an option.\n"},{"id":"426722","messageId":"xmqqk0n4q1t6.fsf@gitster.g","threadId":"55843","inReplyTo":"CAOLTT8TSue=Cx8xos20vnGSi3oOd8+=jTfTw2h82gXmmd4KyLg@mail.gmail.com","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T06:59:33Z","receivedAt":"2021-06-08T06:59:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"ZheNing Hu writes:\n\n> ZheNing Hu 于2021年6月7日周一 下午9:18写道：\n>>\n>> ZheNing Hu 于2021年6月7日周一 下午9:02写道：\n>> >\n>> > Hope we can reach an agreement:\n>> > delete `--rest` and add `--reject-atoms`. ;-)\n>> >\n>>\n>> I forget one thing: %(raw:textconv) and %(raw:filters) can use\n>> the value of \"--rest\" as their . But now if we want delete --rest,\n>> they can not be used for \"for-each-ref\" family, Git will die with\n>> \"missing path for 'xxx'\".\n>>\n>\n> If we actually delete \"--rest\", we will have no way to test %(raw:textconv)\n> and %(raw:filters)... So now I think we can keep --rest (or use\n> another name --path)\n> and let \"git for-each-ref\" family reject %(rest) by default.\n\nI didn't read beyond the %(rest) thing, but do we even need\n%(raw:textconv) to begin with? It is totally useless in the context\nof for-each-ref because textconv by its nature is tied to attributes\nthat by definition needs a blob that is sitting at a path, but the\nobjects for-each-ref and friends visit are mostly commits and tags,\nand even for refs that point at a blob, there isn't any \"path\"\ninformation to pull attribute for.\n\nIs that what you want to add to give \"cat-file --batch\"? Even in\nthe context of \"cat-file --batch\", you can throw an object name for\na blob to the command, but there is no path for the blob (a blob can\nappear at different places in different trees---think \"rename), so I\nam not sure what benefit you are trying to derive from it.\n\nThanks.\n\n\n"},{"id":"426765","messageId":"CAOLTT8SfXq9k1EHOx5a+26Hm5Xk8NerREUb5N6fOcAOv0GOZsQ@mail.gmail.com","threadId":"55843","inReplyTo":"xmqqo8cgq28e.fsf@gitster.g","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-08T12:32:26Z","receivedAt":"2021-06-08T12:32:40Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年6月8日周二 下午2:50写道：\n>\n> ZheNing Hu writes:\n>\n> > First of all, although %(rest) is meaningless in ordinary circumstances,\n> > ref-filter must learn %(rest), it is impossible for us to leave the parsing\n> > of %(rest) in cat-file.c alone.\n>\n> Oh, there is no question about that.\n>\n\nOK.\n\n>\n> I don't understand. Why do you even need to add --reject? Why\n> would any user would want to use it with for-each-ref?\n>\n> Without any end-user input, %(rest) for for-each-ref would not make\n> sense, and %(refname) for cat-file --batch would not make sense, I\n> would imagine, so there is no need to be able to tell --reject=rest\n> to for-each-ref. It is not like giving --no-reject=rest to for-each-ref\n> and make it interpolate to an empty string is a useful feature anyway,\n> so I do not see a need for such an option.\n\nYeah... I might have thought it complicated before. We can reject %(rest) in\nverify_ref_format() easily. It's like we refused -- before. :)\n\nThanks.\n--\nZheNing Hu\n"},{"id":"426767","messageId":"CAOLTT8Tve8woTGsgWMpYb=adesMx8UzSVpCnK4xPV9-6agzQmA@mail.gmail.com","threadId":"55843","inReplyTo":"xmqqk0n4q1t6.fsf@gitster.g","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-08T12:39:00Z","receivedAt":"2021-06-08T12:39:28Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年6月8日周二 下午2:59写道：\n>\n> >\n> > If we actually delete \"--rest\", we will have no way to test %(raw:textconv)\n> > and %(raw:filters)... So now I think we can keep --rest (or use\n> > another name --path)\n> > and let \"git for-each-ref\" family reject %(rest) by default.\n>\n> I didn't read beyond the %(rest) thing, but do we even need\n> %(raw:textconv) to begin with? It is totally useless in the context\n> of for-each-ref because textconv by its nature is tied to attributes\n> that by definition needs a blob that is sitting at a path, but the\n> objects for-each-ref and friends visit are mostly commits and tags,\n> and even for refs that point at a blob, there isn't any \"path\"\n> information to pull attribute for.\n>\n\nAfter thinking about your words, now I think maybe we can leave\n%(raw:textconv) and %(raw:filter) after cat-file --batch start using\nref-filter logic, so that we can provide them with suitable tests,\nand we don't need `--rest` anymore.\n\n> Is that what you want to add to give \"cat-file --batch\"? Even in\n> the context of \"cat-file --batch\", you can throw an object name for\n> a blob to the command, but there is no path for the blob (a blob can\n> appear at different places in different trees---think \"rename), so I\n> am not sure what benefit you are trying to derive from it.\n>\n\nSo I will remove the last two commits.\n\n> Thanks.\n>\n>\n\nThanks.\n--\nZheNing Hu\n"},{"id":"426769","messageId":"CAOLTT8ThnwfrbMxWaFptK=6Eh7EE-qavRWxyq3FWy3vVCVva_g@mail.gmail.com","threadId":"55843","inReplyTo":"xmqqtum8q2lm.fsf@gitster.g","subject":"Re: [PATCH 0/6] [GSOC][RFC] ref-filter: add %(raw:textconv) and %(raw:filters)","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-08T12:52:23Z","receivedAt":"2021-06-08T12:53:51Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年6月8日周二 下午2:42写道：\n>\n> \"ZheNing Hu via GitGitGadget\" writes:\n>\n> > ZheNing Hu (6):\n> > [GSOC] ref-filter: add obj-type check in grab contents\n> > [GSOC] ref-filter: add %(raw) atom\n> > [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n> > [GSOC] ref-filter: add %(rest) atom and --rest option\n> > [GSOC] ref-filter: teach grab_sub_body_contents() return value and err\n> > [GSOC] ref-filter: add %(raw:textconv) and %(raw:filters)\n>\n> I haven't gotten around looking at anything after the %(rest) one,\n> but\n>\n> https://github.com/git/git/runs/2770688471?check_suite_focus=true\n>\n> seems to tell us that there is \"size_t *\" vs \"ulong *\" type\n> confusion, possibly around the textconv thing.\n>\n>\n\nok, I will change it when we use %(raw:textconv) next time.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"426860","messageId":"xmqq4ke7jzee.fsf@gitster.g","threadId":"55843","inReplyTo":"xmqqk0n4q1t6.fsf@gitster.g","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-09T07:00:25Z","receivedAt":"2021-06-09T07:00:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Is that what you want to add to give \"cat-file --batch\"? Even in\n> the context of \"cat-file --batch\", you can throw an object name for\n> a blob to the command, but there is no path for the blob (a blob can\n> appear at different places in different trees---think \"rename), so I\n> am not sure what benefit you are trying to derive from it.\n\nI think I kind-of see what is going on here. There is\n\n git cat-file blob --textconv --path=\"$path\" \"$blob_object_name\"\n\nthat allows a blob to be fed to the command, pretend as if it\nappears at $path in a tree object and grab attribute for it, and\nshow the blob contents converted using the textconv filter. If we\nwere to mimic it by extending the format based substitutions, a\ndesign consistent with the behaviour is to teach --format=%(raw)\nto show the contents after applying the textconv filter instead of\nthe raw blob contents.\n\nAnd there is a corresponding\n\n git cat-file --batch --textconv\n\nThe \"--path=$path\" parameter is omitted when using --batch, as each\nobject would sit at different path in the tree (so the input stream\nwould be given as a run of \" \" to give each item its own\npath).\n\nSo to answer my question in the previous message, yes, this is an\nattempt to support the \"cat-file --textconv\". So in the context of\nthat command, something may need to be added. But I do not think it\nmakes any sense to expose that to for-each-ref and friends, even if\nwe were to share the internal machinery (after all, sharing of the\ninternal machinery is a mere means to an end that is to make it\neasier to give the same syntax and same behaviour to end users and\nis not a goal itself; \"because we use the same machinery, the users\nhave to tolerate that irrelevant %(atoms) are accepted by the parser\"\nis not making a good excuse for a sloppy implementation).\n\nHaving said all that, I somehow doubt that the \"--batch=\"\nwas designed to interact sensibly with the \"--textconv\" option.\nbuiltin/cat-file.c::expand_atom() does not know anything at all that\nthe data could be modified from the raw contents of the blob, so\n--batch=\"%(contents) %(size)\" --textconv, if existed, may show the\nconveted contents with size of blob before conversion, or something\nincoherent like that. And if your rewrite using the shared internal\nmachinery results in a more coherent behaviour, that would be\nexcellent. For example, we could imagine that the machinery, when\ntextconv (or filter) is in use, would first grab the blob contents\nand run the requested conversion, and then work solely on that\nconveted contents when computing what to fill with %(raw:size) and\nother blob-related atoms.\n\nThanks.\n\n\n"},{"id":"426877","messageId":"CAOLTT8QQPfYdE3GzNt9BEPvLvPdJg8zS6-Dn3L26BdQ6M7Tk6g@mail.gmail.com","threadId":"55843","inReplyTo":"xmqq4ke7jzee.fsf@gitster.g","subject":"Re: [PATCH 4/6] [GSOC] ref-filter: add %(rest) atom and --rest option","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-09T12:47:09Z","receivedAt":"2021-06-09T12:47:40Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano 于2021年6月9日周三 下午3:00写道：\n>\n> I think I kind-of see what is going on here. There is\n>\n> git cat-file blob --textconv --path=\"$path\" \"$blob_object_name\"\n>\n> that allows a blob to be fed to the command, pretend as if it\n> appears at $path in a tree object and grab attribute for it, and\n> show the blob contents converted using the textconv filter. If we\n> were to mimic it by extending the format based substitutions, a\n> design consistent with the behaviour is to teach --format=%(raw)\n> to show the contents after applying the textconv filter instead of\n> the raw blob contents.\n>\n\nYes, this is exactly what cat-file --textconv does.\n\n> And there is a corresponding\n>\n> git cat-file --batch --textconv\n>\n> The \"--path=$path\" parameter is omitted when using --batch, as each\n> object would sit at different path in the tree (so the input stream\n> would be given as a run of \" \" to give each item its own\n> path).\n>\n\nJust like let --batch omitted --path, --rest is meaningless for \"for-ech-ref\".\n\n> So to answer my question in the previous message, yes, this is an\n> attempt to support the \"cat-file --textconv\". So in the context of\n> that command, something may need to be added. But I do not think it\n> makes any sense to expose that to for-each-ref and friends, even if\n> we were to share the internal machinery (after all, sharing of the\n> internal machinery is a mere means to an end that is to make it\n> easier to give the same syntax and same behaviour to end users and\n> is not a goal itself; \"because we use the same machinery, the users\n> have to tolerate that irrelevant %(atoms) are accepted by the parser\"\n> is not making a good excuse for a sloppy implementation).\n>\n\nBecause \"git cat-file --batch\" will only print the contents of the object once,\nso when implements the function of textconv/filters in ref-filter,\nwe should really consider whether we should let something like\n\"%(raw) %(raw) %(raw) %(raw:size)\" all pass the conversion of textconv/filters.\nIf it is my previous %(raw:textconv) or %(raw:filter), they can only print\nthe converted content separately, and we need:\n\n$ git for-ecah-ref --format=\"%(raw:filters) %(raw:filters)\n%(raw:filters) %(raw:filters:size)\"\n\nAs you said it might be too complicated for the user....\n\n> Having said all that, I somehow doubt that the \"--batch=\"\n> was designed to interact sensibly with the \"--textconv\" option.\n> builtin/cat-file.c::expand_atom() does not know anything at all that\n> the data could be modified from the raw contents of the blob, so\n> --batch=\"%(contents) %(size)\" --textconv, if existed, may show the\n> conveted contents with size of blob before conversion, or something\n> incoherent like that. And if your rewrite using the shared internal\n> machinery results in a more coherent behaviour, that would be\n> excellent. For example, we could imagine that the machinery, when\n> textconv (or filter) is in use, would first grab the blob contents\n> and run the requested conversion, and then work solely on that\n> conveted contents when computing what to fill with %(raw:size) and\n> other blob-related atoms.\n>\n\nAnd with your --textconv/--filters, we only need:\n\n$ git for-ecah-ref --format=\"%(raw) %(raw) %(raw) %(raw:size)\" --filters\n\nThis will be more concise for users. I will try to build --filters, --textconv\nfor ref-filter . But as stated in the previous reply, it needs to be placed\nafter the transplant of cat-file --batch (Because --path is useless for\nfor-each-ref, and at the same time we need proper testing for\n--filter/--textconv.)\n\nThanks, your reply is very reasonable,\n--\nZheNing Hu\n"}]}