{"thread":{"id":"48484","subject":"[PATCH RFC] ref-filter: start using oid_object_info","startedAt":"2018-05-14T09:59:09Z","lastAt":"2018-05-15T03:54:02Z","messageCount":3,"participants":["Olga Telezhnaya","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"347576","messageId":"010201635e16d282-89bdd0cd-df10-4509-bad7-fd49fd80ff2b-000000@eu-west-1.amazonses.com","threadId":"48484","inReplyTo":null,"subject":"[PATCH RFC] ref-filter: start using oid_object_info","fromName":"Olga Telezhnaya","fromEmail":"olyatelezhnaya@gmail.com","sentAt":"2018-05-14T09:59:04Z","receivedAt":"2018-05-14T09:59:09Z","isPatch":true,"sender":{"key":"olyatelezhnaya@gmail.com","avatar":"https://avatars.githubusercontent.com/u/11246099?v=4"},"body":"Start using oid_object_info_extended(). So, if info from this function\nis enough, we do not need to get and parse whole object (as it was before).\nIt's good for 3 reasons:\n1. Some Git commands potentially will work faster.\n2. It's much easier to add support for objectsize:disk and deltabase.\n   (I have plans to add this support further)\n3. It's easier to move formatting from cat-file command to this logic\n   (It pretends to be unified formatting logic in the end)\n\nSigned-off-by: Olga Telezhnaia <olyatelezhnaya@gmail.com>\n---\n ref-filter.c | 34 +++++++++++++++++++++++++++++++---\n ref-filter.h | 21 +++++++++++++++++++++\n 2 files changed, 52 insertions(+), 3 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 39e2744c949bb..7c4693ed084bb 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -100,6 +100,7 @@ static struct used_atom {\n \t} u;\n } *used_atom;\n static int used_atom_cnt, need_tagged, need_symref;\n+static struct expand_data format_data;\n \n /*\n  * Expand string, append it to strbuf *sb, then return error code ret.\n@@ -267,6 +268,22 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato\n \treturn 0;\n }\n \n+static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\t  const char *arg, struct strbuf *unused_err)\n+{\n+\tformat_data.use_data = 1;\n+\tformat_data.info.typep = &format_data.type;\n+\treturn 0;\n+}\n+\n+static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom,\n+\t\t\t\t  const char *arg, struct strbuf *unused_err)\n+{\n+\tformat_data.use_data = 1;\n+\tformat_data.info.sizep = &format_data.size;\n+\treturn 0;\n+}\n+\n static int objectname_atom_parser(const struct ref_format *format, struct used_atom *atom,\n \t\t\t\t  const char *arg, struct strbuf *err)\n {\n@@ -383,9 +400,9 @@ static struct {\n \tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n \t\t      const char *arg, struct strbuf *err);\n } valid_atom[] = {\n-\t{ \"refname\" , FIELD_STR, refname_atom_parser },\n-\t{ \"objecttype\" },\n-\t{ \"objectsize\", FIELD_ULONG },\n+\t{ \"refname\", FIELD_STR, refname_atom_parser },\n+\t{ \"objecttype\", FIELD_STR, objecttype_atom_parser },\n+\t{ \"objectsize\", FIELD_ULONG, objectsize_atom_parser },\n \t{ \"objectname\", FIELD_STR, objectname_atom_parser },\n \t{ \"tree\" },\n \t{ \"parent\" },\n@@ -1536,6 +1553,13 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\tcontinue;\n \t\t} else if (!deref && grab_objectname(name, &ref->objectname, v, atom)) {\n \t\t\tcontinue;\n+\t\t} else if (!deref && !strcmp(name, \"objecttype\")) {\n+\t\t\tv->s = type_name(format_data.type);\n+\t\t\tcontinue;\n+\t\t} else if (!deref && !strcmp(name, \"objectsize\")) {\n+\t\t\tv->value = format_data.size;\n+\t\t\tv->s = xstrfmt(\"%lu\", format_data.size);\n+\t\t\tcontinue;\n \t\t} else if (!strcmp(name, \"HEAD\")) {\n \t\t\tif (atom->u.head && !strcmp(ref->refname, atom->u.head))\n \t\t\t\tv->s = \"*\";\n@@ -2226,6 +2250,10 @@ int format_ref_array_item(struct ref_array_item *info,\n {\n \tconst char *cp, *sp, *ep;\n \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n+\tformat_data.oid = info->objectname;\n+\tif (format_data.use_data && oid_object_info_extended(&format_data.oid, &format_data.info,\n+\t\t\t\t\t\t\t     OBJECT_INFO_LOOKUP_REPLACE) < 0)\n+\t\treturn strbuf_addf_ret(error_buf, -1, \"format: could not get object info\");\n \n \tstate.quote_style = format->quote_style;\n \tpush_stack_element(&state.stack);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 85c8ebc3b904e..857e8c5318a8f 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -85,6 +85,27 @@ struct ref_format {\n \tint need_color_reset_at_eol;\n };\n \n+struct expand_data {\n+\tstruct object_id oid;\n+\tenum object_type type;\n+\tunsigned long size;\n+\toff_t disk_size;\n+\tstruct object_id delta_base_oid;\n+\n+\t/*\n+\t * This object_info is set up to be passed to oid_object_info_extended.\n+\t * It will point to the data elements above, so you can retrieve\n+\t * the response from there.\n+\t */\n+\tstruct object_info info;\n+\n+\t/*\n+\t * This flag will be true if the requested format and options\n+\t * require us to call oid_object_info_extended.\n+\t */\n+\tunsigned use_data : 1;\n+};\n+\n #define REF_FORMAT_INIT { NULL, 0, -1 }\n \n /*  Macros for checking --merged and --no-merged options */\n\n--\nhttps://github.com/git/git/pull/493\n"},{"id":"347678","messageId":"xmqqd0xxa9dp.fsf@gitster-ct.c.googlers.com","threadId":"48484","inReplyTo":"010201635e16d282-89bdd0cd-df10-4509-bad7-fd49fd80ff2b-000000@eu-west-1.amazonses.com","subject":"Re: [PATCH RFC] ref-filter: start using oid_object_info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-15T03:24:50Z","receivedAt":"2018-05-15T03:24:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Olga Telezhnaya <olyatelezhnaya@gmail.com> writes:\n\n> Start using oid_object_info_extended(). So, if info from this function\n> is enough, we do not need to get and parse whole object (as it was before).\n> It's good for 3 reasons:\n> 1. Some Git commands potentially will work faster.\n> 2. It's much easier to add support for objectsize:disk and deltabase.\n>    (I have plans to add this support further)\n> 3. It's easier to move formatting from cat-file command to this logic\n>    (It pretends to be unified formatting logic in the end)\n>\n> Signed-off-by: Olga Telezhnaia <olyatelezhnaya@gmail.com>\n> ---\n>  ref-filter.c | 34 +++++++++++++++++++++++++++++++---\n>  ref-filter.h | 21 +++++++++++++++++++++\n>  2 files changed, 52 insertions(+), 3 deletions(-)\n>...\n> @@ -383,9 +400,9 @@ static struct {\n>  \tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n>  \t\t      const char *arg, struct strbuf *err);\n>  } valid_atom[] = {\n> -\t{ \"refname\" , FIELD_STR, refname_atom_parser },\n> -\t{ \"objecttype\" },\n> -\t{ \"objectsize\", FIELD_ULONG },\n> +\t{ \"refname\", FIELD_STR, refname_atom_parser },\n> +\t{ \"objecttype\", FIELD_STR, objecttype_atom_parser },\n> +\t{ \"objectsize\", FIELD_ULONG, objectsize_atom_parser },\n>  \t{ \"objectname\", FIELD_STR, objectname_atom_parser },\n>  \t{ \"tree\" },\n>  \t{ \"parent\" },\n\nHmph, so this patch does not teach us to interpolate any new %(field-type)\nbut changes the way %(objecttype) and %(objectsize) are computed.\n\n> @@ -1536,6 +1553,13 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n>  \t\t\tcontinue;\n>  \t\t} else if (!deref && grab_objectname(name, &ref->objectname, v, atom)) {\n>  \t\t\tcontinue;\n> +\t\t} else if (!deref && !strcmp(name, \"objecttype\")) {\n> +\t\t\tv->s = type_name(format_data.type);\n> +\t\t\tcontinue;\n> +\t\t} else if (!deref && !strcmp(name, \"objectsize\")) {\n> +\t\t\tv->value = format_data.size;\n> +\t\t\tv->s = xstrfmt(\"%lu\", format_data.size);\n> +\t\t\tcontinue;\n>  \t\t} else if (!strcmp(name, \"HEAD\")) {\n>  \t\t\tif (atom->u.head && !strcmp(ref->refname, atom->u.head))\n>  \t\t\t\tv->s = \"*\";\n\nBecause this addition is made to the early \"Fill in specials first\"\nloop of the populate_value() function, we may be able to satisfy\nsome requests early without calling get_object() which then calls\nparse_object().\n\n> @@ -2226,6 +2250,10 @@ int format_ref_array_item(struct ref_array_item *info,\n>  {\n>  \tconst char *cp, *sp, *ep;\n>  \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n> +\tformat_data.oid = info->objectname;\n> +\tif (format_data.use_data && oid_object_info_extended(&format_data.oid, &format_data.info,\n\nStyle: fold the line after \" &&\".\n\nAnd this checks the .use_data field to see if these fields whose\nvalue could be computed by a call to oid_object_info_extended()\nwithout calling parse_object().  If there is one, we call it;\notherwise we don't.\n\nSo there are three possible cases:\n\n - The request does not ask for these fields that can be filled from\n   \"format_data\" (by the way, that is a horrible name---all the data\n   in this codepath is for formatting, and in that sense the\n   variable is not named after its most significant trait, which is\n   that it is to grab data needed for formatting via a call to\n   a function in the object_info() family.  Perhaps object_info_data\n   or oi_data for brevity).  We do not call object_info() and the\n   performance characteristic of the code stays as before.\n\n - The request asks for these fields that are helped by\n   \"object-info\" and no other fields.  We make a call to\n   \"object-info\", instead of parse_object(), which hopefully is more\n   efficient (we need to measure, if we are selling this as an\n   optimization).\n\n - The request asks for both.  We end up calling object-info and\n   also parse_object(), so presumably there is degradation of\n   performance.\n\nIn the third case, after v->s and v->value are filled by the new\ncode that copies from format_data, grab_values() will again fill\nobjecttype/objectsize by overwriting v->s field.  Doesn't this cause\nmemory leaks?  type_name() returns a constant string that does not\nleak, but your objectsize seems to use xstrfmt(), so...\n\nI think it was OK before this patch as grab_common_values() was the\nonly place that did v->s = xstrfmt() for the field, but now the code\nwith this patch can do the same assignment from two places, we would\nneed to be a bit more careful about memory ownership?\n\n"},{"id":"347679","messageId":"xmqq8t8la81a.fsf@gitster-ct.c.googlers.com","threadId":"48484","inReplyTo":"xmqqd0xxa9dp.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH RFC] ref-filter: start using oid_object_info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-15T03:53:53Z","receivedAt":"2018-05-15T03:54:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Olga Telezhnaya <olyatelezhnaya@gmail.com> writes:\n>\n>> Start using oid_object_info_extended(). So, if info from this function\n>> is enough, we do not need to get and parse whole object (as it was before).\n>> It's good for 3 reasons:\n>> 1. Some Git commands potentially will work faster.\n>> 2. It's much easier to add support for objectsize:disk and deltabase.\n>>    (I have plans to add this support further)\n>> 3. It's easier to move formatting from cat-file command to this logic\n>>    (It pretends to be unified formatting logic in the end)\n>>\n>> Signed-off-by: Olga Telezhnaia <olyatelezhnaya@gmail.com>\n>> ---\n>>  ref-filter.c | 34 +++++++++++++++++++++++++++++++---\n>>  ref-filter.h | 21 +++++++++++++++++++++\n>>  2 files changed, 52 insertions(+), 3 deletions(-)\n>>...\n>> @@ -383,9 +400,9 @@ static struct {\n>>  \tint (*parser)(const struct ref_format *format, struct used_atom *atom,\n>>  \t\t      const char *arg, struct strbuf *err);\n>>  } valid_atom[] = {\n>> -\t{ \"refname\" , FIELD_STR, refname_atom_parser },\n>> -\t{ \"objecttype\" },\n>> -\t{ \"objectsize\", FIELD_ULONG },\n>> +\t{ \"refname\", FIELD_STR, refname_atom_parser },\n>> +\t{ \"objecttype\", FIELD_STR, objecttype_atom_parser },\n>> +\t{ \"objectsize\", FIELD_ULONG, objectsize_atom_parser },\n>>  \t{ \"objectname\", FIELD_STR, objectname_atom_parser },\n>>  \t{ \"tree\" },\n>>  \t{ \"parent\" },\n>\n> Hmph, so this patch does not teach us to interpolate any new %(field-type)\n> but changes the way %(objecttype) and %(objectsize) are computed.\n>\n>> @@ -1536,6 +1553,13 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n>>  \t\t\tcontinue;\n>>  \t\t} else if (!deref && grab_objectname(name, &ref->objectname, v, atom)) {\n>>  \t\t\tcontinue;\n>> +\t\t} else if (!deref && !strcmp(name, \"objecttype\")) {\n>> +\t\t\tv->s = type_name(format_data.type);\n>> +\t\t\tcontinue;\n>> +\t\t} else if (!deref && !strcmp(name, \"objectsize\")) {\n>> +\t\t\tv->value = format_data.size;\n>> +\t\t\tv->s = xstrfmt(\"%lu\", format_data.size);\n>> +\t\t\tcontinue;\n>>  \t\t} else if (!strcmp(name, \"HEAD\")) {\n>>  \t\t\tif (atom->u.head && !strcmp(ref->refname, atom->u.head))\n>>  \t\t\t\tv->s = \"*\";\n>\n> Because this addition is made to the early \"Fill in specials first\"\n> loop of the populate_value() function, we may be able to satisfy\n> some requests early without calling get_object() which then calls\n> parse_object().\n>\n>> @@ -2226,6 +2250,10 @@ int format_ref_array_item(struct ref_array_item *info,\n>>  {\n>>  \tconst char *cp, *sp, *ep;\n>>  \tstruct ref_formatting_state state = REF_FORMATTING_STATE_INIT;\n>> +\tformat_data.oid = info->objectname;\n>> +\tif (format_data.use_data && oid_object_info_extended(&format_data.oid, &format_data.info,\n>\n> Style: fold the line after \" &&\".\n>\n> And this checks the .use_data field to see if these fields whose\n> value could be computed by a call to oid_object_info_extended()\n> without calling parse_object().  If there is one, we call it;\n> otherwise we don't.\n>\n> So there are three possible cases:\n>\n>  - The request does not ask for these fields that can be filled from\n>    \"format_data\" (by the way, that is a horrible name---all the data\n>    in this codepath is for formatting, and in that sense the\n>    variable is not named after its most significant trait, which is\n>    that it is to grab data needed for formatting via a call to\n>    a function in the object_info() family.  Perhaps object_info_data\n>    or oi_data for brevity).  We do not call object_info() and the\n>    performance characteristic of the code stays as before.\n>\n>  - The request asks for these fields that are helped by\n>    \"object-info\" and no other fields.  We make a call to\n>    \"object-info\", instead of parse_object(), which hopefully is more\n>    efficient (we need to measure, if we are selling this as an\n>    optimization).\n>\n>  - The request asks for both.  We end up calling object-info and\n>    also parse_object(), so presumably there is degradation of\n>    performance.\n>\n> In the third case, after v->s and v->value are filled by the new\n> code that copies from format_data, grab_values() will again fill\n> objecttype/objectsize by overwriting v->s field.  Doesn't this cause\n> memory leaks?  type_name() returns a constant string that does not\n> leak, but your objectsize seems to use xstrfmt(), so...\n>\n> I think it was OK before this patch as grab_common_values() was the\n> only place that did v->s = xstrfmt() for the field, but now the code\n> with this patch can do the same assignment from two places, we would\n> need to be a bit more careful about memory ownership?\n\nAnother thing that came to my mind while reading the patch aloud in\nmy previous message was if we can easily tell the latter two cases\napart, before actually going into the populate_value() codepath.\n\nWe can easily tell that fields that can benefit from object-info\nwere requested with your .use_data field technique.  If we can also\neasily tell that fields that do need full get_object() and call to\nparse_object(), then we can avoid calling object_info() and grab\nvalues for objectsize etc. in the old way.\n\nOf course, if you plan to extend the set of fields further so that\nsome traits about an object that can only be learned by calling\nobject_info(), then we may have to make calls to both object_info()\nand parse_object().  And when that happens we'd restructure the code\nagain, but until then, the above sounds like an optimization worth\nconsidering.\n\nThanks.\n"}]}