{"thread":{"id":"56239","subject":"[GSoC] Git Blog 12","startedAt":"2021-08-09T05:56:27Z","lastAt":"2021-08-13T08:26:09Z","messageCount":7,"participants":["ZheNing Hu","Christian Couder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"432285","messageId":"CAOLTT8TYiUbXbkSTPY50jCU1bYP3JFx2P+hVmGL58S8V=AUMbA@mail.gmail.com","threadId":"56239","inReplyTo":null,"subject":"[GSoC] Git Blog 12","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-09T05:56:06Z","receivedAt":"2021-08-09T05:56:27Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"My twelfth week blog finished:\nThe web version is here:\nhttps://adlternative.github.io/GSOC-Git-Blog-12/\n\n## Week12 Avoid repetitive parsing\n\nAfter we brought around `10%` optimization to `git cat-file --batch`\nthrough skip\nparse_object_buffer(), let's take a look at the result of gprof again:\n\n```\n  %\n time     calls(before)       calls(after)     name\n  1.24               0              349756     format_ref_array_item\n  1.24               0              349756     get_object\n  0.83         4184784             4534690     do_xmalloc\n  0.83               0             1399028     parse_ref_filter_atom\n  0.41         4184936             5932565     memory_limit_check\n  0.41          701711             1400412     strbuf_add\n  0.41               0             1399024     append_atom\n  0.41               0             1399024     quote_formatting\n  0.41              14              349770     strbuf_init\n  0.41               0              349756     populate_value\n  0.00         2100807             2449753     strbuf_grow\n  0.00         1973422             1973568     xmallocz\n  0.00               0             1399024     get_ref_atom_value\n  0.00               0             1399024     grab_values\n  0.00              77              699589     xstrdup\n  0.00              46              699558     xcalloc\n```\n\ngprof tells us that cat-file --batch will make a lot of copies by\n`xstrdup()`, `strbuf_add()`... after\nusing the logic of ref-filter. But at present, the overhead of these\ncopies cannot be easily avoided\ndue to the inherent logic of ref-filter. So there are no good\noptimization points in ref-filter ? We must\nre-observe the whole problem from a macro perspective.\n\n`oid_object_info_extended()` can get some metadata of the object, e.g.\n`size`, `type`, `deltabase`,\nthen we can use `grab_common_values()` to grab them. And those data in\nthe content of the object\nlike commits' `tree-oid`, `parent-oid` or tags' `deref-oid`, which are\nparsed by `parse_object_buffer()`,\nthen in `grab_tag_values()` or `grab_commit_values()`, we can grab\nthem. But many attributes of\ncommit and tag are not obtained through `parse_object_buffer()`, such\nas `author-info` ,`commiter-info`,\n`tagger-info` etc.\n\nWe need to call grab_sub_body_contents(), grab_person() to rescan the\nbuffer and extract the data.\nWhat if we can combine these multiple scanning and parsing into one completion?\nAt least intuitively, this has an opportunity to improve performance.\nSo I check the implementation\ndetails of `parse_commit_buffer()` and `parse_tag_buffer()`, maybe we\ncan pass some \"hook pointer\"\nto these parsing functions like `oid_object_info_extended()` does to\nextract only the information we need?\nThe commit-slab caught my attention. It can be used to get some\nspecified data content from the object.\nI am thinking about whether it is possible to design a `struct\nobject_view` (temporarily called\n`struct commit_view`) to store the offset of the parsed data in the\nobject content. `parse_commit_buffer()`\nwill check whether we need something for in-depth parsing. Like this:\n\n```c\nstruct commit_view {\nint need_tree : 1;\nint need_parents : 1;\n\nint need_author : 1;\nint need_author_name : 1;\nint need_author_email : 1;\nint need_author_date : 1;\n\nint need_committer : 1;\nint need_committer_name : 1;\nint need_committer_email : 1;\nint need_committer_date : 1;\n\nint tree_offset;\nint tree_length;\n\nint parents_nr;\nint *parents_offset;\nint *parents_length;\n\nint author_offset;\nint author_length;\n\nint author_name_offset;\nint author_name_length;\nint author_email_offset;\nint author_email_length;\nint author_date_offset;\nint author_date_length;\n\nint committer_offset;\nint committer_length;\n\nint committer_name_offset;\nint committer_name_length;\nint committer_email_offset;\nint committer_email_length;\nint committer_date_offset;\nint committer_date_length;\n};\n\ndefine_commit_slab(commit_view_slab, struct commit_view);\nstatic struct commit_view_slab view_slab = COMMIT_SLAB_INIT(1, view_slab);\n\nint parse_commit_buffer()\n{\n...\nif (view->need_author) {\nview->author_offset = bufptr - head;\nview->author_length = ident_len;\n}\nif (view->need_author_name || view->need_author_email ||\n    view->need_author_date) {\nif (split_ident_line(&ident, ident_line, ident_len) ||\n!ident.date_begin || !ident.date_end)\nreturn error(\"bad author line in commit %s\",\n     oid_to_hex(&item->object.oid));\nif (view->need_author_name) {\nview->author_name_offset = ident.name_begin - head;\nview->author_name_length = ident.name_end - ident.name_begin;\n}\nif (view->need_author_email) {\nview->author_email_offset = ident.mail_begin - head + 1;\nview->author_email_length = ident.mail_end - ident.mail_begin + 2;\n}\nif (view->need_author_date) {\nview->author_date_offset = ident.date_begin - head;\nview->author_date_length = ident.date_end - ident.date_begin;\n}\n}\n...\n}\n\n```\n\nIt's still in WIP, hope it can bring some help!\n\nThere seems to be no tag-slab similar to commit-slab, do we need to invent it?\n\nIt seems that GSOC has only the last few weeks left, I'm not sure how\nfar this patch series is from\nbeing merged by the master branch. Performance optimization may have\nno end. By the way,\nis there a chance to avoid a large number of copies in the ref-filter?\nThis may be another direction.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"432354","messageId":"CAP8UFD2yG3cC3qM3Ty7prdpVd=gXvArjM1soXD7EUn_NFFODaw@mail.gmail.com","threadId":"56239","inReplyTo":"CAOLTT8TYiUbXbkSTPY50jCU1bYP3JFx2P+hVmGL58S8V=AUMbA@mail.gmail.com","subject":"Re: [GSoC] Git Blog 12","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-08-10T08:04:44Z","receivedAt":"2021-08-10T08:05:16Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Aug 9, 2021 at 7:56 AM ZheNing Hu <adlternative@gmail.com> wrote:\n>\n> My twelfth week blog finished:\n> The web version is here:\n> https://adlternative.github.io/GSOC-Git-Blog-12/\n\nThanks!\n\n> ## Week12 Avoid repetitive parsing\n>\n> After we brought around `10%` optimization to `git cat-file --batch`\n> through skip\n\ns/skip/skipping/\n\n> parse_object_buffer(), let's take a look at the result of gprof again:\n>\n> ```\n>   %\n>  time     calls(before)       calls(after)     name\n>   1.24               0              349756     format_ref_array_item\n>   1.24               0              349756     get_object\n>   0.83         4184784             4534690     do_xmalloc\n>   0.83               0             1399028     parse_ref_filter_atom\n>   0.41         4184936             5932565     memory_limit_check\n>   0.41          701711             1400412     strbuf_add\n>   0.41               0             1399024     append_atom\n>   0.41               0             1399024     quote_formatting\n>   0.41              14              349770     strbuf_init\n>   0.41               0              349756     populate_value\n>   0.00         2100807             2449753     strbuf_grow\n>   0.00         1973422             1973568     xmallocz\n>   0.00               0             1399024     get_ref_atom_value\n>   0.00               0             1399024     grab_values\n>   0.00              77              699589     xstrdup\n>   0.00              46              699558     xcalloc\n> ```\n>\n> gprof tells us that cat-file --batch will make a lot of copies by\n> `xstrdup()`, `strbuf_add()`... after\n> using the logic of ref-filter. But at present, the overhead of these\n> copies cannot be easily avoided\n> due to the inherent logic of ref-filter. So there are no good\n> optimization points in ref-filter ? We must\n> re-observe the whole problem from a macro perspective.\n>\n> `oid_object_info_extended()` can get some metadata of the object, e.g.\n> `size`, `type`, `deltabase`,\n> then we can use `grab_common_values()` to grab them. And those data in\n> the content of the object\n> like commits' `tree-oid`, `parent-oid` or tags' `deref-oid`, which are\n> parsed by `parse_object_buffer()`,\n> then in `grab_tag_values()` or `grab_commit_values()`, we can grab\n> them. But many attributes of\n> commit and tag are not obtained through `parse_object_buffer()`, such\n> as `author-info` ,`commiter-info`,\n> `tagger-info` etc.\n>\n> We need to call grab_sub_body_contents(), grab_person() to rescan the\n> buffer and extract the data.\n> What if we can combine these multiple scanning and parsing into one completion?\n> At least intuitively, this has an opportunity to improve performance.\n\nYeah, but is there a way to check that we indeed scan or parse the\nsame objects multiple times? This way we might get an idea about how\nmuch scanning and parsing we could save.\n\n> So I check the implementation\n> details of `parse_commit_buffer()` and `parse_tag_buffer()`, maybe we\n> can pass some \"hook pointer\"\n> to these parsing functions like `oid_object_info_extended()` does to\n> extract only the information we need?\n\nWould this also avoid scanning and parsing the same object many times?\n\n> The commit-slab caught my attention. It can be used to get some\n> specified data content from the object.\n\nI thought it was for storing commit data in an efficient way.\n\n> I am thinking about whether it is possible to design a `struct\n> object_view` (temporarily called\n> `struct commit_view`) to store the offset of the parsed data in the\n> object content. `parse_commit_buffer()`\n> will check whether we need something for in-depth parsing. Like this:\n>\n> ```c\n> struct commit_view {\n> int need_tree : 1;\n> int need_parents : 1;\n>\n> int need_author : 1;\n> int need_author_name : 1;\n> int need_author_email : 1;\n> int need_author_date : 1;\n>\n> int need_committer : 1;\n> int need_committer_name : 1;\n> int need_committer_email : 1;\n> int need_committer_date : 1;\n\nIs the above info specific for each commit? Or will the above be the\nsame for all the commits we are processing?\n\n> int tree_offset;\n> int tree_length;\n>\n> int parents_nr;\n> int *parents_offset;\n> int *parents_length;\n>\n> int author_offset;\n> int author_length;\n>\n> int author_name_offset;\n> int author_name_length;\n> int author_email_offset;\n> int author_email_length;\n> int author_date_offset;\n> int author_date_length;\n>\n> int committer_offset;\n> int committer_length;\n>\n> int committer_name_offset;\n> int committer_name_length;\n> int committer_email_offset;\n> int committer_email_length;\n> int committer_date_offset;\n> int committer_date_length;\n> };\n>\n> define_commit_slab(commit_view_slab, struct commit_view);\n\nOk, so the idea is to use the commit slab feature to store the struct\ncommit_view instances. That seems reasonable to me.\n\n> static struct commit_view_slab view_slab = COMMIT_SLAB_INIT(1, view_slab);\n>\n> int parse_commit_buffer()\n> {\n> ...\n> if (view->need_author) {\n> view->author_offset = bufptr - head;\n> view->author_length = ident_len;\n> }\n> if (view->need_author_name || view->need_author_email ||\n>     view->need_author_date) {\n> if (split_ident_line(&ident, ident_line, ident_len) ||\n> !ident.date_begin || !ident.date_end)\n> return error(\"bad author line in commit %s\",\n>      oid_to_hex(&item->object.oid));\n> if (view->need_author_name) {\n> view->author_name_offset = ident.name_begin - head;\n> view->author_name_length = ident.name_end - ident.name_begin;\n> }\n> if (view->need_author_email) {\n> view->author_email_offset = ident.mail_begin - head + 1;\n> view->author_email_length = ident.mail_end - ident.mail_begin + 2;\n> }\n> if (view->need_author_date) {\n> view->author_date_offset = ident.date_begin - head;\n> view->author_date_length = ident.date_end - ident.date_begin;\n> }\n> }\n> ...\n> }\n>\n> ```\n>\n> It's still in WIP, hope it can bring some help!\n>\n> There seems to be no tag-slab similar to commit-slab, do we need to invent it?\n\nLet's first see if it helps when doing this for commits only. I also\nthink that usually the number of tags is much smaller than the number\nof commits.\n\n> It seems that GSOC has only the last few weeks left, I'm not sure how\n> far this patch series is from\n> being merged by the master branch. Performance optimization may have\n> no end.\n\nYeah, but the idea for now is just to make using the ref-filter code\nas fast as the current code.\n\n> By the way,\n> is there a chance to avoid a large number of copies in the ref-filter?\n> This may be another direction.\n\nYeah, but I think it's less promising than reducing the number of\ntimes objects are scanned or parsed. The first step though should be\nto show that it can indeed be reduced further (after the 10%\noptimization you made by skipping parse_object_buffer() when\npossible).\n"},{"id":"432377","messageId":"CAOLTT8TQxz2xkQE7etDa3cgb-kRfho-WAyhfHTsK=FrYVsPN1A@mail.gmail.com","threadId":"56239","inReplyTo":"CAP8UFD2yG3cC3qM3Ty7prdpVd=gXvArjM1soXD7EUn_NFFODaw@mail.gmail.com","subject":"Re: [GSoC] Git Blog 12","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-10T14:20:21Z","receivedAt":"2021-08-10T14:27:46Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder <christian.couder@gmail.com> 于2021年8月10日周二 下午4:04写道：\n\n> > parse_object_buffer(), let's take a look at the result of gprof again:\n> >\n> > We need to call grab_sub_body_contents(), grab_person() to rescan the\n> > buffer and extract the data.\n> > What if we can combine these multiple scanning and parsing into one completion?\n> > At least intuitively, this has an opportunity to improve performance.\n>\n> Yeah, but is there a way to check that we indeed scan or parse the\n> same objects multiple times? This way we might get an idea about how\n> much scanning and parsing we could save.\n>\n\nI think find_subpos() called by grab_sub_body_contents() and find_wholine()\ncalled by grab_person() are evidences that we are repeating iteratively.\nBut the proportion of time they occupy is too small. 0.0142% and 0.0109%\n\nSorry, but my attempts over the past two days have not gone well, the changes\nhere will make the program very complicated, the optimization here is not worth\ndoing.\n\n> > So I check the implementation\n> > details of `parse_commit_buffer()` and `parse_tag_buffer()`, maybe we\n> > can pass some \"hook pointer\"\n> > to these parsing functions like `oid_object_info_extended()` does to\n> > extract only the information we need?\n>\n> Would this also avoid scanning and parsing the same object many times?\n>\n\noid_object_info_extended()? I think it can set the pointer and extract\nthe required\nvalue. Well, the problem it solves may be a little different from here.\n\n> > The commit-slab caught my attention. It can be used to get some\n> > specified data content from the object.\n>\n> I thought it was for storing commit data in an efficient way.\n>\n\nYeah.\n\n> > I am thinking about whether it is possible to design a `struct\n> > object_view` (temporarily called\n> > `struct commit_view`) to store the offset of the parsed data in the\n> > object content. `parse_commit_buffer()`\n> > will check whether we need something for in-depth parsing. Like this:\n> >\n> > ```c\n> > struct commit_view {\n> > int need_tree : 1;\n> > int need_parents : 1;\n> >\n> > int need_author : 1;\n> > int need_author_name : 1;\n> > int need_author_email : 1;\n> > int need_author_date : 1;\n> >\n> > int need_committer : 1;\n> > int need_committer_name : 1;\n> > int need_committer_email : 1;\n> > int need_committer_date : 1;\n>\n> Is the above info specific for each commit? Or will the above be the\n> same for all the commits we are processing?\n>\n\nAccording to the my previous thoughts, I think it is same for all commits.\n\n>\n> Ok, so the idea is to use the commit slab feature to store the struct\n> commit_view instances. That seems reasonable to me.\n>\n\nIt's a pity that it may not bring much optimization in real situations.\n\n> > It seems that GSOC has only the last few weeks left, I'm not sure how\n> > far this patch series is from\n> > being merged by the master branch. Performance optimization may have\n> > no end.\n>\n> Yeah, but the idea for now is just to make using the ref-filter code\n> as fast as the current code.\n>\n\nIt seems difficult to achieve consistent performance. As we discussed\nbefore, the\nprevious `git cat-file --batch` can only provide a few kinds of\nmetadata that does not\nneed to be parsed, and after using ref-filter logic allows cat-file to\nuse more structured\ninformation about git objects. But this means it needs a harder path,\nIt requires many\ntraversals and many copies, If we really use the logic in ref-filter,\nwe can only do partial\noptimization, we can't expect it to be as fast as the old function.\nUnless we have the\nopportunity to not use the logic in ref-filter, but use the new atoms\nin ref-filter, this may\nhas a chance to escape the copy in the ref-filter. I don't know what\nyour opinion are...\n\n\nThanks.\n--\nZheNing Hu\n"},{"id":"432437","messageId":"CAOLTT8S0etdnFNKuD7z5QMxWdERZ-FWimn4Sc+u+7Ra1F2WAmA@mail.gmail.com","threadId":"56239","inReplyTo":"CAOLTT8TQxz2xkQE7etDa3cgb-kRfho-WAyhfHTsK=FrYVsPN1A@mail.gmail.com","subject":"Re: [GSoC] Git Blog 12","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-11T03:47:27Z","receivedAt":"2021-08-11T03:47:41Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"ZheNing Hu <adlternative@gmail.com> 于2021年8月10日周二 下午10:20写道：\n>\n> Christian Couder <christian.couder@gmail.com> 于2021年8月10日周二 下午4:04写道：\n>\n> > > parse_object_buffer(), let's take a look at the result of gprof again:\n> > >\n> > > We need to call grab_sub_body_contents(), grab_person() to rescan the\n> > > buffer and extract the data.\n> > > What if we can combine these multiple scanning and parsing into one completion?\n> > > At least intuitively, this has an opportunity to improve performance.\n> >\n> > Yeah, but is there a way to check that we indeed scan or parse the\n> > same objects multiple times? This way we might get an idea about how\n> > much scanning and parsing we could save.\n> >\n>\n> I think find_subpos() called by grab_sub_body_contents() and find_wholine()\n> called by grab_person() are evidences that we are repeating iteratively.\n> But the proportion of time they occupy is too small. 0.0142% and 0.0109%\n>\n\nUsing such a method may reduce some unnecessary scans [1]\nBut it can do very little optimization... 1.6%.\nOn the other hand, our optimization should focus on the default format of\n`git cat-file --batch`.\n\nMy new idea is to need a fast path: when we use the default format,\nlet us directly execute get_object() to avoid unnecessary traversal\nand checking.\n\nThanks,\n--\nZheNing Hu\n\n[1]: https://github.com/adlternative/git/commit/7d274971d2b5e1d4e6061d1e29e4a0b2c6a10ea5\n"},{"id":"432466","messageId":"CAP8UFD3bgWVH9pfEmvYSO9vs1sA-9vpoocTKqv=jS8zZBLZGWw@mail.gmail.com","threadId":"56239","inReplyTo":"CAOLTT8TQxz2xkQE7etDa3cgb-kRfho-WAyhfHTsK=FrYVsPN1A@mail.gmail.com","subject":"Re: [GSoC] Git Blog 12","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-08-11T08:15:10Z","receivedAt":"2021-08-11T08:15:24Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Aug 10, 2021 at 4:20 PM ZheNing Hu <adlternative@gmail.com> wrote:\n>\n> Christian Couder <christian.couder@gmail.com> 于2021年8月10日周二 下午4:04写道：\n>\n> > > parse_object_buffer(), let's take a look at the result of gprof again:\n> > >\n> > > We need to call grab_sub_body_contents(), grab_person() to rescan the\n> > > buffer and extract the data.\n> > > What if we can combine these multiple scanning and parsing into one completion?\n> > > At least intuitively, this has an opportunity to improve performance.\n> >\n> > Yeah, but is there a way to check that we indeed scan or parse the\n> > same objects multiple times? This way we might get an idea about how\n> > much scanning and parsing we could save.\n>\n> I think find_subpos() called by grab_sub_body_contents() and find_wholine()\n> called by grab_person() are evidences that we are repeating iteratively.\n> But the proportion of time they occupy is too small. 0.0142% and 0.0109%\n\nCould adding traces help with measuring how many times we call\nfunctions like get_object() or format_ref_array_item() for each\nobject?\n\nI think it's interesting to have a better idea about how many times\nfunctions that seem to take time like the above, or functions that\nread objects, like oid_object_info_extended(), are called for each\nobject.\n\n> Sorry, but my attempts over the past two days have not gone well, the changes\n> here will make the program very complicated, the optimization here is not worth\n> doing.\n\nMaybe but it would be interesting to document what didn't work and why.\n\n> > > So I check the implementation\n> > > details of `parse_commit_buffer()` and `parse_tag_buffer()`, maybe we\n> > > can pass some \"hook pointer\"\n> > > to these parsing functions like `oid_object_info_extended()` does to\n> > > extract only the information we need?\n> >\n> > Would this also avoid scanning and parsing the same object many times?\n>\n> oid_object_info_extended()? I think it can set the pointer and extract\n> the required\n> value. Well, the problem it solves may be a little different from here.\n\nCould you explain a bit more?\n\nI wonder if we sometimes call for example oid_object_info_extended()\nonce for an object and then parse_commit_buffer() for the same object,\nwhen perhaps calling only parse_commit_buffer() once would have given\nall the information we need.\n\n> > > I am thinking about whether it is possible to design a `struct\n> > > object_view` (temporarily called\n> > > `struct commit_view`) to store the offset of the parsed data in the\n> > > object content. `parse_commit_buffer()`\n> > > will check whether we need something for in-depth parsing. Like this:\n> > >\n> > > ```c\n> > > struct commit_view {\n> > > int need_tree : 1;\n> > > int need_parents : 1;\n> > >\n> > > int need_author : 1;\n> > > int need_author_name : 1;\n> > > int need_author_email : 1;\n> > > int need_author_date : 1;\n> > >\n> > > int need_committer : 1;\n> > > int need_committer_name : 1;\n> > > int need_committer_email : 1;\n> > > int need_committer_date : 1;\n> >\n> > Is the above info specific for each commit? Or will the above be the\n> > same for all the commits we are processing?\n>\n> According to the my previous thoughts, I think it is same for all commits.\n\nYeah, so I think it doesn't make sense to duplicate it for each\nobject. It could be a separate struct and we wouldn't need to have it\nin a commit slab.\n\nAnd I think maybe the above could be extended with fields like\n\"need_full_parsing\" or \"need_object_info\" that could be computed from\nthe above fields. This could then help avoid several calls to\ndifferent functions when one could be enough.\n\nAlso aren't bitfields like the above \"unsigned int\" instead of \"int\"\nin our code base?\n\n> > Ok, so the idea is to use the commit slab feature to store the struct\n> > commit_view instances. That seems reasonable to me.\n>\n> It's a pity that it may not bring much optimization in real situations.\n\nMaybe it's because copying data isn't actually a performance issue.\n\n> > > It seems that GSOC has only the last few weeks left, I'm not sure how\n> > > far this patch series is from\n> > > being merged by the master branch. Performance optimization may have\n> > > no end.\n> >\n> > Yeah, but the idea for now is just to make using the ref-filter code\n> > as fast as the current code.\n>\n> It seems difficult to achieve consistent performance. As we discussed\n> before, the\n> previous `git cat-file --batch` can only provide a few kinds of\n> metadata that does not\n> need to be parsed, and after using ref-filter logic allows cat-file to\n> use more structured\n> information about git objects.\n\nSo the issue seems to be that we can't detect if only a few metadata\nthat don't need to be parsed are needed?\n\nCould fields like \"need_full_parsing\" or \"need_object_info\" that I\nsuggest above help with that?\n\n> But this means it needs a harder path,\n> It requires many\n> traversals and many copies,\n\nCould you explain why it needs many traversal and copies in more\ndetails? And what could help avoid that?\n\n> If we really use the logic in ref-filter,\n> we can only do partial\n> optimization, we can't expect it to be as fast as the old function.\n> Unless we have the\n> opportunity to not use the logic in ref-filter, but use the new atoms\n> in ref-filter, this may\n> has a chance to escape the copy in the ref-filter. I don't know what\n> your opinion are...\n\nAre you sure we couldn't detect in the ref-filter code that we need\nonly a few metadata that don't need to be parsed? And then skip\nfetching and parsing data we don't need (without using a completely\nseparate code, like a fast path)?\n"},{"id":"432467","messageId":"CAP8UFD19PG7DnFzHF42PUDQ49pr8126ucYLVF6H4mAp61+Od2g@mail.gmail.com","threadId":"56239","inReplyTo":"CAOLTT8S0etdnFNKuD7z5QMxWdERZ-FWimn4Sc+u+7Ra1F2WAmA@mail.gmail.com","subject":"Re: [GSoC] Git Blog 12","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-08-11T08:20:07Z","receivedAt":"2021-08-11T08:20:22Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Aug 11, 2021 at 5:47 AM ZheNing Hu <adlternative@gmail.com> wrote:\n>\n> ZheNing Hu <adlternative@gmail.com> 于2021年8月10日周二 下午10:20写道：\n\n> > I think find_subpos() called by grab_sub_body_contents() and find_wholine()\n> > called by grab_person() are evidences that we are repeating iteratively.\n> > But the proportion of time they occupy is too small. 0.0142% and 0.0109%\n> >\n>\n> Using such a method may reduce some unnecessary scans [1]\n> But it can do very little optimization... 1.6%.\n> On the other hand, our optimization should focus on the default format of\n> `git cat-file --batch`.\n>\n> My new idea is to need a fast path: when we use the default format,\n> let us directly execute get_object() to avoid unnecessary traversal\n> and checking.\n\nI think it should be done not only when we use the default format but\nwhen we use any format that only needs the metadata provided by\nget_object(). This way that wouldn't be a special case fast path, but\nrather a generally useful optimization.\n"},{"id":"432661","messageId":"CAOLTT8QKapv6mFAr_AicDvivPPxZhan2hpbvQXvXMX6WeVMBpQ@mail.gmail.com","threadId":"56239","inReplyTo":"CAP8UFD19PG7DnFzHF42PUDQ49pr8126ucYLVF6H4mAp61+Od2g@mail.gmail.com","subject":"Re: [GSoC] Git Blog 12","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-13T08:25:56Z","receivedAt":"2021-08-13T08:26:09Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder <christian.couder@gmail.com> 于2021年8月11日周三 下午4:20写道：\n>\n> On Wed, Aug 11, 2021 at 5:47 AM ZheNing Hu <adlternative@gmail.com> wrote:\n> >\n> > ZheNing Hu <adlternative@gmail.com> 于2021年8月10日周二 下午10:20写道：\n>\n> > > I think find_subpos() called by grab_sub_body_contents() and find_wholine()\n> > > called by grab_person() are evidences that we are repeating iteratively.\n> > > But the proportion of time they occupy is too small. 0.0142% and 0.0109%\n> > >\n> >\n> > Using such a method may reduce some unnecessary scans [1]\n> > But it can do very little optimization... 1.6%.\n> > On the other hand, our optimization should focus on the default format of\n> > `git cat-file --batch`.\n> >\n> > My new idea is to need a fast path: when we use the default format,\n> > let us directly execute get_object() to avoid unnecessary traversal\n> > and checking.\n>\n> I think it should be done not only when we use the default format but\n> when we use any format that only needs the metadata provided by\n> get_object(). This way that wouldn't be a special case fast path, but\n> rather a generally useful optimization.\n\nSorry, the reply is a bit late. These two days have been busy implementing\nmore valuable performance optimization patches. I finally did not implement\nthe above plan. At present, ref-filter does have some small areas worth\noptimizing. A large part of the remaining performance loss comes from\nmore allocated memory and more copying. We must bear a certain gap\nin performance. But we can minimize it as much as possible. This is perhaps\nthe most reasonable at present.\n\nLet’s forget about the previous commit-slab for now and take a look at my\nnew optimization these two days:\nhttps://lore.kernel.org/git/pull.1016.git.1628842990.gitgitgadget@gmail.com/\n\nThanks.\n--\nZheNing Hu\n"}]}