{"thread":{"id":"56157","subject":"[GSOC] How to improve the performance of git cat-file --batch","startedAt":"2021-07-24T14:22:02Z","lastAt":"2021-07-28T15:36:49Z","messageCount":8,"participants":["ZheNing Hu","Ævar Arnfjörð Bjarmason","Christian Couder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"431119","messageId":"CAOLTT8RR3nvtXotqhSO8xPCzGQpGUA8dnNgraAjREZ6uLf4n4w@mail.gmail.com","threadId":"56157","inReplyTo":null,"subject":"[GSOC] How to improve the performance of git cat-file --batch","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-24T14:22:26Z","receivedAt":"2021-07-24T14:22:02Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"After reusing ref-filter logic for cat-file --batch, the function of\ncat-file --batch has been enhanced,\nbut the performance of cat-file is severely degraded. So we need a\nbetter solution to solve it.\nits last version is here:\nhttps://lore.kernel.org/git/pull.993.v2.git.1626363626.gitgitgadget@gmail.com/\n\nUse google doc to show some of my recent ideas:\nhttps://docs.google.com/document/d/1hyM-ecMd8C_TJ3OsOn46nMr8ZxCUxeK7HKj-bRkq3sk/edit?usp=sharing\n\nAnyone is welcome to comment and suggest better solutions!\n\nThanks!\n--\nZheNing Hu\n"},{"id":"431121","messageId":"87im0zs8wn.fsf@evledraar.gmail.com","threadId":"56157","inReplyTo":"CAOLTT8RR3nvtXotqhSO8xPCzGQpGUA8dnNgraAjREZ6uLf4n4w@mail.gmail.com","subject":"Re: [GSOC] How to improve the performance of git cat-file --batch","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-24T21:20:46Z","receivedAt":"2021-07-24T21:23:33Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Jul 24 2021, ZheNing Hu wrote:\n\n> After reusing ref-filter logic for cat-file --batch, the function of\n> cat-file --batch has been enhanced,\n> but the performance of cat-file is severely degraded. So we need a\n> better solution to solve it.\n> its last version is here:\n> https://lore.kernel.org/git/pull.993.v2.git.1626363626.gitgitgadget@gmail.com/\n>\n> Use google doc to show some of my recent ideas:\n> https://docs.google.com/document/d/1hyM-ecMd8C_TJ3OsOn46nMr8ZxCUxeK7HKj-bRkq3sk/edit?usp=sharing\n>\n> Anyone is welcome to comment and suggest better solutions!\n\nHaving looked at that Google Doc it seems to just discuss \"Plan A\", or\nthere some multi-doc magic and there's a \"Plan B\", or...? I'm not very\nfamiliar with Google Docs.\n\nAnyway, I'd encourage you to just send this as an E-Mail to the mailing\nlist.\n\nHaving skimmed it I'm a bit confused about this in reference to\nperformance generally. I haven't looked into the case you're discussing,\nbut as I noted in\nhttps://lore.kernel.org/git/87im1p6x34.fsf@evledraar.gmail.com/ the\nprofiling clearly shows that the main problem is that you've added\nobject lookups we skipped before.\n\nI think any plan to improve performance should start with a perf test,\nsee which of your patches have performance regressions, if some don't\nand are otherwise OK perhaps those can go in earlier.\n\nThen we'll have the smallest possible set of changes that's correct, but\nintroduces performance regressions, and can look at what those are doing\nto slow things down...\n"},{"id":"431131","messageId":"CAOLTT8Sbusr8=iJbG1qXcSerivZqP5xm-GS8R7TqDMh7QXSZaA@mail.gmail.com","threadId":"56157","inReplyTo":"87im0zs8wn.fsf@evledraar.gmail.com","subject":"Re: [GSOC] How to improve the performance of git cat-file --batch","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-25T12:05:12Z","receivedAt":"2021-07-25T12:04:45Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> 于2021年7月25日周日 上午5:23写道：\n>\n>\n> On Sat, Jul 24 2021, ZheNing Hu wrote:\n>\n> > After reusing ref-filter logic for cat-file --batch, the function of\n> > cat-file --batch has been enhanced,\n> > but the performance of cat-file is severely degraded. So we need a\n> > better solution to solve it.\n> > its last version is here:\n> > https://lore.kernel.org/git/pull.993.v2.git.1626363626.gitgitgadget@gmail.com/\n> >\n> > Use google doc to show some of my recent ideas:\n> > https://docs.google.com/document/d/1hyM-ecMd8C_TJ3OsOn46nMr8ZxCUxeK7HKj-bRkq3sk/edit?usp=sharing\n> >\n> > Anyone is welcome to comment and suggest better solutions!\n>\n> Having looked at that Google Doc it seems to just discuss \"Plan A\", or\n> there some multi-doc magic and there's a \"Plan B\", or...? I'm not very\n> familiar with Google Docs.\n>\n\nThere may be several plan that have not been written to it for the time being.\n\n> Anyway, I'd encourage you to just send this as an E-Mail to the mailing\n> list.\n>\n\nOk, I can migrate them here.\n\n> Having skimmed it I'm a bit confused about this in reference to\n> performance generally. I haven't looked into the case you're discussing,\n> but as I noted in\n> https://lore.kernel.org/git/87im1p6x34.fsf@evledraar.gmail.com/ the\n> profiling clearly shows that the main problem is that you've added\n> object lookups we skipped before.\n>\n\nYeah, you showed me last time that lookup_object() took up a lot of time.\n\n> I think any plan to improve performance should start with a perf test,\n> see which of your patches have performance regressions, if some don't\n> and are otherwise OK perhaps those can go in earlier.\n>\n\nMakes sense. Last time the test performance of commits is here:\nhttps://lore.kernel.org/git/CAOLTT8RR4+tUuT2yc2PDL9NwCburW8bM_Sht6nhKJ_fYV8fGsQ@mail.gmail.com/\n\nThe main commit with performance degradation is \"[GSOC] cat-file:\nreuse ref-filter logic\",\nwe need to focus on it.\n\n> Then we'll have the smallest possible set of changes that's correct, but\n> introduces performance regressions, and can look at what those are doing\n> to slow things down...\n\nOk.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"431168","messageId":"CAP8UFD1szPO-qmusSqiLRWxynV0gcy8UsnZORNn0USYAmRUGVw@mail.gmail.com","threadId":"56157","inReplyTo":"CAOLTT8Sbusr8=iJbG1qXcSerivZqP5xm-GS8R7TqDMh7QXSZaA@mail.gmail.com","subject":"Re: [GSOC] How to improve the performance of git cat-file --batch","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-26T09:38:20Z","receivedAt":"2021-07-26T09:38:34Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sun, Jul 25, 2021 at 2:04 PM ZheNing Hu <adlternative@gmail.com> wrote:\n> Ævar Arnfjörð Bjarmason <avarab@gmail.com> 于2021年7月25日周日 上午5:23写道：\n\n> > Having skimmed it I'm a bit confused about this in reference to\n> > performance generally. I haven't looked into the case you're discussing,\n> > but as I noted in\n> > https://lore.kernel.org/git/87im1p6x34.fsf@evledraar.gmail.com/ the\n> > profiling clearly shows that the main problem is that you've added\n> > object lookups we skipped before.\n>\n> Yeah, you showed me last time that lookup_object() took up a lot of time.\n\nCould the document explain with some details why there are more calls\nto lookup_object()? For example it could take an example `git cat-file\n--batch ...` command (if possible a simple one), and say which\nfunctions like lookup_object() it was using (and how many times) to\nget the data it needs before using the ref-filter logic, and then the\nsame information after using the ref-filter logic.\n\nIt could be nice if there were also some data about how much time used\nto be spent in lookup_object() and how much time is now spent there,\nand how this compares with the whole slowdown we are seeing. If Ævar\nalready showed that, you can of course reuse what he already did.\n\nThe GIT_TRACE_PERFORMANCE, GIT_TRACE2_PERF env variables and the\nassociated trace_*() and trace2_*() functions might help you with\nmeasuring how much time is spent in different functions.\n"},{"id":"431279","messageId":"CAOLTT8TdL7UhfVSOzbpmo-WFNrcKwmy=E720tNt4KM9o_p=keg@mail.gmail.com","threadId":"56157","inReplyTo":"CAP8UFD1szPO-qmusSqiLRWxynV0gcy8UsnZORNn0USYAmRUGVw@mail.gmail.com","subject":"Re: [GSOC] How to improve the performance of git cat-file --batch","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-27T01:37:44Z","receivedAt":"2021-07-27T01:37:16Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder <christian.couder@gmail.com> 于2021年7月26日周一 下午5:38写道：\n>\n> On Sun, Jul 25, 2021 at 2:04 PM ZheNing Hu <adlternative@gmail.com> wrote:\n> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> 于2021年7月25日周日 上午5:23写道：\n>\n> > > Having skimmed it I'm a bit confused about this in reference to\n> > > performance generally. I haven't looked into the case you're discussing,\n> > > but as I noted in\n> > > https://lore.kernel.org/git/87im1p6x34.fsf@evledraar.gmail.com/ the\n> > > profiling clearly shows that the main problem is that you've added\n> > > object lookups we skipped before.\n> >\n> > Yeah, you showed me last time that lookup_object() took up a lot of time.\n>\n> Could the document explain with some details why there are more calls\n> to lookup_object()? For example it could take an example `git cat-file\n> --batch ...` command (if possible a simple one), and say which\n> functions like lookup_object() it was using (and how many times) to\n> get the data it needs before using the ref-filter logic, and then the\n> same information after using the ref-filter logic.\n>\n\nSorry but this time I use gprof but can’t observe the same effect as before.\nlookup_object() is indeed a part of the time overhead, but its proportion is\nnot very large this time.\n\n> It could be nice if there were also some data about how much time used\n> to be spent in lookup_object() and how much time is now spent there,\n> and how this compares with the whole slowdown we are seeing. If Ævar\n> already showed that, you can of course reuse what he already did.\n>\n\nThis is my test for git cat-file --batch --batch-all-objects >/dev/null:\n\ndaab8a564: The fifth batch (upstream/master)\n\nFlat profile:\n\nEach sample counts as 0.01 seconds.\n  %   cumulative   self              self     total\n time   seconds   seconds    calls   s/call   s/call  name\n 38.13      0.61     0.61  1968866     0.00     0.00  patch_delta\n 13.75      0.83     0.22  6568488     0.00     0.00\nunpack_object_header_buffer\n 11.25      1.01     0.18   344036     0.00     0.00  unpack_entry\n  7.50      1.13     0.12  1964667     0.00     0.00  hashmap_remove\n  6.88      1.24     0.11  6153182     0.00     0.00  hashmap_get\n  1.88      1.27     0.03  7746299     0.00     0.00  zlib_post_call\n  1.88      1.30     0.03   842731     0.00     0.00  bsearch_hash\n  1.88      1.33     0.03   827663     0.00     0.00  nth_packed_object_offset\n  1.25      1.35     0.02 15385422     0.00     0.00  use_pack\n  1.25      1.37     0.02  6236120     0.00     0.00  get_delta_base\n  1.25      1.39     0.02  2581859     0.00     0.00  git_inflate_end\n  1.25      1.41     0.02   826650     0.00     0.00\ndo_oid_object_info_extended\n  1.25      1.43     0.02   826650     0.00     0.00  find_pack_entry\n  1.25      1.45     0.02   825692     0.00     0.00  packed_to_object_type\n  1.25      1.47     0.02   378521     0.00     0.00  get_size_from_delta\n\n\nd3b5272a94: [GSOC] cat-file: reuse ref-filter logic\n\nFlat profile:\n\nEach sample counts as 0.01 seconds.\n  %   cumulative   self              self     total\n time   seconds   seconds    calls   s/call   s/call  name\n 27.06      0.59     0.59  1968866     0.00     0.00  patch_delta\n 16.51      0.95     0.36  2202293     0.00     0.00\nunpack_object_header_buffer\n 13.76      1.25     0.30  5327015     0.00     0.00  hashmap_get\n 11.47      1.50     0.25   344036     0.00     0.00  unpack_entry\n  8.72      1.69     0.19   521278     0.00     0.00  lookup_object\n  4.13      1.78     0.09  1964667     0.00     0.00  hashmap_remove\n  2.75      1.84     0.06   348709     0.00     0.00  get_object\n  2.29      1.89     0.05        1     0.05     2.17  oid_array_for_each_unique\n  1.38      1.92     0.03  6373452     0.00     0.00  use_pack\n  0.92      1.94     0.02  2202293     0.00     0.00  unpack_compressed_entry\n  0.92      1.96     0.02  1394836     0.00     0.00  grab_sub_body_contents\n  0.92      1.98     0.02   348709     0.00     0.00  create_object\n  0.92      2.00     0.02   348709     0.00     0.00  format_ref_array_item\n  0.92      2.02     0.02    74557     0.00     0.00  fill_commit_graph_info\n\nBecause we called parse_object_buffer() in get_object(), lookup_object()\nis called indirectly...\n\nWe can see that some functions are called the same times: patch_delta(),\nunpack_entry(), hashmap_remove()... But after using my patch,\nformat_ref_array_item(), grab_sub_body_contents(), get_object(), lookup_object()\nbegin to occupy a certain proportion.\n\nThis is my test for git cat-file --batch-check --batch-all-objects >/dev/null:\n\ndaab8a564: The fifth batch (upstream/master)\n\nFlat profile:\n\nEach sample counts as 0.01 seconds.\n  %   cumulative   self              self     total\n time   seconds   seconds    calls  ms/call  ms/call  name\n 47.83      0.11     0.11  3385670     0.00     0.00\nunpack_object_header_buffer\n 13.04      0.14     0.03  6941590     0.00     0.00  use_pack\n  8.70      0.16     0.02  1046130     0.00     0.00  expand_format\n  4.35      0.17     0.01   349013     0.00     0.00  oid_array_append\n  4.35      0.18     0.01   348710     0.00     0.00  strbuf_expand\n  4.35      0.19     0.01   348709     0.00     0.00  find_pack_entry\n  4.35      0.20     0.01   348230     0.00     0.00  packed_to_object_type\n  4.35      0.21     0.01   259719     0.00     0.00  git_inflate\n  4.35      0.22     0.01        1    10.00   210.00  oid_array_for_each_unique\n  4.35      0.23     0.01                             pack_basename\n\nd3b5272a94: [GSOC] cat-file: reuse ref-filter logic\n\nFlat profile:\n\nEach sample counts as 0.01 seconds.\n  %   cumulative   self              self     total\n time   seconds   seconds    calls  ms/call  ms/call  name\n 52.00      0.13     0.13  3385670     0.00     0.00\nunpack_object_header_buffer\n 16.00      0.17     0.04  6941590     0.00     0.00  use_pack\n  8.00      0.19     0.02  3296680     0.00     0.00  get_delta_base\n  4.00      0.20     0.01   348709     0.00     0.00  find_pack_entry\n  4.00      0.21     0.01   348709     0.00     0.00  oid_to_hex\n  4.00      0.22     0.01   348709     0.00     0.00  populate_value\n  4.00      0.23     0.01   348230     0.00     0.00  packed_object_info\n  4.00      0.24     0.01   348230     0.00     0.00  packed_to_object_type\n  4.00      0.25     0.01                             void_hashcmp\n\nSimilarly, unpack_object_header_buffer(), use_pack(),\nfind_pack_entry(), packed_to_object_type(), they are still called same\ntimes as before; populate_value(),\npacked_object_info(), oid_to_hex() are new.\n\n> The GIT_TRACE_PERFORMANCE, GIT_TRACE2_PERF env variables and the\n> associated trace_*() and trace2_*() functions might help you with\n> measuring how much time is spent in different functions.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"431386","messageId":"CAP8UFD1WtSX59AqfG=d0Ge2BcK+8LdyZk0mQuftpu=FKX-877Q@mail.gmail.com","threadId":"56157","inReplyTo":"CAOLTT8TdL7UhfVSOzbpmo-WFNrcKwmy=E720tNt4KM9o_p=keg@mail.gmail.com","subject":"Re: [GSOC] How to improve the performance of git cat-file --batch","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-28T07:34:39Z","receivedAt":"2021-07-28T07:34:59Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Jul 27, 2021 at 3:37 AM ZheNing Hu <adlternative@gmail.com> wrote:\n>\n> Christian Couder <christian.couder@gmail.com> 于2021年7月26日周一 下午5:38写道：\n> >\n> > On Sun, Jul 25, 2021 at 2:04 PM ZheNing Hu <adlternative@gmail.com> wrote:\n> > > Ævar Arnfjörð Bjarmason <avarab@gmail.com> 于2021年7月25日周日 上午5:23写道：\n> >\n> > > > Having skimmed it I'm a bit confused about this in reference to\n> > > > performance generally. I haven't looked into the case you're discussing,\n> > > > but as I noted in\n> > > > https://lore.kernel.org/git/87im1p6x34.fsf@evledraar.gmail.com/ the\n> > > > profiling clearly shows that the main problem is that you've added\n> > > > object lookups we skipped before.\n> > >\n> > > Yeah, you showed me last time that lookup_object() took up a lot of time.\n> >\n> > Could the document explain with some details why there are more calls\n> > to lookup_object()?\n\nPlease note that here we are looking for the number of times the\nlookup_object() function is called. This means that to measure that\nproperly, it might actually be better to have some way to count this\nnumber of times the lookup_object() function is called, rather than\ncount the time spent in the function.\n\nFor example you could add a trace_printf(...) call in the\nlookup_object() function, set GIT_TRACE=/tmp/git_trace.log, and then\njust run `git cat-file --batch ...` and count the number of times the\nnew trace from lookup_object() appears in the log file.\n\n> > For example it could take an example `git cat-file\n> > --batch ...` command (if possible a simple one), and say which\n> > functions like lookup_object() it was using (and how many times) to\n> > get the data it needs before using the ref-filter logic, and then the\n> > same information after using the ref-filter logic.\n>\n> Sorry but this time I use gprof but can’t observe the same effect as before.\n> lookup_object() is indeed a part of the time overhead, but its proportion is\n> not very large this time.\n\nI am not sure gprof is a good tool for this. It looks like it tries to\nattribute spent times to functions by splitting time between many low\nlevel functions, and it doesn't seem like the right approach to me.\nFor example if lookup_object() is called 5% more often, it could be\nthat the excess time is attributed to some low level functions and not\nto lookup_object() itself.\n\nThat's why we might get a more accurate view of what happens by just\ncounting the number of time the function is called.\n\n> > It could be nice if there were also some data about how much time used\n> > to be spent in lookup_object() and how much time is now spent there,\n> > and how this compares with the whole slowdown we are seeing. If Ævar\n> > already showed that, you can of course reuse what he already did.\n\nNow I regret having wrote the above, sorry, as it might not be the\nbest way to look at this.\n\n> This is my test for git cat-file --batch --batch-all-objects >/dev/null:\n\n[...]\n\n> Because we called parse_object_buffer() in get_object(), lookup_object()\n> is called indirectly...\n\nIt would be nice if you could add a bit more details about how\nlookup_object() is called (both before and after the changes that\ndegrade performance).\n\n> We can see that some functions are called the same times:\n\nWhen you say \"the same times\" I guess you mean that the same amount of\ntime is spent in these functions.\n\n> patch_delta(),\n> unpack_entry(), hashmap_remove()... But after using my patch,\n> format_ref_array_item(), grab_sub_body_contents(), get_object(), lookup_object()\n> begin to occupy a certain proportion.\n\nThanks!\n"},{"id":"431396","messageId":"CAOLTT8QvtJ70X8mQx4K4gD0T=7i-ryd0QL81-QeSTqSWyHuWLQ@mail.gmail.com","threadId":"56157","inReplyTo":"CAP8UFD1WtSX59AqfG=d0Ge2BcK+8LdyZk0mQuftpu=FKX-877Q@mail.gmail.com","subject":"Re: [GSOC] How to improve the performance of git cat-file --batch","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-07-28T13:38:33Z","receivedAt":"2021-07-28T13:38:03Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder <christian.couder@gmail.com> 于2021年7月28日周三 下午3:34写道：\n>\n> On Tue, Jul 27, 2021 at 3:37 AM ZheNing Hu <adlternative@gmail.com> wrote:\n> >\n> > Christian Couder <christian.couder@gmail.com> 于2021年7月26日周一 下午5:38写道：\n> > >\n> > > On Sun, Jul 25, 2021 at 2:04 PM ZheNing Hu <adlternative@gmail.com> wrote:\n> > > > Ævar Arnfjörð Bjarmason <avarab@gmail.com> 于2021年7月25日周日 上午5:23写道：\n> > >\n> > > > > Having skimmed it I'm a bit confused about this in reference to\n> > > > > performance generally. I haven't looked into the case you're discussing,\n> > > > > but as I noted in\n> > > > > https://lore.kernel.org/git/87im1p6x34.fsf@evledraar.gmail.com/ the\n> > > > > profiling clearly shows that the main problem is that you've added\n> > > > > object lookups we skipped before.\n> > > >\n> > > > Yeah, you showed me last time that lookup_object() took up a lot of time.\n> > >\n> > > Could the document explain with some details why there are more calls\n> > > to lookup_object()?\n>\n> Please note that here we are looking for the number of times the\n> lookup_object() function is called. This means that to measure that\n> properly, it might actually be better to have some way to count this\n> number of times the lookup_object() function is called, rather than\n> count the time spent in the function.\n>\n\nOk, therefore we need an accurate number of call times about lookup_object(),\nalthough the conclusion is obvious: 0 (upstream/master) and a big\nnumber (with my patch).\n\n> For example you could add a trace_printf(...) call in the\n> lookup_object() function, set GIT_TRACE=/tmp/git_trace.log, and then\n> just run `git cat-file --batch ...` and count the number of times the\n> new trace from lookup_object() appears in the log file.\n>\n\nAfter adding a trace_printf() in lookup_object(), here is the result:\n\nCheckout to d3b5272a94 [GSOC] cat-file: reuse ref-filter logic\n\n$ GIT_TRACE=/tmp/git_trace.log ggg cat-file --batch --batch-all-objects\n$  cat /tmp/git_trace.log | wc -l\n522710\n\nCheckout to eb27b338a3e7 (upstream/master)\n\n$ rm /tmp/git_trace.log\n$ GIT_TRACE=/tmp/git_trace.log ggg cat-file --batch --batch-all-objects\n$  cat /tmp/git_trace.log | wc -l\n1\n\nThis is the only 1 time left is printed by git.c, which show that after using\nmy patch, we additionally call  lookup_object() when we use --batch option.\nAccording to the results of the previous gprof test: lookup_object()\noccupies 8.72%\nof the total time. (Though below you seem to think that the effect of\ngprof is not\nreliable enough.) This may be a place worthy of optimization.\n\n> > > For example it could take an example `git cat-file\n> > > --batch ...` command (if possible a simple one), and say which\n> > > functions like lookup_object() it was using (and how many times) to\n> > > get the data it needs before using the ref-filter logic, and then the\n> > > same information after using the ref-filter logic.\n> >\n> > Sorry but this time I use gprof but can’t observe the same effect as before.\n> > lookup_object() is indeed a part of the time overhead, but its proportion is\n> > not very large this time.\n>\n> I am not sure gprof is a good tool for this. It looks like it tries to\n> attribute spent times to functions by splitting time between many low\n> level functions, and it doesn't seem like the right approach to me.\n> For example if lookup_object() is called 5% more often, it could be\n> that the excess time is attributed to some low level functions and not\n> to lookup_object() itself.\n>\n\nMaybe you are talking about \"cumulative seconds\" part of gprof?\nit's \"self seconds\" part  is the number of seconds accounted for by\nthis function alone.\n\n> That's why we might get a more accurate view of what happens by just\n> counting the number of time the function is called.\n>\n\nAccording to https://sourceware.org/binutils/docs/gprof/Flat-Profile.html :\n\ncalls\nThis is the total number of times the function was called. If the\nfunction was never called,\nor the number of times it was called cannot be determined (probably\nbecause the function\nwas not compiled with profiling enabled), the calls field is blank.\n\nTherefore, there are accurate numbers in the previous gprof test results.\n\n> > > It could be nice if there were also some data about how much time used\n> > > to be spent in lookup_object() and how much time is now spent there,\n> > > and how this compares with the whole slowdown we are seeing. If Ævar\n> > > already showed that, you can of course reuse what he already did.\n>\n> Now I regret having wrote the above, sorry, as it might not be the\n> best way to look at this.\n>\n\nYes, because his previous results are based on a version of my patch set,\nand some changes have taken place with my patch later.\n\n> > This is my test for git cat-file --batch --batch-all-objects >/dev/null:\n>\n> [...]\n>\n> > Because we called parse_object_buffer() in get_object(), lookup_object()\n> > is called indirectly...\n>\n> It would be nice if you could add a bit more details about how\n> lookup_object() is called (both before and after the changes that\n> degrade performance).\n>\n\nAfter we letting git cat-file --batch reuse the logic of ref-filter,\nwe will use get_object()\nto grab the object's data. Since we used atom %(raw), it will require\nus to grab the raw data\nof the object, oi->info.contentp will be set, parse_object_buffer() in\nget_object() will be\ncalled, parse_object_buffer() calls lookup_commit(), lookup_blob(),\nlookup_tree(),\nand lookup_tag(), they call lookup_object(). As we have seen,\nlookup_object() seems to\ntake a lot of time.\n\nSo let us think, can we skip this parse_object_buffer() in some scenarios?\nparse_object_buffer() parses the data of the object into a \"struct\nobject *obj\", and then we use this\nobj feed to grab_values(), and then grab_values() feed obj to\ngrab_tag_values() or grab_commit_values()\nto handle some logic about %(tag), %(type), %(object), %(tree),\n%(parent), %(numparent).\n\nBut git cat-file --batch can avaid handle there atoms with default format.\n\nTherefore, maybe we can skip parsing object buffer if we really don't\ncare about these atoms.\n\n> > We can see that some functions are called the same times:\n>\n> When you say \"the same times\" I guess you mean that the same amount of\n> time is spent in these functions.\n\nNo... What I want to express is that the number of calls to these\nfunctions is same,\nsee gprof \"calls part\" with  \"patch_delta()\", both are 1968866.\n\n>\n> > patch_delta(),\n> > unpack_entry(), hashmap_remove()... But after using my patch,\n> > format_ref_array_item(), grab_sub_body_contents(), get_object(), lookup_object()\n> > begin to occupy a certain proportion.\n>\n> Thanks!\n\nThanks for hint!!!\n--\nZheNing Hu\n"},{"id":"431400","messageId":"CAP8UFD1tS++xEMob7p5UeDQ_enoyy1aOQTG=28r-ge2k4n3ECw@mail.gmail.com","threadId":"56157","inReplyTo":"CAOLTT8QvtJ70X8mQx4K4gD0T=7i-ryd0QL81-QeSTqSWyHuWLQ@mail.gmail.com","subject":"Re: [GSOC] How to improve the performance of git cat-file --batch","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-07-28T15:36:33Z","receivedAt":"2021-07-28T15:36:49Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 28, 2021 at 3:38 PM ZheNing Hu <adlternative@gmail.com> wrote:\n\n> Ok, therefore we need an accurate number of call times about lookup_object(),\n> although the conclusion is obvious: 0 (upstream/master) and a big\n> number (with my patch).\n\n[...]\n\n> This is the only 1 time left is printed by git.c, which show that after using\n> my patch, we additionally call  lookup_object() when we use --batch option.\n> According to the results of the previous gprof test: lookup_object()\n> occupies 8.72%\n> of the total time. (Though below you seem to think that the effect of\n> gprof is not\n> reliable enough.) This may be a place worthy of optimization.\n\nFirst, yeah, I hadn't seen the \"calls\" columns in your gprof reports.\nSorry! It's nice to see that your manual check with a trace_printf()\nfunction gives the same result as gprof about this though.\n\nAnyway if you agree that it might be a place worthy of optimization,\nthen it might be a good idea to explain the reason for the numerous\nlookup_object() calls when using the ref-filter code.\n\n[...]\n\n> > It would be nice if you could add a bit more details about how\n> > lookup_object() is called (both before and after the changes that\n> > degrade performance).\n>\n> After we letting git cat-file --batch reuse the logic of ref-filter,\n> we will use get_object()\n> to grab the object's data. Since we used atom %(raw), it will require\n> us to grab the raw data\n> of the object, oi->info.contentp will be set, parse_object_buffer() in\n> get_object() will be\n> called, parse_object_buffer() calls lookup_commit(), lookup_blob(),\n> lookup_tree(),\n> and lookup_tag(), they call lookup_object(). As we have seen,\n> lookup_object() seems to\n> take a lot of time.\n\nNot sure why you are talking about %(raw). Is the root of the issue\nthat we now use atom %(raw), or how we implemented it?\n\n> So let us think, can we skip this parse_object_buffer() in some scenarios?\n> parse_object_buffer() parses the data of the object into a \"struct\n> object *obj\", and then we use this\n> obj feed to grab_values(), and then grab_values() feed obj to\n> grab_tag_values() or grab_commit_values()\n> to handle some logic about %(tag), %(type), %(object), %(tree),\n> %(parent), %(numparent).\n>\n> But git cat-file --batch can avaid handle there atoms with default format.\n>\n> Therefore, maybe we can skip parsing object buffer if we really don't\n> care about these atoms.\n\nYeah, maybe oi->info.contentp should be set only if the user specified\none of the atoms that really needs the content provided by\nparse_object_buffer().\n\nThanks!\n"}]}