{"thread":{"id":"56191","subject":"[GSoC] Git Blog 11","startedAt":"2021-08-01T06:45:34Z","lastAt":"2021-08-05T04:49:39Z","messageCount":6,"participants":["ZheNing Hu","Christian Couder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"431646","messageId":"CAOLTT8RwwMF07f=XxWN=zGsPU0VQ8FqPVdyepQp78Ei8WZpSrw@mail.gmail.com","threadId":"56191","inReplyTo":null,"subject":"[GSoC] Git Blog 11","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-01T06:46:09Z","receivedAt":"2021-08-01T06:45:34Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"My eleventh week blog finished:\nThe web version is here:\nhttps://adlternative.github.io/GSOC-Git-Blog-11/\n\n### Attempt to optimize performance\n\nThis week, at the prompt of my mentor Christian, I used `gprof` for some\nperformance tests about `git cat-file --batch`:\n[Re: [GSOC] How to improve the performance of git cat-file --batch]\n(https://lore.kernel.org/git/CAOLTT8TdL7UhfVSOzbpmo-WFNrcKwmy=E720tNt4KM9o_p=keg@mail.gmail.com/)\n\n```\nThis is my test for git cat-file --batch --batch-all-objects >/dev/null:\n\ndaab8a564: The fifth batch (upstream/master)\n\nFlat profile:\n\nEach sample counts as 0.01 seconds.\n % cumulative self self total\n time seconds seconds calls s/call s/call name\n 38.13 0.61 0.61 1968866 0.00 0.00 patch_delta\n 13.75 0.83 0.22 6568488 0.00 0.00\nunpack_object_header_buffer\n 11.25 1.01 0.18 344036 0.00 0.00 unpack_entry\n 7.50 1.13 0.12 1964667 0.00 0.00 hashmap_remove\n 6.88 1.24 0.11 6153182 0.00 0.00 hashmap_get\n 1.88 1.27 0.03 7746299 0.00 0.00 zlib_post_call\n 1.88 1.30 0.03 842731 0.00 0.00 bsearch_hash\n 1.88 1.33 0.03 827663 0.00 0.00 nth_packed_object_offset\n 1.25 1.35 0.02 15385422 0.00 0.00 use_pack\n 1.25 1.37 0.02 6236120 0.00 0.00 get_delta_base\n 1.25 1.39 0.02 2581859 0.00 0.00 git_inflate_end\n 1.25 1.41 0.02 826650 0.00 0.00\ndo_oid_object_info_extended\n 1.25 1.43 0.02 826650 0.00 0.00 find_pack_entry\n 1.25 1.45 0.02 825692 0.00 0.00 packed_to_object_type\n 1.25 1.47 0.02 378521 0.00 0.00 get_size_from_delta\n\n\nd3b5272a94: [GSOC] cat-file: reuse ref-filter logic\n\nFlat profile:\n\nEach sample counts as 0.01 seconds.\n % cumulative self self total\n time seconds seconds calls s/call s/call name\n 27.06 0.59 0.59 1968866 0.00 0.00 patch_delta\n 16.51 0.95 0.36 2202293 0.00 0.00\nunpack_object_header_buffer\n 13.76 1.25 0.30 5327015 0.00 0.00 hashmap_get\n 11.47 1.50 0.25 344036 0.00 0.00 unpack_entry\n 8.72 1.69 0.19 521278 0.00 0.00 lookup_object\n 4.13 1.78 0.09 1964667 0.00 0.00 hashmap_remove\n 2.75 1.84 0.06 348709 0.00 0.00 get_object\n 2.29 1.89 0.05 1 0.05 2.17 oid_array_for_each_unique\n 1.38 1.92 0.03 6373452 0.00 0.00 use_pack\n 0.92 1.94 0.02 2202293 0.00 0.00 unpack_compressed_entry\n 0.92 1.96 0.02 1394836 0.00 0.00 grab_sub_body_contents\n 0.92 1.98 0.02 348709 0.00 0.00 create_object\n 0.92 2.00 0.02 348709 0.00 0.00 format_ref_array_item\n 0.92 2.02 0.02 74557 0.00 0.00 fill_commit_graph_info\n```\n\nBefore, I thought that the proportion of `lookup_object()` is not very\nlarge(11.47%), so\nI didn't care about it. But Christian strongly recommends that I use\n`trace_printf()` to\nobserve the number of calls to `lookup_object()`.\n\nHere is an amazing fact:\n\nThe number of calls to `lookup_object()` before and after using my\npatch are 0 and\n522709 respectively. Therefore, I am very surprised, why do we have\nthese additional calls?\n\n```\n(gdb) bt\n#0 lookup_object (r=r@entry=0x5555558b8cc0 ,\noid=oid@entry=0x5555558b8980 ) at object.c:92\n#1 0x0000555555665572 in lookup_commit (r=0x5555558b8cc0 ,\noid=0x5555558b8980 ) at commit.c:62\n#2 0x00005555556edff5 in parse_object_buffer (r=0x5555558b8cc0\n, oid=oid@entry=0x5555558b8980 , type=OBJ_COMMIT,\nsize=788, buffer=0x5555558d0080, eaten_p=eaten_p@entry=0x7fffffffcc0c)\n at object.c:214\n#3 0x000055555571da42 in get_object (ref=ref@entry=0x7fffffffcf30,\nderef=deref@entry=0, obj=obj@entry=0x7fffffffcc90,\noi=oi@entry=0x5555558b8980 , err=err@entry=0x7fffffffcf10)\n at ref-filter.c:1774\n#4 0x000055555571fdc2 in populate_value\n(ref=ref@entry=0x7fffffffcf30, err=err@entry=0x7fffffffcf10) at\nref-filter.c:1999\n#5 0x00005555557202eb in get_ref_atom_value\n(ref=ref@entry=0x7fffffffcf30, atom=0, v=v@entry=0x7fffffffcea8,\nerr=err@entry=0x7fffffffcf10) at ref-filter.c:2033\n#6 0x00005555557212d6 in format_ref_array_item\n(info=info@entry=0x7fffffffcf30, format=format@entry=0x7fffffffd0f0,\nfinal_buf=final_buf@entry=0x7fffffffd060,\n error_buf=error_buf@entry=0x7fffffffcf10) at ref-filter.c:2627\n#7 0x00005555555859d8 in batch_object_write (scratch=0x7fffffffd060,\nopt=0x7fffffffd0d0, data=, obj_name=0x0) at\nbuiltin/cat-file.c:224\n```\n\nAfter printing the call stack of `lookup_object()`, we can know that\nthe `parse_buffer()`\nis calling them. A very straightforward idea, can we avoid calling\nthis function?\n\nIn `parse_object_buffer()`, `parse_blob_buffer()`, ``parse_tree_buffer()`,\n`parse_commit_buffer()`, and `parse_tag_buffer()` parse the object\ndata, and then store\nit in `struct object *obj`, finally return it to the caller.\n\n`get_object()` will feed the `obj` to `grab_values()`, and then\n`grab_values()` will feed the\n`obj` to `grab_tag_values()`, `grab_commit_values()`, which can fill\nthe object info in `obj` to\nimplement some atom, e.g. `%(tag)`, `%(type)`, `%(object)`, `%(tree)`,\n`%(numparent)`,`%(parent)`.\nIt is worth noting that `%(objectname)`, `%(objecttype)`,\n`%(objectsize)`, `%(deltabase)`, `%(rest)`,\n`%(raw)` are did not appear in them, this means that we can avoid\nparsing object buffer when we\ndon't use those atoms which require `obj`'s information!\n\nAfter some processing and adaptation, I made the patch which can skip\n`parse_object_buffer()`\nin some cases, this is the result of the performance test of\n`t/perf/p1006-cat-file.sh`:\n\n```\nTest HEAD~ HEAD\n------------------------------------------------------------------------------------\n1006.2: cat-file --batch-check 0.10(0.09+0.00)\n0.11(0.10+0.00) +10.0%\n1006.3: cat-file --batch-check with atoms 0.09(0.08+0.01)\n0.09(0.06+0.03) +0.0%\n1006.4: cat-file --batch 0.62(0.58+0.04)\n0.57(0.54+0.03) -8.1%\n1006.5: cat-file --batch with atoms 0.63(0.60+0.02)\n0.52(0.49+0.02) -17.5%\n```\n\nWe can see that the performance of `git cat-file --batch` has been a\ncertain improvement!\n\nTell a joke: removing 1984531500 if checking can reduce the startup\ntime of GTA5 by 70%. :-D\n\nCurrently the patch has not been submitted to the mailing list, let us\nwait a bit...\n\n--\nZheNing Hu\n"},{"id":"431662","messageId":"CAP8UFD16pqERdF4Mdq=_ktvFB1rcXwB9PRy8TjNq=ThH5HbeqA@mail.gmail.com","threadId":"56191","inReplyTo":"CAOLTT8RwwMF07f=XxWN=zGsPU0VQ8FqPVdyepQp78Ei8WZpSrw@mail.gmail.com","subject":"Re: [GSoC] Git Blog 11","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-08-02T06:25:35Z","receivedAt":"2021-08-02T06:25:49Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sun, Aug 1, 2021 at 8:45 AM ZheNing Hu wrote:\n>\n> My eleventh week blog finished:\n> The web version is here:\n> https://adlternative.github.io/GSOC-Git-Blog-11/\n\nGreat, thanks!\n\n> ### Attempt to optimize performance\n>\n> This week, at the prompt of my mentor Christian, I used `gprof` for some\n> performance tests about `git cat-file --batch`:\n> [Re: [GSOC] How to improve the performance of git cat-file --batch]\n> (https://lore.kernel.org/git/CAOLTT8TdL7UhfVSOzbpmo-WFNrcKwmy=E720tNt4KM9o_p=keg@mail.gmail.com/)\n\n[...]\n\n> Here is an amazing fact:\n>\n> The number of calls to `lookup_object()` before and after using my\n> patch are 0 and\n> 522709 respectively. Therefore, I am very surprised, why do we have\n> these additional calls?\n\n> After printing the call stack of `lookup_object()`, we can know that\n> the `parse_buffer()`\n> is calling them.\n\ns/the `parse_buffer()`/`parse_buffer()`/\n\nor\n\ns/the `parse_buffer()`/the `parse_buffer()` function/\n\nAlso: s/them/it/\n\n> A very straightforward idea, can we avoid calling\n> this function?\n>\n> In `parse_object_buffer()`, `parse_blob_buffer()`, ``parse_tree_buffer()`,\n> `parse_commit_buffer()`, and `parse_tag_buffer()` parse the object\n\ns/parse/we parse/\n\n> data, and then store\n> it in `struct object *obj`, finally return it to the caller.\n\nMaybe: s/finally/and finally/\n\n> `get_object()` will feed the `obj` to `grab_values()`, and then\n> `grab_values()` will feed the\n> `obj` to `grab_tag_values()`, `grab_commit_values()`, which can fill\n> the object info in `obj` to\n> implement some atom, e.g. `%(tag)`, `%(type)`, `%(object)`, `%(tree)`,\n> `%(numparent)`,`%(parent)`.\n> It is worth noting that `%(objectname)`, `%(objecttype)`,\n> `%(objectsize)`, `%(deltabase)`, `%(rest)`,\n> `%(raw)` are did not appear in them, this means that we can avoid\n\ns/are did not/don't/\n\n> parsing object buffer when we\n> don't use those atoms which require `obj`'s information!\n>\n> After some processing and adaptation, I made the patch which can skip\n\ns/the patch/a patch/\n\n> `parse_object_buffer()`\n> in some cases, this is the result of the performance test of\n> `t/perf/p1006-cat-file.sh`:\n>\n> ```\n> Test HEAD~ HEAD\n> ------------------------------------------------------------------------------------\n> 1006.2: cat-file --batch-check 0.10(0.09+0.00)\n> 0.11(0.10+0.00) +10.0%\n> 1006.3: cat-file --batch-check with atoms 0.09(0.08+0.01)\n> 0.09(0.06+0.03) +0.0%\n> 1006.4: cat-file --batch 0.62(0.58+0.04)\n> 0.57(0.54+0.03) -8.1%\n> 1006.5: cat-file --batch with atoms 0.63(0.60+0.02)\n> 0.52(0.49+0.02) -17.5%\n> ```\n>\n> We can see that the performance of `git cat-file --batch` has been a\n> certain improvement!\n\nYeah, sure -8.1% or -17.5% is really nice! But why +10.0% for\n`cat-file --batch-check`?\n\n> Tell a joke: removing 1984531500 if checking can reduce the startup\n> time of GTA5 by 70%. :-D\n\ns/if checking/checks/\n\nAs this joke refers to:\n\nhttps://rockstarintel.com/a-fan-reduces-gta-online-loading-times-by-70\n\nit might be nice to add a link to help people like me who didn't know\nabout this and had to google it.\n\n> Currently the patch has not been submitted to the mailing list, let us\n> wait a bit...\n\nLooking forward to it...\n"},{"id":"431794","messageId":"CAOLTT8SaXqWF3AJssqudrDK-KZN3oESO7Xh0JTYB_=HR9L9sHg@mail.gmail.com","threadId":"56191","inReplyTo":"CAP8UFD16pqERdF4Mdq=_ktvFB1rcXwB9PRy8TjNq=ThH5HbeqA@mail.gmail.com","subject":"Re: [GSoC] Git Blog 11","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-03T02:37:06Z","receivedAt":"2021-08-03T02:36:25Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年8月2日周一 下午2:25写道：\n>\n> On Sun, Aug 1, 2021 at 8:45 AM ZheNing Hu wrote:\n>\n> > in some cases, this is the result of the performance test of\n> > `t/perf/p1006-cat-file.sh`:\n> >\n> > ```\n> > Test HEAD~ HEAD\n> > ------------------------------------------------------------------------------------\n> > 1006.2: cat-file --batch-check 0.10(0.09+0.00)\n> > 0.11(0.10+0.00) +10.0%\n> > 1006.3: cat-file --batch-check with atoms 0.09(0.08+0.01)\n> > 0.09(0.06+0.03) +0.0%\n> > 1006.4: cat-file --batch 0.62(0.58+0.04)\n> > 0.57(0.54+0.03) -8.1%\n> > 1006.5: cat-file --batch with atoms 0.63(0.60+0.02)\n> > 0.52(0.49+0.02) -17.5%\n> > ```\n> >\n> > We can see that the performance of `git cat-file --batch` has been a\n> > certain improvement!\n>\n> Yeah, sure -8.1% or -17.5% is really nice! But why +10.0% for\n> `cat-file --batch-check`?\n>\n\nI think it's not very important. Because our optimization is skipping\nparse_object_buffer(), git cat-file --batch-check will not set oi->contentp\nby default, parse_object_buffer() will not be executed, Therefore, we did\nnot optimize `git cat-file --batch-check` at all. 10% may be small enough\nfor git cat-file --batch-check. The noise of environment even will cover it...\n\n> > Tell a joke: removing 1984531500 if checking can reduce the startup\n> > time of GTA5 by 70%. :-D\n>\n> s/if checking/checks/\n>\n> As this joke refers to:\n>\n> https://rockstarintel.com/a-fan-reduces-gta-online-loading-times-by-70\n>\n> it might be nice to add a link to help people like me who didn't know\n> about this and had to google it.\n>\n\nEhhh, makes sense.\n\n> > Currently the patch has not been submitted to the mailing list, let us\n> > wait a bit...\n>\n> Looking forward to it...\n\nI am afraid we need to wait until ref-filter-raw-data merged into master.\n\nThe preview version is here:\nhttps://github.com/adlternative/git/commit/6fcf51c4b1cbb510dac775a6fbf3db361bc1c327\n\nThanks.\n--\nZheNing Hu\n"},{"id":"431795","messageId":"CAOLTT8QDo1A4tRqU0w-yas8OxrHB6uQ_PECswUQtQ7yY2-iWgw@mail.gmail.com","threadId":"56191","inReplyTo":"CAOLTT8SaXqWF3AJssqudrDK-KZN3oESO7Xh0JTYB_=HR9L9sHg@mail.gmail.com","subject":"Re: [GSoC] Git Blog 11","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-03T02:49:18Z","receivedAt":"2021-08-03T02:48:39Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"ZheNing Hu 于2021年8月3日周二 上午10:37写道：\n>\n> Christian Couder 于2021年8月2日周一 下午2:25写道：\n> >\n> > On Sun, Aug 1, 2021 at 8:45 AM ZheNing Hu wrote:\n> >\n> > > in some cases, this is the result of the performance test of\n> > > `t/perf/p1006-cat-file.sh`:\n> > >\n> > > ```\n> > > Test HEAD~ HEAD\n> > > ------------------------------------------------------------------------------------\n> > > 1006.2: cat-file --batch-check 0.10(0.09+0.00)\n> > > 0.11(0.10+0.00) +10.0%\n> > > 1006.3: cat-file --batch-check with atoms 0.09(0.08+0.01)\n> > > 0.09(0.06+0.03) +0.0%\n> > > 1006.4: cat-file --batch 0.62(0.58+0.04)\n> > > 0.57(0.54+0.03) -8.1%\n> > > 1006.5: cat-file --batch with atoms 0.63(0.60+0.02)\n> > > 0.52(0.49+0.02) -17.5%\n> > > ```\n> > >\n> > > We can see that the performance of `git cat-file --batch` has been a\n> > > certain improvement!\n> >\n> > Yeah, sure -8.1% or -17.5% is really nice! But why +10.0% for\n> > `cat-file --batch-check`?\n> >\n>\n> I think it's not very important. Because our optimization is skipping\n> parse_object_buffer(), git cat-file --batch-check will not set oi->contentp\n> by default, parse_object_buffer() will not be executed, Therefore, we did\n> not optimize `git cat-file --batch-check` at all. 10% may be small enough\n> for git cat-file --batch-check. The noise of environment even will cover it...\n>\n\nBy the way, its performance may still be worse than \"upstream/master\", but it\nwill be better than before optimization.\n\nTest HEAD~ this tree\n------------------------------------------------------------------------------------\n1006.2: cat-file --batch-check 0.10(0.09+0.01)\n0.09(0.08+0.01) -10.0%\n1006.3: cat-file --batch-check with atoms 0.09(0.07+0.02)\n0.08(0.05+0.03) -11.1%\n1006.4: cat-file --batch 0.61(0.59+0.02)\n0.53(0.51+0.02) -13.1%\n1006.5: cat-file --batch with atoms 0.60(0.57+0.02)\n0.52(0.49+0.03) -13.3%\n\nTest upstream/master this\ntree\n------------------------------------------------------------------------------------\n1006.2: cat-file --batch-check 0.08(0.07+0.01)\n0.10(0.07+0.02) +25.0%\n1006.3: cat-file --batch-check with atoms 0.06(0.05+0.01)\n0.08(0.08+0.00) +33.3%\n1006.4: cat-file --batch 0.49(0.46+0.03)\n0.53(0.50+0.03) +8.2%\n1006.5: cat-file --batch with atoms 0.48(0.45+0.03)\n0.51(0.48+0.02) +6.3%\n\nTest upstream/master HEAD~\n------------------------------------------------------------------------------------\n1006.2: cat-file --batch-check 0.08(0.07+0.01)\n0.10(0.07+0.03) +25.0%\n1006.3: cat-file --batch-check with atoms 0.06(0.05+0.00)\n0.08(0.08+0.00) +33.3%\n1006.4: cat-file --batch 0.48(0.46+0.02)\n0.60(0.57+0.03) +25.0%\n1006.5: cat-file --batch with atoms 0.47(0.44+0.03)\n0.63(0.59+0.04) +34.0%\n\nThanks.\n--\nZheNing Hu\n"},{"id":"431944","messageId":"CAP8UFD3E9oR9E4S=f8iReKOnvVO_WrXVziyztHZJCiScUAxDRg@mail.gmail.com","threadId":"56191","inReplyTo":"CAOLTT8QDo1A4tRqU0w-yas8OxrHB6uQ_PECswUQtQ7yY2-iWgw@mail.gmail.com","subject":"Re: [GSoC] Git Blog 11","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-08-04T08:56:51Z","receivedAt":"2021-08-04T08:57:05Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Aug 3, 2021 at 4:48 AM ZheNing Hu wrote:\n>\n> ZheNing Hu 于2021年8月3日周二 上午10:37写道：\n> >\n> > Christian Couder 于2021年8月2日周一 下午2:25写道：\n> > >\n> > > On Sun, Aug 1, 2021 at 8:45 AM ZheNing Hu wrote:\n> > >\n> > > > in some cases, this is the result of the performance test of\n> > > > `t/perf/p1006-cat-file.sh`:\n> > > >\n> > > > ```\n> > > > Test HEAD~ HEAD\n> > > > ------------------------------------------------------------------------------------\n> > > > 1006.2: cat-file --batch-check 0.10(0.09+0.00)\n> > > > 0.11(0.10+0.00) +10.0%\n> > > > 1006.3: cat-file --batch-check with atoms 0.09(0.08+0.01)\n> > > > 0.09(0.06+0.03) +0.0%\n> > > > 1006.4: cat-file --batch 0.62(0.58+0.04)\n> > > > 0.57(0.54+0.03) -8.1%\n> > > > 1006.5: cat-file --batch with atoms 0.63(0.60+0.02)\n> > > > 0.52(0.49+0.02) -17.5%\n> > > > ```\n> > > >\n> > > > We can see that the performance of `git cat-file --batch` has been a\n> > > > certain improvement!\n> > >\n> > > Yeah, sure -8.1% or -17.5% is really nice! But why +10.0% for\n> > > `cat-file --batch-check`?\n> >\n> > I think it's not very important. Because our optimization is skipping\n> > parse_object_buffer(), git cat-file --batch-check will not set oi->contentp\n> > by default, parse_object_buffer() will not be executed.\n\nDo you think that if git cat-file --batch-check would set\noi->contentp, there would be no performance regression for `cat-file\n--batch-check`?\nCould you test that?\n\n> > Therefore, we did\n> > not optimize `git cat-file --batch-check` at all. 10% may be small enough\n> > for git cat-file --batch-check. The noise of environment even will cover it...\n>\n> By the way, its performance may still be worse than \"upstream/master\", but it\n> will be better than before optimization.\n\nNice that there is some improvement, but it would be better if it was\nsimilar to \"upstream/master\".\n\n> Test HEAD~ this tree\n> ------------------------------------------------------------------------------------\n> 1006.2: cat-file --batch-check 0.10(0.09+0.01)\n> 0.09(0.08+0.01) -10.0%\n> 1006.3: cat-file --batch-check with atoms 0.09(0.07+0.02)\n> 0.08(0.05+0.03) -11.1%\n> 1006.4: cat-file --batch 0.61(0.59+0.02)\n> 0.53(0.51+0.02) -13.1%\n> 1006.5: cat-file --batch with atoms 0.60(0.57+0.02)\n> 0.52(0.49+0.03) -13.3%\n\nYeah, your patch seems to be an overall improvement when the\nref-filter code is used.\n\n> Test upstream/master this\n> tree\n> ------------------------------------------------------------------------------------\n> 1006.2: cat-file --batch-check 0.08(0.07+0.01)\n> 0.10(0.07+0.02) +25.0%\n> 1006.3: cat-file --batch-check with atoms 0.06(0.05+0.01)\n> 0.08(0.08+0.00) +33.3%\n> 1006.4: cat-file --batch 0.49(0.46+0.03)\n> 0.53(0.50+0.03) +8.2%\n> 1006.5: cat-file --batch with atoms 0.48(0.45+0.03)\n> 0.51(0.48+0.02) +6.3%\n\nThis means that some further performance improvements are still needed\nboth for --batch and --batch-check though.\n\nHave you tried to see, using gprof or something else, what is still\ndegrading the performance compared to when the ref-filter code isn't\nused?\n"},{"id":"432051","messageId":"CAOLTT8Sd6OCU_Ufrhqstz-Mw0Ej=9F2Y20BjPOpkgsuB5D-4Nw@mail.gmail.com","threadId":"56191","inReplyTo":"CAP8UFD3E9oR9E4S=f8iReKOnvVO_WrXVziyztHZJCiScUAxDRg@mail.gmail.com","subject":"Re: [GSoC] Git Blog 11","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-05T04:50:21Z","receivedAt":"2021-08-05T04:49:39Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年8月4日周三 下午4:57写道：\n>\n> On Tue, Aug 3, 2021 at 4:48 AM ZheNing Hu wrote:\n> >\n> > ZheNing Hu 于2021年8月3日周二 上午10:37写道：\n> > >\n> > > Christian Couder 于2021年8月2日周一 下午2:25写道：\n> > > >\n> > > > On Sun, Aug 1, 2021 at 8:45 AM ZheNing Hu wrote:\n> > > >\n> > > > > in some cases, this is the result of the performance test of\n> > > > > `t/perf/p1006-cat-file.sh`:\n> > > > >\n> > > > > ```\n> > > > > Test HEAD~ HEAD\n> > > > > ------------------------------------------------------------------------------------\n> > > > > 1006.2: cat-file --batch-check 0.10(0.09+0.00)\n> > > > > 0.11(0.10+0.00) +10.0%\n> > > > > 1006.3: cat-file --batch-check with atoms 0.09(0.08+0.01)\n> > > > > 0.09(0.06+0.03) +0.0%\n> > > > > 1006.4: cat-file --batch 0.62(0.58+0.04)\n> > > > > 0.57(0.54+0.03) -8.1%\n> > > > > 1006.5: cat-file --batch with atoms 0.63(0.60+0.02)\n> > > > > 0.52(0.49+0.02) -17.5%\n> > > > > ```\n> > > > >\n> > > > > We can see that the performance of `git cat-file --batch` has been a\n> > > > > certain improvement!\n> > > >\n> > > > Yeah, sure -8.1% or -17.5% is really nice! But why +10.0% for\n> > > > `cat-file --batch-check`?\n> > >\n> > > I think it's not very important. Because our optimization is skipping\n> > > parse_object_buffer(), git cat-file --batch-check will not set oi->contentp\n> > > by default, parse_object_buffer() will not be executed.\n>\n> Do you think that if git cat-file --batch-check would set\n> oi->contentp, there would be no performance regression for `cat-file\n> --batch-check`?\n> Could you test that?\n>\n\nOh, I mean that if git cat-file --batch-check with its default format\n\"%(objectname) %(objecttype)\n%(objectsize)\", it will not have any optimization; But if git cat-file\n--batch set with \"%(contents)\" or\nsome other atoms, it will indeed be optimized. See 1006.4:\n\nTest this tree\nHEAD~\n---------------------------------------------------------------------------------------------\n1006.2: cat-file --batch-check 0.15(0.12+0.02)\n0.15(0.13+0.01) +0.0%\n1006.3: cat-file --batch-check with basic atoms 0.12(0.10+0.01)\n0.12(0.10+0.02) +0.0%\n1006.4: cat-file --batch-check with contents atoms 0.66(0.63+0.02)\n0.75(0.72+0.02) +13.6%\n1006.5: cat-file --batch 0.61(0.57+0.04)\n0.70(0.65+0.05) +14.8%\n1006.6: cat-file --batch with atoms 0.58(0.57+0.01)\n0.67(0.63+0.03) +15.5%\n\n> > > Therefore, we did\n> > > not optimize `git cat-file --batch-check` at all. 10% may be small enough\n> > > for git cat-file --batch-check. The noise of environment even will cover it...\n> >\n> > By the way, its performance may still be worse than \"upstream/master\", but it\n> > will be better than before optimization.\n>\n> Nice that there is some improvement, but it would be better if it was\n> similar to \"upstream/master\".\n>\n\nAgree.\n\n> > Test HEAD~ this tree\n> > ------------------------------------------------------------------------------------\n> > 1006.2: cat-file --batch-check 0.10(0.09+0.01)\n> > 0.09(0.08+0.01) -10.0%\n> > 1006.3: cat-file --batch-check with atoms 0.09(0.07+0.02)\n> > 0.08(0.05+0.03) -11.1%\n> > 1006.4: cat-file --batch 0.61(0.59+0.02)\n> > 0.53(0.51+0.02) -13.1%\n> > 1006.5: cat-file --batch with atoms 0.60(0.57+0.02)\n> > 0.52(0.49+0.03) -13.3%\n>\n> Yeah, your patch seems to be an overall improvement when the\n> ref-filter code is used.\n>\n> > Test upstream/master this\n> > tree\n> > ------------------------------------------------------------------------------------\n> > 1006.2: cat-file --batch-check 0.08(0.07+0.01)\n> > 0.10(0.07+0.02) +25.0%\n> > 1006.3: cat-file --batch-check with atoms 0.06(0.05+0.01)\n> > 0.08(0.08+0.00) +33.3%\n> > 1006.4: cat-file --batch 0.49(0.46+0.03)\n> > 0.53(0.50+0.03) +8.2%\n> > 1006.5: cat-file --batch with atoms 0.48(0.45+0.03)\n> > 0.51(0.48+0.02) +6.3%\n>\n> This means that some further performance improvements are still needed\n> both for --batch and --batch-check though.\n>\n> Have you tried to see, using gprof or something else, what is still\n> degrading the performance compared to when the ref-filter code isn't\n> used?\n\nYeah, gprof show that Number of calls of strbuf_add(), xstrdup() has increased\nafter using the logic of ref-filter. But at the same time, I noticed\nthat grab_person()\nseems to be an area worth optimizing. grab_person() uses its parameter\n\"const char *who\"\nfor type comparison, But after we added `enum atom_type` to\nref-filter, We can use it\nfor some comparisons. And there are two for() loops in grab_person(),\nand we can merge\nthem into one. With this patch [1], there is a slight improvement in\nperformance.\n\nTest this tree\nHEAD~\n-------------------------------------------------------------------------------------------\n1006.2: cat-file --batch-check 0.14(0.13+0.01)\n0.15(0.14+0.01) +7.1%\n1006.3: cat-file --batch-check with atoms 0.12(0.10+0.01)\n0.12(0.09+0.02) +0.0%\n1006.4: cat-file --batch-check with contents atom 0.66(0.65+0.01)\n0.66(0.64+0.02) +0.0%\n1006.5: cat-file --batch 0.60(0.57+0.02)\n0.60(0.57+0.03) +0.0%\n1006.6: cat-file --batch with atoms 0.58(0.53+0.04)\n0.58(0.56+0.02) +0.0%\n1006.7: cat-file --batch with person atoms 0.59(0.57+0.02)\n0.60(0.56+0.04) +1.7%\n\nIt’s also worth mentioning that I found that grab_person() seems to be doing\nrepeated parsing which parse_object_buffer() may already be done.\nparse_commit_buffer()\nand parse_tag_buffer() have parsed part of the content of the object,\nand used by\ngrab_tag_values() and grab_commit_values(). For the time being, I\nthink this is a kind of\nshallow parsing, if we can let parse_object_buffer() do in-depth\nparsing, it would be great.\nWe can save a lot of work in grab_person()... Of course this may be a\nlittle difficult.\n\nThanks.\n--\nZheNing Hu\n\n[1]: https://github.com/adlternative/git/commit/cec0ee72e64d651c01d7a2a7fe17a4adab1ef0de\n"}]}