{"thread":{"id":"59385","subject":"Git fetch slow on local repository with 600k refs","startedAt":"2023-03-13T11:57:14Z","lastAt":"2023-03-14T14:51:07Z","messageCount":4,"participants":["程洋","Bagas Sanjaya","Sean Allred"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"473452","messageId":"e28a23e8eb044d26947462b8619e88bd@xiaomi.com","threadId":"59385","inReplyTo":null,"subject":"Git fetch slow on local repository with 600k refs","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2023-03-13T11:54:23Z","receivedAt":"2023-03-13T11:57:14Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"We're holding a Gerrit server cluster. And uses pull-replication plugin to sync changes between master and slave.\nWhen a change is pushed to master, it notify the slave, and slave fetch it from master.\n\nBut we found in a big repository with 600k refs. Fetch takes 5-10 seconds even if fetching a 1 byte change. Here is the GIT_TRACE2_PERF\nI did an experiment to fetch a ref that my slave already have. And we can find git rev-list takes 2 seconds to perform. (I guess it try to find remote object from reachable objects of local refs one by one)\nIs there anyway to optimize such situation?\n\n19:12:55.931180 common-main.c:48 | d0 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n19:12:55.931215 common-main.c:49 | d0 | main | start | | 0.000335 | | | git fetch --no-tags git://10.13.8.10/miui/gerrit/base-test.git refs/changes/27/2741927/1:refs/changes/27/2741927/1\n19:12:55.931302 compat/linux/procinfo.c:170 | d0 | main | cmd_ancestry | | | | | ancestry:[bash sudo bash miauthd miauthd systemd]\n19:12:55.931381 git.c:456 | d0 | main | cmd_name | | | | | fetch (fetch)\n19:12:55.931566 builtin/fetch.c:1579 | d0 | main | region_enter | r0 | 0.000692 | | fetch | label:remote_refs\n19:12:55.936781 connect.c:167 | d0 | main | data | | 0.005907 | 0.005215 | transfer | ..negotiated-version:2\n19:12:55.940447 builtin/fetch.c:1582 | d0 | main | region_leave | r0 | 0.009573 | 0.008881 | fetch | label:remote_refs\n19:12:56.221133 run-command.c:739 | d0 | main | child_start | | 0.290252 | | | [ch0] class:? argv:[git rev-list --objects --stdin --not --all --quiet --alternate-refs --unsorted-input]\n19:12:58.014792 run-command.c:995 | d0 | main | child_exit | | 2.083899 | 1.793647 | | [ch0] pid:81860 code:0\n19:12:58.014855 builtin/fetch.c:1321 | d0 | main | region_enter | r0 | 2.083980 | | fetch | label:consume_refs\n19:12:58.015412 builtin/fetch.c:1326 | d0 | main | region_leave | r0 | 2.084538 | 0.000558 | fetch | label:consume_refs\n19:12:58.015466 run-command.c:739 | d0 | main | child_start | | 2.084590 | | | [ch1] class:? argv:[git maintenance run --auto --no-quiet]\n19:12:58.018879 common-main.c:48 | d1 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n19:12:58.018911 common-main.c:49 | d1 | main | start | | 0.000324 | | | /usr/libexec/git-core/git maintenance run --auto --no-quiet\n19:12:58.019011 compat/linux/procinfo.c:170 | d1 | main | cmd_ancestry | | | | | ancestry:[git bash sudo bash miauthd miauthd systemd]\n19:12:58.019087 git.c:456 | d1 | main | cmd_name | | | | | maintenance (fetch/maintenance)\n19:12:58.019276 git.c:714 | d1 | main | exit | | 0.000690 | | | code:0\n19:12:58.019284 trace2/tr2_tgt_perf.c:213 | d1 | main | atexit | | 0.000698 | | | code:0\n19:12:58.019386 run-command.c:995 | d0 | main | child_exit | | 2.088507 | 0.003917 | | [ch1] pid:81878 code:0\n19:12:58.019411 git.c:714 | d0 | main | exit | | 2.088538 | | | code:0\n19:12:58.019419 trace2/tr2_tgt_perf.c:213 | d0 | main | atexit | | 2.088545 | | | code:0\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"473494","messageId":"73a7f949-e828-fd9c-4be5-63f60b53eb41@gmail.com","threadId":"59385","inReplyTo":"e28a23e8eb044d26947462b8619e88bd@xiaomi.com","subject":"Re: Git fetch slow on local repository with 600k refs","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2023-03-14T08:22:14Z","receivedAt":"2023-03-14T08:24:10Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 3/13/23 18:54, 程洋 wrote:\n> 19:12:55.931180 common-main.c:48 | d0 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n> 19:12:55.931215 common-main.c:49 | d0 | main | start | | 0.000335 | | | git fetch --no-tags git://10.13.8.10/miui/gerrit/base-test.git refs/changes/27/2741927/1:refs/changes/27/2741927/1\n> 19:12:55.931302 compat/linux/procinfo.c:170 | d0 | main | cmd_ancestry | | | | | ancestry:[bash sudo bash miauthd miauthd systemd]\n> 19:12:55.931381 git.c:456 | d0 | main | cmd_name | | | | | fetch (fetch)\n> 19:12:55.931566 builtin/fetch.c:1579 | d0 | main | region_enter | r0 | 0.000692 | | fetch | label:remote_refs\n> 19:12:55.936781 connect.c:167 | d0 | main | data | | 0.005907 | 0.005215 | transfer | ..negotiated-version:2\n> 19:12:55.940447 builtin/fetch.c:1582 | d0 | main | region_leave | r0 | 0.009573 | 0.008881 | fetch | label:remote_refs\n> 19:12:56.221133 run-command.c:739 | d0 | main | child_start | | 0.290252 | | | [ch0] class:? argv:[git rev-list --objects --stdin --not --all --quiet --alternate-refs --unsorted-input]\n> 19:12:58.014792 run-command.c:995 | d0 | main | child_exit | | 2.083899 | 1.793647 | | [ch0] pid:81860 code:0\n> 19:12:58.014855 builtin/fetch.c:1321 | d0 | main | region_enter | r0 | 2.083980 | | fetch | label:consume_refs\n> 19:12:58.015412 builtin/fetch.c:1326 | d0 | main | region_leave | r0 | 2.084538 | 0.000558 | fetch | label:consume_refs\n> 19:12:58.015466 run-command.c:739 | d0 | main | child_start | | 2.084590 | | | [ch1] class:? argv:[git maintenance run --auto --no-quiet]\n> 19:12:58.018879 common-main.c:48 | d1 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n> 19:12:58.018911 common-main.c:49 | d1 | main | start | | 0.000324 | | | /usr/libexec/git-core/git maintenance run --auto --no-quiet\n> 19:12:58.019011 compat/linux/procinfo.c:170 | d1 | main | cmd_ancestry | | | | | ancestry:[git bash sudo bash miauthd miauthd systemd]\n> 19:12:58.019087 git.c:456 | d1 | main | cmd_name | | | | | maintenance (fetch/maintenance)\n> 19:12:58.019276 git.c:714 | d1 | main | exit | | 0.000690 | | | code:0\n> 19:12:58.019284 trace2/tr2_tgt_perf.c:213 | d1 | main | atexit | | 0.000698 | | | code:0\n> 19:12:58.019386 run-command.c:995 | d0 | main | child_exit | | 2.088507 | 0.003917 | | [ch1] pid:81878 code:0\n> 19:12:58.019411 git.c:714 | d0 | main | exit | | 2.088538 | | | code:0\n> 19:12:58.019419 trace2/tr2_tgt_perf.c:213 | d0 | main | atexit | | 2.088545 | | | code:0\n\nFrom above, I see that the hot paths are `git maintenance run` and\n`git rev-list`, right?\n\nNext time, try to send only plain-text email in this ML, as vger isn't\nhappy with HTML emails (most likely spam).\n\nThanks.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n\n"},{"id":"473497","messageId":"0da2848da9254b64a383695c7a4bc91b@xiaomi.com","threadId":"59385","inReplyTo":"73a7f949-e828-fd9c-4be5-63f60b53eb41@gmail.com","subject":"RE: [External Mail]Re: Git fetch slow on local repository with 600k refs","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2023-03-14T11:13:42Z","receivedAt":"2023-03-14T11:16:20Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"\n\n> -----Original Message-----\n> From: Bagas Sanjaya \n> Sent: Tuesday, March 14, 2023 4:22 PM\n> To: 程洋 ; git@vger.kernel.org\n> Cc: 姜浩哲 \n> Subject: [External Mail]Re: Git fetch slow on local repository with 600k refs\n>\n>\n> On 3/13/23 18:54, 程洋 wrote:\n> > 19:12:55.931180 common-main.c:48 | d0 | main | version\n> | | | | | 2.33.1.558.g2bd2f258f4.dirty\n> > 19:12:55.931215 common-main.c:49 | d0 | main | start |\n> | 0.000335 | | | git fetch --no-tags\n> git://10.13.8.10/miui/gerrit/base-test.git\n> refs/changes/27/2741927/1:refs/changes/27/2741927/1\n> > 19:12:55.931302 compat/linux/procinfo.c:170 | d0 | main |\n> cmd_ancestry | | | | | ancestry:[bash sudo bash miauthd\n> miauthd systemd]\n> > 19:12:55.931381 git.c:456 | d0 | main | cmd_name |\n> | | | | fetch (fetch)\n> > 19:12:55.931566 builtin/fetch.c:1579 | d0 | main |\n> region_enter | r0 | 0.000692 | | fetch | label:remote_refs\n> > 19:12:55.936781 connect.c:167 | d0 | main | data | |\n> 0.005907 | 0.005215 | transfer | ..negotiated-version:2\n> > 19:12:55.940447 builtin/fetch.c:1582 | d0 | main |\n> region_leave | r0 | 0.009573 | 0.008881 | fetch | label:remote_refs\n> > 19:12:56.221133 run-command.c:739 | d0 | main |\n> child_start | | 0.290252 | | | [ch0] class:? argv:[git rev-list --\n> objects --stdin --not --all --quiet --alternate-refs --unsorted-input]\n> > 19:12:58.014792 run-command.c:995 | d0 | main | child_exit\n> | | 2.083899 | 1.793647 | | [ch0] pid:81860 code:0\n> > 19:12:58.014855 builtin/fetch.c:1321 | d0 | main |\n> region_enter | r0 | 2.083980 | | fetch | label:consume_refs\n> > 19:12:58.015412 builtin/fetch.c:1326 | d0 | main |\n> region_leave | r0 | 2.084538 | 0.000558 | fetch | label:consume_refs\n> > 19:12:58.015466 run-command.c:739 | d0 | main |\n> child_start | | 2.084590 | | | [ch1] class:? argv:[git maintenance\n> run --auto --no-quiet]\n> > 19:12:58.018879 common-main.c:48 | d1 | main | version\n> | | | | | 2.33.1.558.g2bd2f258f4.dirty\n> > 19:12:58.018911 common-main.c:49 | d1 | main | start |\n> | 0.000324 | | | /usr/libexec/git-core/git maintenance run --auto\n> --no-quiet\n> > 19:12:58.019011 compat/linux/procinfo.c:170 | d1 | main |\n> cmd_ancestry | | | | | ancestry:[git bash sudo bash\n> miauthd miauthd systemd]\n> > 19:12:58.019087 git.c:456 | d1 | main | cmd_name |\n> | | | | maintenance (fetch/maintenance)\n> > 19:12:58.019276 git.c:714 | d1 | main | exit | |\n> 0.000690 | | | code:0\n> > 19:12:58.019284 trace2/tr2_tgt_perf.c:213 | d1 | main | atexit\n> | | 0.000698 | | | code:0\n> > 19:12:58.019386 run-command.c:995 | d0 | main | child_exit\n> | | 2.088507 | 0.003917 | | [ch1] pid:81878 code:0\n> > 19:12:58.019411 git.c:714 | d0 | main | exit | |\n> 2.088538 | | | code:0\n> > 19:12:58.019419 trace2/tr2_tgt_perf.c:213 | d0 | main | atexit\n> | | 2.088545 | | | code:0\n>\n> From above, I see that the hot paths are `git maintenance run` and `git rev-\n> list`, right?\n>\n> Next time, try to send only plain-text email in this ML, as vger isn't happy\n> with HTML emails (most likely spam).\n>\n> Thanks.\n>\n> --\n> An old man doll... just what I always wanted! - Clara\n\nI do send the email with outlook and choose the plaintext mode. Do you still see my mail as HTML?\n\nBTW, I didn't see any performance issue on maintenance, but only on git rev-list\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"473500","messageId":"m0wn3jxu7g.fsf@epic96565.epic.com","threadId":"59385","inReplyTo":"e28a23e8eb044d26947462b8619e88bd@xiaomi.com","subject":"Re: Git fetch slow on local repository with 600k refs","fromName":"Sean Allred","fromEmail":"allred.sean@gmail.com","sentAt":"2023-03-14T14:29:31Z","receivedAt":"2023-03-14T14:51:07Z","isPatch":false,"sender":{"key":"allred.sean@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2082195?v=4"},"body":"\n程洋 writes:\n\n> We're holding a Gerrit server cluster. And uses pull-replication\n> plugin to sync changes between master and slave.\n>\n> When a change is pushed to master, it notify the slave, and slave\n> fetch it from master.\n>\n> But we found in a big repository with 600k refs. Fetch takes 5-10\n> seconds even if fetching a 1 byte change. Here is the GIT_TRACE2_PERF\n>\n> I did an experiment to fetch a ref that my slave already have. And we\n> can find git rev-list takes 2 seconds to perform. (I guess it try to\n> find remote object from reachable objects of local refs one by one)\n>\n> Is there anyway to optimize such situation?\n\nDo you need all those refs as refs -- or are you just looking to keep\nthe commits?\n\nWe found a rather clever solution for the latter we're looking to\nupstream at some point to collect all refs into a single 'archive' ref\nthat collects commits in fake merge commits (there's no actual conflict\nresolution happening -- we just use the same tree over and over). We\nmake each commit message look like show-ref output. For example:\n\nA single ref (refs/archive) pointing to commit (A), with contents\n\n tree \n parent [... 500 other commits 'merged' in ...]\n author \n committer \n\n deadbeef0123456788... refs/tags/very/old/release-1\n deadbeef0123456789... refs/tags/very/old/release-2\n\nWhen we want to pull a ref out of the archive, we have a process in\nplace to do so. This keeps the total number of refs down and the\nfetch/push performance within acceptable limits.\n\n--\nSean Allred\n"}]}