{"thread":{"id":"56676","subject":"why git is so slow for a tiny git push?","startedAt":"2021-10-09T18:06:00Z","lastAt":"2021-11-25T02:55:37Z","messageCount":16,"participants":["程洋","Jeff King","Ævar Arnfjörð Bjarmason","Han-Wen Nienhuys"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"438414","messageId":"576b2f3e162e4f86992d8f4e680d0881@xiaomi.com","threadId":"56676","inReplyTo":"5a6f3e8f29f74c93bf3af5da636df973@xiaomi.com","subject":"why git is so slow for a tiny git push?","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2021-10-09T18:05:56Z","receivedAt":"2021-10-09T18:06:00Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"I have a really big repository which has 9m objects and maybe 300k refs.\nI noticed that git push is really slow for a tiny change. An example shows below\n\n3 objects which is only 7 kb takes 36 seconds to pack-objects (it's the time after i enable pack.usesparse)\nHowever if I manually call “pack-objects” with the exactly same objects SHA1. It only take less than 0.005 second\nWhat is really pass to “pack-objects” when I call “git push”?\n\nI read an article says git will enumerate all \"uninteresting objects\" to determine what to send. but i don't understand, in my case git should only enumerate objects between \"1a2d494b1b71469eebbd42aeabe1736bfa4b51fa..ddf3b84dca1aa4fe209a218380df1482af0d6b48\". It's insane. I have a master server and a slave server serve this repository to my users. And i have a cron job to push every change from master to slave. And i found my master server CPU is full all the time because of the push jobs\n\nIs there any solution?\n\n\nHere is my full output\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nwork@c5-miui-miuigit-slave15:~/repositories/miui/platform/frameworks/base2/base$ GIT_TRACE_PERFORMANCE=1 GIT_TRACE_PACKET=1 git push git://10.172.32.31/miui/platform/frameworks/base.git bsp-qcom-s:bsp-qcom-s\n.....\n.....\n.....\n.....\n01:53:00.595910 pkt-line.c:80 packet: push< 2d5eb3d37e2aa6659920bda688310584c70a37d0 refs/remotes/origin/v9-l-ido-stable2x\n01:53:00.595917 pkt-line.c:80 packet: push< 120ebf9b593c8f914981506b8390dd317fb8c9f1 refs/remotes/origin/c\n01:53:00.595954 pkt-line.c:80 packet: push< 90403256e6775476bd8dee31f28b31af9f7eac89 refs/remotes/origin/v9-o-d1s-dev-backup\n01:53:00.595962 pkt-line.c:80 packet: push< 9a78ddd5a4d6759585d29549466138291b665d35 refs/remotes/origin/v9-o-dipper-dev\n01:53:00.595969 pkt-line.c:80 packet: push< 343dfeb613e411e0ede4ec6327e2b171e39ff523 refs/remotes/origin/v9-o-nitrogen-dev\n01:53:00.595976 pkt-line.c:80 packet: push< 35e23e916c2dccd8603a6fe485ee56d688003ff5 refs/remotes/origin/v9-o-sagit-dev\n01:53:00.595984 pkt-line.c:80 packet: push< dc62ab5c28f7f6154d41d138301fe7f555db160b refs/remotes/origin/v9-o-sakura-dev\n01:53:00.595991 pkt-line.c:80 packet: push< 00b4763e8219de6e8a76b9ce24ecb3460030783f refs/remotes/origin/v9-o-scorpio-dev\n01:53:00.595998 pkt-line.c:80 packet: push< 2f7b3a268712ba187bb1d399c698cb1836c5d47a refs/remotes/origin/v9-o-sirius-dev\n01:53:00.596005 pkt-line.c:80 packet: push< e14f29817e4128463ceeccc531f313ae1b138780 refs/remotes/origin/v9-o-whyred-dev\n01:53:00.596012 pkt-line.c:80 packet: push< c7620c21cf3aade7666c448471306c1264032c5d refs/remotes/origin/v9-o-ysl-dev\n01:53:00.596022 pkt-line.c:80 packet: push< c1cbe43f1bc50dcd422ef354e694ea45ca8aa797 refs/remotes/origin/wt-p-laurus-native\n01:53:00.596029 pkt-line.c:80 packet: push< 3c736c7b701b9023f08c3641bcc77041e35d0eca refs/remotes/origin/wt-q-laurus-miui\n01:53:00.596036 pkt-line.c:80 packet: push< 3dc0351efe79817f5c8bf23564e3c483ff059833 refs/remotes/origin/wt-q-laurus-native\n01:53:00.596044 pkt-line.c:80 packet: push< 58ac901f9302e7f52325f4ce905cc2cbdfa310ca refs/remotes/origin/wt-r-evergreen\n01:53:00.596053 pkt-line.c:80 packet: push< 0000\n01:53:00.601910 pkt-line.c:80 packet: push> 1a2d494b1b71469eebbd42aeabe1736bfa4b51fa ddf3b84dca1aa4fe209a218380df1482af0d6b48 refs/heads/bsp-qcom-s\\0 report-status-v2 side-band-64k object-format=sha1 agent=git/2.32.0\n01:53:00.601959 pkt-line.c:80 packet: push> 0000\nEnumerating objects: 3, done.\nCounting objects: 100% (3/3), done.\nDelta compression using up to 40 threads\nCompressing objects: 100% (3/3), done.\nWriting objects: 100% (3/3), 7.24 KiB | 7.24 MiB/s, done.\nTotal 3 (delta 0), reused 0 (delta 0), pack-reused 0\n01:53:19.256506 trace.c:487 performance: 18.653337755 s: git command: /usr/libexec/git-core/git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress\n01:53:22.298695 pkt-line.c:80 packet: sideband< \\1000eunpack ok001dok refs/heads/bsp-qcom-s0000\n01:53:22.298735 pkt-line.c:80 packet: sideband< 0000\n01:53:22.298745 pkt-line.c:80 packet: push< unpack ok\n01:53:22.298770 pkt-line.c:80 packet: push< ok refs/heads/bsp-qcom-s\n01:53:22.298779 pkt-line.c:80 packet: push< 0000\nTo git://10.172.32.31/miui/platform/frameworks/base.git\n 1a2d494b1b71..ddf3b84dca1a bsp-qcom-s -> bsp-qcom-s\n01:53:22.316441 trace.c:487 performance: 22.883688573 s: git command: git push git://10.172.32.31/miui/platform/frameworks/base.git bsp-qcom-s:bsp-qcom-s\nwork@c5-miui-miuigit-slave15:~/repositories/miui/platform/frameworks/base2/base$ echo 1a2d494b1b71469eebbd42aeabe1736bfa4b51fa..ddf3b84dca1aa4fe209a218380df1482af0d6b48 > 1\nwork@c5-miui-miuigit-slave15:~/repositories/miui/platform/frameworks/base2/base$ time /usr/libexec/git-core/git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress < 1\nEnumerating objects: 5, done.\nCounting objects: 100% (5/5), done.\nDelta compression using up to 40 threads\nCompressing objects: 100% (3/3), done.\nWriting objects: 100% (3/3), 292 bytes | 292.00 KiB/s, done.\n\nTotal 3 (delta 2), reused 0 (delta 0), pack-reused 0\n\nreal 0m0.005s\nuser 0m0.000s\nsys 0m0.004s\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"438480","messageId":"YWRr9g32cMlIc37V@coredump.intra.peff.net","threadId":"56676","inReplyTo":"576b2f3e162e4f86992d8f4e680d0881@xiaomi.com","subject":"Re: why git is so slow for a tiny git push?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-10-11T16:53:10Z","receivedAt":"2021-10-11T16:53:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Oct 09, 2021 at 06:05:56PM +0000, 程洋 wrote:\n\n> I have a really big repository which has 9m objects and maybe 300k refs.\n> I noticed that git push is really slow for a tiny change. An example shows below\n> \n> 3 objects which is only 7 kb takes 36 seconds to pack-objects (it's the time after i enable pack.usesparse)\n> However if I manually call “pack-objects” with the exactly same objects SHA1. It only take less than 0.005 second\n> What is really pass to “pack-objects” when I call “git push”?\n\nDo you have an objects/pack/pack-*.bitmap file on the sending side?\n\nThe bitmap code is eager to produce an exact set difference between what\nis being sent and what the other side has. If you have incomplete bitmap\ncoverage (which is almost a certainty if you have 300k refs), it may do\na lot of traversal filling in the \"what the other side has\" part of the\nbitmap, even though it does not end up helping the final result in this\ncase.\n\nBitmaps are enabled by default on bare repos since Git v2.22.0. You can\noverride this with:\n\n git config repack.writeBitmaps false\n git gc\n\n(or if you don't want to do the gc, you can safely remove the '.bitmap'\nfile).\n\nI notice you used GIT_TRACE_PERFORMANCE below. Try GIT_TRACE2_PERF\ninstead, which goes into detail within particular processes. If this is\nrelated to bitmaps, I'd expect the time to go to the \"enumerate-objects\"\nregion.\n\n-Peff\n"},{"id":"438517","messageId":"ef2aa0d3ea8a4d98b910abdfd55191d0@xiaomi.com","threadId":"56676","inReplyTo":"YWRr9g32cMlIc37V@coredump.intra.peff.net","subject":"RE: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2021-10-12T08:04:44Z","receivedAt":"2021-10-12T08:05:18Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"I have bitmap indeed because my master server also serves as download server.\nHowever I'm using git 2.17.0, and I didn't set repack.writeBitmaps\n\nAlso I tried \"GIT_TRACE2_PERF\" and the it is \"enumerating objects\" cost most of the time.\nBut why bitmaps can cause push to be slow? Do you mean that if writeBitmaps is true, every push will regenerate bitmap file? If that's what you mean, what I see is the only bitmap file in my repo didn't change across time (the modify time is one month ago, long before I run the experiment)\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nI came up with an idea, since I found if I decrease the refs number, push goes much faster than before.\nSo I use receive.hiderefs to hide most of refs. And I comment \"reject_updates_to_hidden\" in receive-pack.c (because I need to update those hide refs, or add new refs)\n\n-----Original Message-----\nFrom: Jeff King \nSent: Tuesday, October 12, 2021 12:53 AM\nTo: 程洋 \nCc: git@vger.kernel.org\nSubject: [External Mail]Re: why git is so slow for a tiny git push?\n\n*This message originated from outside of XIAOMI. Please treat this email with caution*\n\n\nOn Sat, Oct 09, 2021 at 06:05:56PM +0000, 程洋 wrote:\n\n> I have a really big repository which has 9m objects and maybe 300k refs.\n> I noticed that git push is really slow for a tiny change. An example\n> shows below\n>\n> 3 objects which is only 7 kb takes 36 seconds to pack-objects (it's\n> the time after i enable pack.usesparse) However if I manually call\n> “pack-objects” with the exactly same objects SHA1. It only take less than 0.005 second What is really pass to “pack-objects” when I call “git push”?\n\nDo you have an objects/pack/pack-*.bitmap file on the sending side?\n\nThe bitmap code is eager to produce an exact set difference between what is being sent and what the other side has. If you have incomplete bitmap coverage (which is almost a certainty if you have 300k refs), it may do a lot of traversal filling in the \"what the other side has\" part of the bitmap, even though it does not end up helping the final result in this case.\n\nBitmaps are enabled by default on bare repos since Git v2.22.0. You can override this with:\n\n git config repack.writeBitmaps false\n git gc\n\n(or if you don't want to do the gc, you can safely remove the '.bitmap'\nfile).\n\nI notice you used GIT_TRACE_PERFORMANCE below. Try GIT_TRACE2_PERF instead, which goes into detail within particular processes. If this is related to bitmaps, I'd expect the time to go to the \"enumerate-objects\"\nregion.\n\n-Peff\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"438518","messageId":"YWVJyRJhRTdg39tX@coredump.intra.peff.net","threadId":"56676","inReplyTo":"ef2aa0d3ea8a4d98b910abdfd55191d0@xiaomi.com","subject":"Re: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-10-12T08:39:37Z","receivedAt":"2021-10-12T08:39:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 12, 2021 at 08:04:44AM +0000, 程洋 wrote:\n\n> I have bitmap indeed because my master server also serves as download server.\n> However I'm using git 2.17.0, and I didn't set repack.writeBitmaps\n\nOn that version and without the config, then perhaps you (or somebody)\npassed \"-b\" to git-repack.\n\n> But why bitmaps can cause push to be slow? Do you mean that if\n> writeBitmaps is true, every push will regenerate bitmap file? If\n> that's what you mean, what I see is the only bitmap file in my repo\n> didn't change across time (the modify time is one month ago, long\n> before I run the experiment)\n\nNo, it is not regenerating the on-disk bitmaps. But when deciding the\nset of objects to send, pack-objects will generate an internal bitmap\nwhich is the set difference of objects reachable from the pushed refs,\nminus objects reachable from the refs the other the other side told us\nthey had.\n\nIt uses the on-disk bitmaps as much as possible, but there may be\ncommits not covered by bitmaps (either because they were pushed since\nthe last repack which built bitmaps, or simply because it's too\nexpensive to put a bitmap on every commit, so we sprinkle them\nthroughout the commit history). In those cases we have to traverse parts\nof the object graph by walking commits and opening up trees. This can be\nexpensive, and is where your time is going.\n\nReachability bitmaps _usually_ make things faster, but they have some\ncases where they make things worse (especially if you have a ton of\nrefs, or haven't repacked recently).\n\nIf bitmaps are causing a problem for your push, they are likely to be\ncausing problems for fetches, too. But if you want to keep them to serve\nfetches, but not use them for push, you should be able to do:\n\n git -c pack.usebitmaps=false push\n\n-Peff\n"},{"id":"438521","messageId":"d8abab7fb5184a94a9e9e5b7c1f28695@xiaomi.com","threadId":"56676","inReplyTo":"YWVJyRJhRTdg39tX@coredump.intra.peff.net","subject":"RE: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2021-10-12T09:08:08Z","receivedAt":"2021-10-12T09:08:37Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"Oh my god.\nJesus. It works for me. After disable writebitmap, time cost decrease from 33 seconds to 0.9 seconds.\n\nBut now it turns out that, remote side takes 13 seconds to receive the pack, since git receive-pack is triggered automatically from remote side, is there anyway to enable GIT_TRACE2_PERF on server side?\n\n-----Original Message-----\nFrom: Jeff King \nSent: Tuesday, October 12, 2021 4:40 PM\nTo: 程洋 \nCc: git@vger.kernel.org\nSubject: Re: [External Mail]Re: why git is so slow for a tiny git push?\n\n*This message originated from outside of XIAOMI. Please treat this email with caution*\n\n\nOn Tue, Oct 12, 2021 at 08:04:44AM +0000, 程洋 wrote:\n\n> I have bitmap indeed because my master server also serves as download server.\n> However I'm using git 2.17.0, and I didn't set repack.writeBitmaps\n\nOn that version and without the config, then perhaps you (or somebody) passed \"-b\" to git-repack.\n\n> But why bitmaps can cause push to be slow? Do you mean that if\n> writeBitmaps is true, every push will regenerate bitmap file? If\n> that's what you mean, what I see is the only bitmap file in my repo\n> didn't change across time (the modify time is one month ago, long\n> before I run the experiment)\n\nNo, it is not regenerating the on-disk bitmaps. But when deciding the set of objects to send, pack-objects will generate an internal bitmap which is the set difference of objects reachable from the pushed refs, minus objects reachable from the refs the other the other side told us they had.\n\nIt uses the on-disk bitmaps as much as possible, but there may be commits not covered by bitmaps (either because they were pushed since the last repack which built bitmaps, or simply because it's too expensive to put a bitmap on every commit, so we sprinkle them throughout the commit history). In those cases we have to traverse parts of the object graph by walking commits and opening up trees. This can be expensive, and is where your time is going.\n\nReachability bitmaps _usually_ make things faster, but they have some cases where they make things worse (especially if you have a ton of refs, or haven't repacked recently).\n\nIf bitmaps are causing a problem for your push, they are likely to be causing problems for fetches, too. But if you want to keep them to serve fetches, but not use them for push, you should be able to do:\n\n git -c pack.usebitmaps=false push\n\n-Peff\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"438531","messageId":"87pmsak0hl.fsf@evledraar.gmail.com","threadId":"56676","inReplyTo":"YWVJyRJhRTdg39tX@coredump.intra.peff.net","subject":"Re: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-12T10:06:04Z","receivedAt":"2021-10-12T10:20:59Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Oct 12 2021, Jeff King wrote:\n\n> On Tue, Oct 12, 2021 at 08:04:44AM +0000, 程洋 wrote:\n>\n>> I have bitmap indeed because my master server also serves as download server.\n>> However I'm using git 2.17.0, and I didn't set repack.writeBitmaps\n>\n> On that version and without the config, then perhaps you (or somebody)\n> passed \"-b\" to git-repack.\n>\n>> But why bitmaps can cause push to be slow? Do you mean that if\n>> writeBitmaps is true, every push will regenerate bitmap file? If\n>> that's what you mean, what I see is the only bitmap file in my repo\n>> didn't change across time (the modify time is one month ago, long\n>> before I run the experiment)\n>\n> No, it is not regenerating the on-disk bitmaps. But when deciding the\n> set of objects to send, pack-objects will generate an internal bitmap\n> which is the set difference of objects reachable from the pushed refs,\n> minus objects reachable from the refs the other the other side told us\n> they had.\n>\n> It uses the on-disk bitmaps as much as possible, but there may be\n> commits not covered by bitmaps (either because they were pushed since\n> the last repack which built bitmaps, or simply because it's too\n> expensive to put a bitmap on every commit, so we sprinkle them\n> throughout the commit history). In those cases we have to traverse parts\n> of the object graph by walking commits and opening up trees. This can be\n> expensive, and is where your time is going.\n>\n> Reachability bitmaps _usually_ make things faster, but they have some\n> cases where they make things worse (especially if you have a ton of\n> refs, or haven't repacked recently).\n>\n> If bitmaps are causing a problem for your push, they are likely to be\n> causing problems for fetches, too. But if you want to keep them to serve\n> fetches, but not use them for push, you should be able to do:\n>\n> git -c pack.usebitmaps=false push\n\nFor the last on-list discussion to (probably the same) problem, which in\nturn references an even earlier one:\nhttps://lore.kernel.org/git/878s6nfq54.fsf@evledraar.gmail.com/\n\nI don't remember if my own report from mid-2019 said so or contradicts\nthis (and I didn't re-read the thread), but FWIW I *vaguely* recall that\nthe case I ran into *might* have had to do with a user running into this\non a shared \"staging\" server.\n\nI.e. one where users logged in as their own user, cd'd to a shared git\nrepo they got a lock on, and ran fetch/push/deploy commands. One user\nhad a pack.useBitmaps=true or equivalent in their config (or had\nmanually run such a repack), so there were some very old stale bitmaps\naround.\n\nThis was also a setup with a gc.bigPackThreshold configured, which now\nthat I think about it might have made it much worse, i.e. we'd keep that\n*.bitmap on the \"big pack\" around forever.\n\nBut more generally with these side-indexes it seems to me that the code\ninvolved might not be considering these sorts of edge cases, i.e. my\nunderstanding from you above is that if we have bitmaps anywhere we'll\ntry to in-memory use them for all the objects in play? Or that otherwise\nhaving \"partial\" bitmaps leads to pathological behavior.\n\ntl;dr: We can't assume that a config of \"I like to write side-index\nA/B/C/... when I repack\" means that the repo is in that state *now*, but\nit seems that we do.\n"},{"id":"438601","messageId":"YWYAjRfLsDtc1Dkm@coredump.intra.peff.net","threadId":"56676","inReplyTo":"d8abab7fb5184a94a9e9e5b7c1f28695@xiaomi.com","subject":"Re: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-10-12T21:39:25Z","receivedAt":"2021-10-12T21:39:29Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 12, 2021 at 09:08:08AM +0000, 程洋 wrote:\n\n> Jesus. It works for me. After disable writebitmap, time cost decrease\n> from 33 seconds to 0.9 seconds.\n> \n> But now it turns out that, remote side takes 13 seconds to receive the\n> pack, since git receive-pack is triggered automatically from remote\n> side, is there anyway to enable GIT_TRACE2_PERF on server side?\n\nFor the environment variable, it depends on your protocol. If you can\npush over ssh (and the other side lets you execute arbitrary commands),\nthen:\n\n git push --receive-pack='GIT_TRACE2_PERF=/tmp/foo.trace git-receive-pack'\n\nOtherwise, you can look at setting the trace2.perfTarget config option\non the server side. I haven't played with it myself before.\n\n-Peff\n"},{"id":"438603","messageId":"YWYCIndv/u67lNQU@coredump.intra.peff.net","threadId":"56676","inReplyTo":"87pmsak0hl.fsf@evledraar.gmail.com","subject":"Re: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-10-12T21:46:10Z","receivedAt":"2021-10-12T21:46:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 12, 2021 at 12:06:04PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> But more generally with these side-indexes it seems to me that the code\n> involved might not be considering these sorts of edge cases, i.e. my\n> understanding from you above is that if we have bitmaps anywhere we'll\n> try to in-memory use them for all the objects in play? Or that otherwise\n> having \"partial\" bitmaps leads to pathological behavior.\n\nSure, if there was an easy way to know beforehand whether the bitmap was\ngoing to help or run into these pathological cases, it would be nice to\ndetect it. I don't know what that is (and I've given it quite a lot of\nthought over the past 8 years).\n\nI suspect the most direction would be to teach the bitmap code to behave\nmore like the regular traversal by just walking down to the\nUNINTERESTING commits. Right now it gets a complete bitmap for the\ncommits we don't want, and then a bitmap for the ones we do want, and\ntakes a set difference.\n\nIt could instead walk both sides in the usual way, filling in the bitmap\nfor each, and then stop when it hits boundary commits. The bitmap for\nthe boundary commit (if we don't have a full one on-disk) is filled in\nwith what's in its tree. That means it's incomplete, and the result\nmight include some extra objects (e.g., if boundary~100 had a blob that\nwent away, but later came back in a descendant that isn't marked\nuninteresting). That's the same tradeoff the non-bitmap traversal makes.\n\nIt would be pretty major surgery to the bitmap code. I haven't actually\ntried it before.\n\n-Peff\n"},{"id":"438749","messageId":"5063751fd4b94ccca3987080c96370d8@xiaomi.com","threadId":"56676","inReplyTo":"YWYAjRfLsDtc1Dkm@coredump.intra.peff.net","subject":"RE: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2021-10-14T06:47:31Z","receivedAt":"2021-10-14T06:48:01Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"Seems that git receive-pack only takes 1 seconds.\nAfter that, git rev-list takes the most time, I don't know what is it doing\n\n-----Original Message-----\nFrom: Jeff King \nSent: Wednesday, October 13, 2021 5:39 AM\nTo: 程洋 \nCc: git@vger.kernel.org\nSubject: Re: [External Mail]Re: why git is so slow for a tiny git push?\n\n*This message originated from outside of XIAOMI. Please treat this email with caution*\n\n\nOn Tue, Oct 12, 2021 at 09:08:08AM +0000, 程洋 wrote:\n\n> Jesus. It works for me. After disable writebitmap, time cost decrease\n> from 33 seconds to 0.9 seconds.\n>\n> But now it turns out that, remote side takes 13 seconds to receive the\n> pack, since git receive-pack is triggered automatically from remote\n> side, is there anyway to enable GIT_TRACE2_PERF on server side?\n\nFor the environment variable, it depends on your protocol. If you can push over ssh (and the other side lets you execute arbitrary commands),\nthen:\n\n git push --receive-pack='GIT_TRACE2_PERF=/tmp/foo.trace git-receive-pack'\n\nOtherwise, you can look at setting the trace2.perfTarget config option on the server side. I haven't played with it myself before.\n\n-Peff\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"439693","messageId":"YXh5FoeyfePqr1k3@coredump.intra.peff.net","threadId":"56676","inReplyTo":"5063751fd4b94ccca3987080c96370d8@xiaomi.com","subject":"Re: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-10-26T21:54:30Z","receivedAt":"2021-10-26T21:54:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 14, 2021 at 06:47:31AM +0000, 程洋 wrote:\n\n> Seems that git receive-pack only takes 1 seconds.\n> After that, git rev-list takes the most time, I don't know what is it doing\n\nIt's checking connectivity of what was sent (i.e., that the other side\nsent us all the objects). If you have a ton of refs on the server side,\nthere are some known issues here, as just loading the \"we already have\nthis\" side of the traversal can be very expensive.\n\nThere's been some recent work to make this faster, like f559d6d45e\n(revision: avoid hitting packfiles when commits are in commit-graph,\n2021-08-09) and f45022dc2f (connected: do not sort input revisions,\n2021-08-09), which will be in the upcoming release. I'm sure there are\nmore improvements that could be made on top, but those show the general\ndirection.\n\n-Peff\n"},{"id":"439708","messageId":"5c7ac5d1ee2442bdb268daa5708806d3@xiaomi.com","threadId":"56676","inReplyTo":"YXh5FoeyfePqr1k3@coredump.intra.peff.net","subject":"RE: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2021-10-27T02:48:52Z","receivedAt":"2021-10-27T02:49:15Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"That's cool. Thanks for you help. I will try to compile it myself\n\n-----Original Message-----\nFrom: Jeff King \nSent: Wednesday, October 27, 2021 5:55 AM\nTo: 程洋 \nCc: git@vger.kernel.org\nSubject: Re: [External Mail]Re: why git is so slow for a tiny git push?\n\n*This message originated from outside of XIAOMI. Please treat this email with caution*\n\n\nOn Thu, Oct 14, 2021 at 06:47:31AM +0000, 程洋 wrote:\n\n> Seems that git receive-pack only takes 1 seconds.\n> After that, git rev-list takes the most time, I don't know what is it\n> doing\n\nIt's checking connectivity of what was sent (i.e., that the other side sent us all the objects). If you have a ton of refs on the server side, there are some known issues here, as just loading the \"we already have this\" side of the traversal can be very expensive.\n\nThere's been some recent work to make this faster, like f559d6d45e\n(revision: avoid hitting packfiles when commits are in commit-graph,\n2021-08-09) and f45022dc2f (connected: do not sort input revisions, 2021-08-09), which will be in the upcoming release. I'm sure there are more improvements that could be made on top, but those show the general direction.\n\n-Peff\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"439868","messageId":"CAFQ2z_P8y_zze6sBz4OQzEuEEC7ygGNk2UqHhu+8QkF=Cqv1xA@mail.gmail.com","threadId":"56676","inReplyTo":"576b2f3e162e4f86992d8f4e680d0881@xiaomi.com","subject":"Re: why git is so slow for a tiny git push?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-10-28T13:17:04Z","receivedAt":"2021-10-28T13:17:21Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Sat, Oct 9, 2021 at 8:06 PM 程洋 wrote:\n>\n> I have a really big repository which has 9m objects and maybe 300k refs.\n> I noticed that git push is really slow for a tiny change. An example shows below\n>\n> 3 objects which is only 7 kb takes 36 seconds to pack-objects (it's the time after i enable pack.usesparse)\n> However if I manually call “pack-objects” with the exactly same objects SHA1. It only take less than 0.005 second\n> What is really pass to “pack-objects” when I call “git push”?\n>\n> I read an article says git will enumerate all \"uninteresting objects\" to determine what to send. but i don't understand, in my case git should only enumerate objects between \"1a2d494b1b71469eebbd42aeabe1736bfa4b51fa..ddf3b84dca1aa4fe209a218380df1482af0d6b48\". It's insane. I have a master server and a slave server serve this repository to my users. And i have a cron job to push every change from master to slave. And i found my master server CPU is full all the time because of the push jobs\n>\n> Is there any solution?\n\nnot sure if this is your problem , but we heard reports of bitmaps\nslowing down push replication for Gerrit (it looks like you're working\nwith Android repositories.). See\nhttps://groups.google.com/g/repo-discuss/c/Xb8TbBXUYxw/m/jv5hqZ2PCQAJ\nfor background and suggestions.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"442094","messageId":"56994f02ed4542b89cc6f9586d703e05@xiaomi.com","threadId":"56676","inReplyTo":"YWYCIndv/u67lNQU@coredump.intra.peff.net","subject":"RE: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2021-11-23T06:42:12Z","receivedAt":"2021-11-23T06:44:33Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"I got another problem here.\nWhen I tries to clone from remote server. It took me 25 seconds to enumerating objects. And then 1 second to `couting objects` by bitmap.\nI don't understand, why a fresh clone need `enumerating objects` ? Is `couting objects` enough for the server to determine what to send?\n\nHere is the remote server trace:\n\n\n11:49:12.438519 common-main.c:48 | d0 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n11:49:12.438556 common-main.c:49 | d0 | main | start | | 0.000274 | | | git daemon --inetd --syslog --export-all --enable=upload-pack --enable=receive-pack --base-path=/home/work/repositories\n11:49:12.438607 compat/linux/procinfo.c:170 | d0 | main | cmd_ancestry | | | | | ancestry:[xinetd systemd]\n11:49:12.438655 git.c:737 | d0 | main | cmd_name | | | | | _run_dashed_ (_run_dashed_)\n11:49:12.438668 run-command.c:739 | d0 | main | child_start | | 0.000390 | | | [ch0] class:dashed argv:[git-daemon --inetd --syslog --export-all --enable=upload-pack --enable=receive-pack --base-path=/home/work/repositories]\n11:49:12.439555 common-main.c:48 | d1 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n11:49:12.439589 common-main.c:49 | d1 | main | start | | 0.000242 | | | /usr/libexec/git-core/git-daemon --inetd --syslog --export-all --enable=upload-pack --enable=receive-pack --base-path=/home/work/repositories\n11:49:12.439645 compat/linux/procinfo.c:170 | d1 | main | cmd_ancestry | | | | | ancestry:[git xinetd systemd]\n11:49:12.439809 run-command.c:739 | d1 | main | child_start | | 0.000467 | | | [ch0] class:? argv:[git upload-pack --strict --timeout=0 .]\n11:49:12.440747 common-main.c:48 | d2 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n11:49:12.440772 common-main.c:49 | d2 | main | start | | 0.000252 | | | /usr/libexec/git-core/git upload-pack --strict --timeout=0 .\n11:49:12.440833 compat/linux/procinfo.c:170 | d2 | main | cmd_ancestry | | | | | ancestry:[git-daemon git xinetd systemd]\n11:49:12.440853 git.c:456 | d2 | main | cmd_name | | | | | upload-pack (_run_dashed_/upload-pack)\n11:49:12.441013 protocol.c:76 | d2 | main | data | | 0.000494 | 0.000494 | transfer | negotiated-version:2\n11:49:12.481208 run-command.c:739 | d2 | main | child_start | | 0.040684 | | | [ch0] class:? argv:[git pack-objects --revs --thin --stdout --progress --delta-base-offset]\n11:49:12.482307 common-main.c:48 | d3 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n11:49:12.482334 common-main.c:49 | d3 | main | start | | 0.000220 | | | /usr/libexec/git-core/git pack-objects --revs --thin --stdout --progress --delta-base-offset\n11:49:12.482405 compat/linux/procinfo.c:170 | d3 | main | cmd_ancestry | | | | | ancestry:[git git-daemon git xinetd systemd]\n11:49:12.482500 git.c:456 | d3 | main | cmd_name | | | | | pack-objects (_run_dashed_/upload-pack/pack-objects)\n11:49:12.482632 builtin/pack-objects.c:4140 | d3 | main | region_enter | r0 | 0.000522 | | pack-objects | label:enumerate-objects\n11:49:12.482825 progress.c:268 | d3 | main | region_enter | r0 | 0.000715 | | progress | ..label:Enumerating objects\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n11:49:21.477783 progress.c:329 | d3 | main | data | r0 | 8.995670 | 8.994955 | progress | ....total_objects:0\n11:49:21.477848 progress.c:336 | d3 | main | region_leave | r0 | 8.995738 | 8.995023 | progress | ..label:Enumerating objects\n11:49:21.477880 builtin/pack-objects.c:4162 | d3 | main | region_leave | r0 | 8.995770 | 8.995248 | pack-objects | label:enumerate-objects\n11:49:21.477891 builtin/pack-objects.c:4168 | d3 | main | region_enter | r0 | 8.995782 | | pack-objects | label:prepare-pack\n11:49:21.477903 progress.c:268 | d3 | main | region_enter | r0 | 8.995794 | | progress | ..label:Counting objects\n11:49:22.316806 progress.c:329 | d3 | main | data | r0 | 9.834695 | 0.838901 | progress | ....total_objects:1383396\n11:49:22.316848 progress.c:336 | d3 | main | region_leave | r0 | 9.834738 | 0.838944 | progress | ..label:Counting objects\n11:49:22.366109 progress.c:268 | d3 | main | region_enter | r0 | 9.883998 | | progress | ..label:Compressing objects\n11:49:34.208323 trace2/tr2_tgt_perf.c:201 | d2 | main | signal | | 21.767795 | | | signo:13\n11:49:34.208372 trace2/tr2_tgt_perf.c:201 | d3 | main | signal | | 21.726219 | | | ....signo:13\n11:49:34.218767 run-command.c:995 | d1 | main | child_exit | | 21.779417 | 21.778950 | | [ch0] pid:48725 code:141\n11:49:34.218809 common-main.c:54 | d1 | main | exit | | 21.779469 | | | code:141\n11:49:34.218822 trace2/tr2_tgt_perf.c:213 | d1 | main | atexit | | 21.779482 | | | code:141\n11:49:34.219135 run-command.c:995 | d0 | main | child_exit | | 21.780855 | 21.780465 | | [ch0] pid:48724 code:141\n11:49:34.219170 git.c:759 | d0 | main | exit | | 21.780893 | | | code:141\n11:49:34.219182 trace2/tr2_tgt_perf.c:213 | d0 | main | atexit | | 21.780906 | | | code:141\n-----Original Message-----\nFrom: Jeff King \nSent: Wednesday, October 13, 2021 5:46 AM\nTo: Ævar Arnfjörð Bjarmason \nCc: 程洋 ; git@vger.kernel.org\nSubject: Re: [External Mail]Re: why git is so slow for a tiny git push?\n\n*This message originated from outside of XIAOMI. Please treat this email with caution*\n\n\nOn Tue, Oct 12, 2021 at 12:06:04PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> But more generally with these side-indexes it seems to me that the\n> code involved might not be considering these sorts of edge cases, i.e.\n> my understanding from you above is that if we have bitmaps anywhere\n> we'll try to in-memory use them for all the objects in play? Or that\n> otherwise having \"partial\" bitmaps leads to pathological behavior.\n\nSure, if there was an easy way to know beforehand whether the bitmap was going to help or run into these pathological cases, it would be nice to detect it. I don't know what that is (and I've given it quite a lot of thought over the past 8 years).\n\nI suspect the most direction would be to teach the bitmap code to behave more like the regular traversal by just walking down to the UNINTERESTING commits. Right now it gets a complete bitmap for the commits we don't want, and then a bitmap for the ones we do want, and takes a set difference.\n\nIt could instead walk both sides in the usual way, filling in the bitmap for each, and then stop when it hits boundary commits. The bitmap for the boundary commit (if we don't have a full one on-disk) is filled in with what's in its tree. That means it's incomplete, and the result might include some extra objects (e.g., if boundary~100 had a blob that went away, but later came back in a descendant that isn't marked uninteresting). That's the same tradeoff the non-bitmap traversal makes.\n\nIt would be pretty major surgery to the bitmap code. I haven't actually tried it before.\n\n-Peff\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"442234","messageId":"feaf3bace8394837ad0ec020d01b545c@xiaomi.com","threadId":"56676","inReplyTo":"YWYCIndv/u67lNQU@coredump.intra.peff.net","subject":"RE: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2021-11-24T08:07:15Z","receivedAt":"2021-11-24T08:07:41Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"It seems that, \"get_object_list_from_bitmap\" takes 9 seconds.\nDoes it meet the expectation?\n\nI'm not sure. But here is my guess:\nSince I have 300k refs. But clone with `--no-tags` only requires \"refs/heads/*\". Git has to search and filter refs in the whole bitmap file, which takes a lot of time.\nI think jgit do it in a really smart way. It pack all refs/heads into one bitmapfile ,and the other refs in another bitmap file. Because 90% of clone operation only requires all refs/heads.\n\n\n-----Original Message-----\nFrom: 程洋\nSent: Tuesday, November 23, 2021 2:42 PM\nTo: 'Jeff King' ; Ævar Arnfjörð Bjarmason \nCc: git@vger.kernel.org\nSubject: RE: [External Mail]Re: why git is so slow for a tiny git push?\n\nI got another problem here.\nWhen I tries to clone from remote server. It took me 25 seconds to enumerating objects. And then 1 second to `couting objects` by bitmap.\nI don't understand, why a fresh clone need `enumerating objects` ? Is `couting objects` enough for the server to determine what to send?\n\nHere is the remote server trace:\n\n\n11:49:12.438519 common-main.c:48 | d0 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n11:49:12.438556 common-main.c:49 | d0 | main | start | | 0.000274 | | | git daemon --inetd --syslog --export-all --enable=upload-pack --enable=receive-pack --base-path=/home/work/repositories\n11:49:12.438607 compat/linux/procinfo.c:170 | d0 | main | cmd_ancestry | | | | | ancestry:[xinetd systemd]\n11:49:12.438655 git.c:737 | d0 | main | cmd_name | | | | | _run_dashed_ (_run_dashed_)\n11:49:12.438668 run-command.c:739 | d0 | main | child_start | | 0.000390 | | | [ch0] class:dashed argv:[git-daemon --inetd --syslog --export-all --enable=upload-pack --enable=receive-pack --base-path=/home/work/repositories]\n11:49:12.439555 common-main.c:48 | d1 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n11:49:12.439589 common-main.c:49 | d1 | main | start | | 0.000242 | | | /usr/libexec/git-core/git-daemon --inetd --syslog --export-all --enable=upload-pack --enable=receive-pack --base-path=/home/work/repositories\n11:49:12.439645 compat/linux/procinfo.c:170 | d1 | main | cmd_ancestry | | | | | ancestry:[git xinetd systemd]\n11:49:12.439809 run-command.c:739 | d1 | main | child_start | | 0.000467 | | | [ch0] class:? argv:[git upload-pack --strict --timeout=0 .]\n11:49:12.440747 common-main.c:48 | d2 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n11:49:12.440772 common-main.c:49 | d2 | main | start | | 0.000252 | | | /usr/libexec/git-core/git upload-pack --strict --timeout=0 .\n11:49:12.440833 compat/linux/procinfo.c:170 | d2 | main | cmd_ancestry | | | | | ancestry:[git-daemon git xinetd systemd]\n11:49:12.440853 git.c:456 | d2 | main | cmd_name | | | | | upload-pack (_run_dashed_/upload-pack)\n11:49:12.441013 protocol.c:76 | d2 | main | data | | 0.000494 | 0.000494 | transfer | negotiated-version:2\n11:49:12.481208 run-command.c:739 | d2 | main | child_start | | 0.040684 | | | [ch0] class:? argv:[git pack-objects --revs --thin --stdout --progress --delta-base-offset]\n11:49:12.482307 common-main.c:48 | d3 | main | version | | | | | 2.33.1.558.g2bd2f258f4.dirty\n11:49:12.482334 common-main.c:49 | d3 | main | start | | 0.000220 | | | /usr/libexec/git-core/git pack-objects --revs --thin --stdout --progress --delta-base-offset\n11:49:12.482405 compat/linux/procinfo.c:170 | d3 | main | cmd_ancestry | | | | | ancestry:[git git-daemon git xinetd systemd]\n11:49:12.482500 git.c:456 | d3 | main | cmd_name | | | | | pack-objects (_run_dashed_/upload-pack/pack-objects)\n11:49:12.482632 builtin/pack-objects.c:4140 | d3 | main | region_enter | r0 | 0.000522 | | pack-objects | label:enumerate-objects\n11:49:12.482825 progress.c:268 | d3 | main | region_enter | r0 | 0.000715 | | progress | ..label:Enumerating objects\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n11:49:21.477783 progress.c:329 | d3 | main | data | r0 | 8.995670 | 8.994955 | progress | ....total_objects:0\n11:49:21.477848 progress.c:336 | d3 | main | region_leave | r0 | 8.995738 | 8.995023 | progress | ..label:Enumerating objects\n11:49:21.477880 builtin/pack-objects.c:4162 | d3 | main | region_leave | r0 | 8.995770 | 8.995248 | pack-objects | label:enumerate-objects\n11:49:21.477891 builtin/pack-objects.c:4168 | d3 | main | region_enter | r0 | 8.995782 | | pack-objects | label:prepare-pack\n11:49:21.477903 progress.c:268 | d3 | main | region_enter | r0 | 8.995794 | | progress | ..label:Counting objects\n11:49:22.316806 progress.c:329 | d3 | main | data | r0 | 9.834695 | 0.838901 | progress | ....total_objects:1383396\n11:49:22.316848 progress.c:336 | d3 | main | region_leave | r0 | 9.834738 | 0.838944 | progress | ..label:Counting objects\n11:49:22.366109 progress.c:268 | d3 | main | region_enter | r0 | 9.883998 | | progress | ..label:Compressing objects\n11:49:34.208323 trace2/tr2_tgt_perf.c:201 | d2 | main | signal | | 21.767795 | | | signo:13\n11:49:34.208372 trace2/tr2_tgt_perf.c:201 | d3 | main | signal | | 21.726219 | | | ....signo:13\n11:49:34.218767 run-command.c:995 | d1 | main | child_exit | | 21.779417 | 21.778950 | | [ch0] pid:48725 code:141\n11:49:34.218809 common-main.c:54 | d1 | main | exit | | 21.779469 | | | code:141\n11:49:34.218822 trace2/tr2_tgt_perf.c:213 | d1 | main | atexit | | 21.779482 | | | code:141\n11:49:34.219135 run-command.c:995 | d0 | main | child_exit | | 21.780855 | 21.780465 | | [ch0] pid:48724 code:141\n11:49:34.219170 git.c:759 | d0 | main | exit | | 21.780893 | | | code:141\n11:49:34.219182 trace2/tr2_tgt_perf.c:213 | d0 | main | atexit | | 21.780906 | | | code:141\n-----Original Message-----\nFrom: Jeff King \nSent: Wednesday, October 13, 2021 5:46 AM\nTo: Ævar Arnfjörð Bjarmason \nCc: 程洋 ; git@vger.kernel.org\nSubject: Re: [External Mail]Re: why git is so slow for a tiny git push?\n\n*This message originated from outside of XIAOMI. Please treat this email with caution*\n\n\nOn Tue, Oct 12, 2021 at 12:06:04PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> But more generally with these side-indexes it seems to me that the\n> code involved might not be considering these sorts of edge cases, i.e.\n> my understanding from you above is that if we have bitmaps anywhere\n> we'll try to in-memory use them for all the objects in play? Or that\n> otherwise having \"partial\" bitmaps leads to pathological behavior.\n\nSure, if there was an easy way to know beforehand whether the bitmap was going to help or run into these pathological cases, it would be nice to detect it. I don't know what that is (and I've given it quite a lot of thought over the past 8 years).\n\nI suspect the most direction would be to teach the bitmap code to behave more like the regular traversal by just walking down to the UNINTERESTING commits. Right now it gets a complete bitmap for the commits we don't want, and then a bitmap for the ones we do want, and takes a set difference.\n\nIt could instead walk both sides in the usual way, filling in the bitmap for each, and then stop when it hits boundary commits. The bitmap for the boundary commit (if we don't have a full one on-disk) is filled in with what's in its tree. That means it's incomplete, and the result might include some extra objects (e.g., if boundary~100 had a blob that went away, but later came back in a descendant that isn't marked uninteresting). That's the same tradeoff the non-bitmap traversal makes.\n\nIt would be pretty major surgery to the bitmap code. I haven't actually tried it before.\n\n-Peff\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"},{"id":"442253","messageId":"YZ6BJpAY4ZAWaqkr@coredump.intra.peff.net","threadId":"56676","inReplyTo":"56994f02ed4542b89cc6f9586d703e05@xiaomi.com","subject":"Re: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-11-24T18:15:02Z","receivedAt":"2021-11-24T18:15:09Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 23, 2021 at 06:42:12AM +0000, 程洋 wrote:\n\n> I got another problem here.\n> When I tries to clone from remote server. It took me 25 seconds to enumerating objects. And then 1 second to `couting objects` by bitmap.\n> I don't understand, why a fresh clone need `enumerating objects` ? Is `couting objects` enough for the server to determine what to send?\n\nIn older versions of Git, the \"counting objects\" progress meter used to\nbe the actual object graph traversal. That changed in v2.18 (via\n5af050437a), but you may still seem some reference to \"counting objects\nis expensive\".\n\nThese days that is called \"enumerating objects\", and \"counting objects\"\nis just doing a quick-ish pass over that list to do some light analysis\n(e.g., if we can reuse an on-disk delta). I'd expect \"enumerating\" to be\nexpensive in general, and \"counting\" to be quick in general.\n\nThe \"enumerating\" phase is where we determine what to send whether it's\nfor a clone or a fetch, and may involve opening up a bunch of trees to\nwalk the graph. It's what reachability bitmaps are supposed to make\nfaster. But if you have 300k refs, as you've mentioned, you almost\ncertainly don't have complete coverage of all of the ref tips, so we'll\nhave to fallback to doing at least a partial graph traversal.\n\nTaylor (cc'd) has been looking at some tricks for speeding up cases like\nthis with a lot of refs. But I don't think there's anything to show\npublicly yet.\n\n-Peff\n"},{"id":"442279","messageId":"a39b72ed59f045c19fad8fcb259821d2@xiaomi.com","threadId":"56676","inReplyTo":"YZ6BJpAY4ZAWaqkr@coredump.intra.peff.net","subject":"RE: [External Mail]Re: why git is so slow for a tiny git push?","fromName":"程洋","fromEmail":"chengyang@xiaomi.com","sentAt":"2021-11-25T02:53:33Z","receivedAt":"2021-11-25T02:55:37Z","isPatch":false,"sender":{"key":"chengyang@xiaomi.com","avatar":null},"body":"Well, we do have 300k refs, but only 1000 refs/heads.\nHowever, I think most users only requires refs/heads, a few people only require refs/tags. As for other refs, we hardly see any user case.\n\nSo jgit treat it with a smart way, it will create 2 pack files and 2 bitmaps, pack A contain all refs/heads, and pack B contains other refs. And when user do a fresh clone, it just need to send the pack A without determine if we can reuse or not\n\n-----Original Message-----\nFrom: Jeff King \nSent: Thursday, November 25, 2021 2:15 AM\nTo: 程洋 \nCc: Taylor Blau ; Ævar Arnfjörð Bjarmason ; git@vger.kernel.org\nSubject: Re: [External Mail]Re: why git is so slow for a tiny git push?\n\n*This message originated from outside of XIAOMI. Please treat this email with caution*\n\n\nOn Tue, Nov 23, 2021 at 06:42:12AM +0000, 程洋 wrote:\n\n> I got another problem here.\n> When I tries to clone from remote server. It took me 25 seconds to enumerating objects. And then 1 second to `couting objects` by bitmap.\n> I don't understand, why a fresh clone need `enumerating objects` ? Is `couting objects` enough for the server to determine what to send?\n\nIn older versions of Git, the \"counting objects\" progress meter used to be the actual object graph traversal. That changed in v2.18 (via 5af050437a), but you may still seem some reference to \"counting objects is expensive\".\n\nThese days that is called \"enumerating objects\", and \"counting objects\"\nis just doing a quick-ish pass over that list to do some light analysis (e.g., if we can reuse an on-disk delta). I'd expect \"enumerating\" to be expensive in general, and \"counting\" to be quick in general.\n\nThe \"enumerating\" phase is where we determine what to send whether it's for a clone or a fetch, and may involve opening up a bunch of trees to walk the graph. It's what reachability bitmaps are supposed to make faster. But if you have 300k refs, as you've mentioned, you almost certainly don't have complete coverage of all of the ref tips, so we'll have to fallback to doing at least a partial graph traversal.\n\nTaylor (cc'd) has been looking at some tricks for speeding up cases like this with a lot of refs. But I don't think there's anything to show publicly yet.\n\n-Peff\n#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#\n"}]}