{"thread":{"id":"44279","subject":"Make `git fetch --all` parallel?","startedAt":"2016-10-11T20:13:07Z","lastAt":"2016-10-12T06:48:18Z","messageCount":12,"participants":["Ram Rachum","Stefan Beller","Junio C Hamano","Jeff King","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"303921","messageId":"CANXboVZvfPkTQ10PWop+LgPFpc2bD3-u-e5ix0itGawiwCxOuQ@mail.gmail.com","threadId":"44279","inReplyTo":null,"subject":"Make `git fetch --all` parallel?","fromName":"Ram Rachum","fromEmail":"ram@rachum.com","sentAt":"2016-10-11T20:12:41Z","receivedAt":"2016-10-11T20:13:07Z","isPatch":false,"sender":{"key":"ram@rachum.com","avatar":"https://gravatar.com/avatar/e8ab34af7f6351b96e027ece1b3f9b3bd5d6362b38372071db2e6e4e6de5cce4?d=mp&s=160"},"body":"Hi everyone!\n\nI have a repo that has a bunch of different remotes, and I noticed\nslowness when doing `git fetch --all`. Is it currently made\nsequentially? Do you think that maybe it could be done in parallel so\nit could be much faster?\n\nThanks,\nRam.\n"},{"id":"303925","messageId":"CAGZ79kZmrYZqi4+bSkRykn+Upt7bEyZ0N8VhiQ-h8DhSMym-FA@mail.gmail.com","threadId":"44279","inReplyTo":"CANXboVZvfPkTQ10PWop+LgPFpc2bD3-u-e5ix0itGawiwCxOuQ@mail.gmail.com","subject":"Re: Make `git fetch --all` parallel?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-11T20:53:03Z","receivedAt":"2016-10-11T20:53:15Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Oct 11, 2016 at 1:12 PM, Ram Rachum wrote:\n> Hi everyone!\n>\n> I have a repo that has a bunch of different remotes, and I noticed\n> slowness when doing `git fetch --all`. Is it currently made\n> sequentially? Do you think that maybe it could be done in parallel so\n> it could be much faster?\n>\n> Thanks,\n> Ram.\n\nIf you were to run fetching from each remote in parallel\nassuming the work load is unchanged, this would speed up the\nexecution by the number of remotes.\n\nThis translation sounds pretty easy at first, but when looking into\nthe details it is not as easy any more:\n\nWhat if 2 remotes have the same object (e.g. the same commit)?\nCurrently this is easy: The first remote to fetch from will deliver that\nobject to you.\n\nWhen fetching in parallel, we would want to download that object from\njust one remote, preferably the remote with better network connectivity(?)\n\nSo I do think it would be much faster, but I also think patches for this would\nrequire some thought and a lot of refactoring of the fetch code.\n\nThe current fetch protocol is roughly:\n\nremote: I have these refs:\n8a36cd87b7c85a651ab388d403629865ffa3ba0d HEAD\n10d26b0d1ef1ebfd09418ec61bdadc299ac988e2 refs/heads/ab/gitweb-abbrev-links\n77947bbe24e0306d1ce5605c962c4a25f5aca22f refs/heads/ab/gitweb-link-html-escape\n...\n\nclient: I want 8a36cd87b7c85a651ab388d403629865ffa3ba0d,\nand I have 231ce93d2a0b0b4210c810e865eb5db7ba3032b2\nand I have 02d0927973782f4b8b7317b499979fada1105be6\nand I have 1172e16af07d6e15bca6398f0ded18a0ae7b9249\n\nremote: I don't know about 231ce93d2a0b0b4210c810e865eb5db7ba3032b2,\nnor 02d0927973782f4b8b7317b499979fada1105be6, but\nI know about 1172e16af07d6e15bca6398f0ded18a0ae7b9249\n\n.... conversation continues...\n\nremote: Ok I figured out what you need, here is a packfile:\n\n\n\nDuring the negotiation phase a client would have to be able to change its\nmind (add more \"haves\", or in case of the parallel fetching these become\n\"will-have-soons\", although the remote figured out the client did not have it\nearlier.)\n\nIf you want to see more details, see Documentation/technical/pack-protocol.txt\n"},{"id":"303942","messageId":"xmqqa8ea7bsh.fsf@gitster.mtv.corp.google.com","threadId":"44279","inReplyTo":"CAGZ79kZmrYZqi4+bSkRykn+Upt7bEyZ0N8VhiQ-h8DhSMym-FA@mail.gmail.com","subject":"Re: Make `git fetch --all` parallel?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-11T22:37:02Z","receivedAt":"2016-10-11T22:39:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> So I do think it would be much faster, but I also think patches for this would\n> require some thought and a lot of refactoring of the fetch code.\n> ...\n> During the negotiation phase a client would have to be able to change its\n> mind (add more \"haves\", or in case of the parallel fetching these become\n> \"will-have-soons\", although the remote figured out the client did not have it\n> earlier.)\n\nEven though a fancy optimization as you outlined might be ideal, I\nsuspect that users would be happier if the network bandwidth is\nutilized to talk to multiple remotes at the same time even if they\nend up receiving the same recent objects from more than one place in\nthe end.\n\nIs the order in which \"git fetch --all\" iterates over \"all remotes\"\npredictable and documented? If so, listing the remotes from more\npowerful and well connected place to slower ones and then doing an\nequivalent of stupid\n\n\tfor remote in $list_of_remotes_ordered_in_such_a_way\n\tdo\n\t\tgit fetch \"$remote\" &\n\t\tsleep 2\n\tdone\n\nmight be fairly easy thing to bring happiness.\n"},{"id":"303943","messageId":"CAGZ79kZNvTvk4uZa8xhxZABKtzS9A5HoumJ37AacuZnHaZ4+Xw@mail.gmail.com","threadId":"44279","inReplyTo":"xmqqa8ea7bsh.fsf@gitster.mtv.corp.google.com","subject":"Re: Make `git fetch --all` parallel?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-11T22:50:36Z","receivedAt":"2016-10-11T22:50:43Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Oct 11, 2016 at 3:37 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> So I do think it would be much faster, but I also think patches for this would\n>> require some thought and a lot of refactoring of the fetch code.\n>> ...\n>> During the negotiation phase a client would have to be able to change its\n>> mind (add more \"haves\", or in case of the parallel fetching these become\n>> \"will-have-soons\", although the remote figured out the client did not have it\n>> earlier.)\n>\n> Even though a fancy optimization as you outlined might be ideal, I\n> suspect that users would be happier if the network bandwidth is\n> utilized to talk to multiple remotes at the same time even if they\n> end up receiving the same recent objects from more than one place in\n> the end.\n\nI agree. Though even for implementing the \"dumb\" case of fetching\nobjects twice we'd have to take care of some racing issues, I would assume.\n\nWhy did you put a \"sleep 2\" below?\n* a slow start to better spread load locally? (keep the workstation responsive?)\n* a slow start to have different fetches in a different phase of the\nfetch protocol?\n* avoiding some subtle race?\n\nAt the very least we would need a similar thing as Jeff recently sent for the\npush case with objects quarantined and then made available in one go?\n\n>\n> Is the order in which \"git fetch --all\" iterates over \"all remotes\"\n> predictable and documented?\n\nit is predictable, as it is just the same order as put by grep in\n$ grep \"\\[remote \" .git/config, i.e. in order of the file, which in my\ncase turns out to be sorted by importance/history quite naturally.\nBut reordering my config file would be not a big deal.\n\nI dunno, if documented though.\n\n> If so, listing the remotes from more\n> powerful and well connected place to slower ones and then doing an\n> equivalent of stupid\n>\n> for remote in $list_of_remotes_ordered_in_such_a_way\n\nlist_of_remotes_ordered_in_such_a_way is roughly:\n$(git config --get-regexp remote.*.url | tr '.' ' ' |awk '{print $2}')\n\n> do\n> git fetch \"$remote\" &\n> sleep 2\n> done\n>\n> might be fairly easy thing to bring happiness.\n\nI would love to see the implementation though, as over time I accumulate\na lot or remotes. (Someone published patches on the mailing list and made\nthem available somewhere hosted? Grabbing them from their hosting site\nis easier than applying patches for me, so I'd rather fetch them... so I have\nsome remotes now)\n"},{"id":"303944","messageId":"CAGZ79kb_-k2B=+3dFw=AUS=mWafKObAiGDMShfekmW=37hSYJA@mail.gmail.com","threadId":"44279","inReplyTo":"CAGZ79kZNvTvk4uZa8xhxZABKtzS9A5HoumJ37AacuZnHaZ4+Xw@mail.gmail.com","subject":"Re: Make `git fetch --all` parallel?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-11T22:58:16Z","receivedAt":"2016-10-11T22:58:23Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> I dunno, if documented though.\n\nhttp://stackoverflow.com/questions/26373995/how-to-control-the-order-of-fetching-when-fetching-all-remotes-by-git-fetch-al\n\nWe do not give promises about the order of --all (checked with our\ndocumentation as well), however there seems to be a\ngrouping scheme for remotes that you can order via\nsetting remotes.default (which is not documented in man git config,\nbut only in the git remote man page)\n"},{"id":"303945","messageId":"xmqq60oy7at4.fsf@gitster.mtv.corp.google.com","threadId":"44279","inReplyTo":"CAGZ79kZNvTvk4uZa8xhxZABKtzS9A5HoumJ37AacuZnHaZ4+Xw@mail.gmail.com","subject":"Re: Make `git fetch --all` parallel?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-11T22:58:15Z","receivedAt":"2016-10-11T22:58:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> Why did you put a \"sleep 2\" below?\n\nThe assumption was to fetch from faster and near the center of the\nproject universe early, so by giving them head-start, fetches that\nstart in later rounds may have chance to see newly updated remote\ntracking refs when telling the poorer other ends what we have.\n\n> At the very least we would need a similar thing as Jeff recently sent for the\n> push case with objects quarantined and then made available in one go?\n\nThere is no race; ref updates are done only after objects are fully\nfinalized. You can do the quarantine but that would defeat the \"let\nones from the center of universe finish early so later ones from\npoor periphery have more .have's to work with\" idea, I suspect.\n\n"},{"id":"303946","messageId":"20161011225942.tvqbbzxglvu7lldi@sigill.intra.peff.net","threadId":"44279","inReplyTo":"CAGZ79kZNvTvk4uZa8xhxZABKtzS9A5HoumJ37AacuZnHaZ4+Xw@mail.gmail.com","subject":"Re: Make `git fetch --all` parallel?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-10-11T22:59:42Z","receivedAt":"2016-10-11T22:59:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 11, 2016 at 03:50:36PM -0700, Stefan Beller wrote:\n\n> I agree. Though even for implementing the \"dumb\" case of fetching\n> objects twice we'd have to take care of some racing issues, I would assume.\n> \n> Why did you put a \"sleep 2\" below?\n> * a slow start to better spread load locally? (keep the workstation responsive?)\n> * a slow start to have different fetches in a different phase of the\n> fetch protocol?\n> * avoiding some subtle race?\n> \n> At the very least we would need a similar thing as Jeff recently sent for the\n> push case with objects quarantined and then made available in one go?\n\nI don't think so. The object database is perfectly happy with multiple\nsimultaneous writers, and nothing impacts the have/wants until actual\nrefs are written. Quarantining objects before the refs are written is an\northogonal concept.\n\nI'm not altogether convinced that parallel fetch would be that much\nfaster, though. Your bottleneck for a fetch is generally the network for\nmost of the time, then a brief spike of CPU during delta resolution. You\nmight get some small benefit from overlapping the fetches so that you\nspend CPU on one while you spend network on the other, but I doubt it\nwould be nearly as beneficial as the parallel submodule clones (which\ngenerally have a bigger CPU segment, and also are generally considered\nindependent, so there's no real tradeoff of getting duplicate objects).\n\nSometimes the bottleneck is the server preparing the back, but if that\nis the case, you should probably complain to your server admin to enable\nbitmaps. :)\n\n> I would love to see the implementation though, as over time I accumulate\n> a lot or remotes. (Someone published patches on the mailing list and made\n> them available somewhere hosted? Grabbing them from their hosting site\n> is easier than applying patches for me, so I'd rather fetch them... so I have\n> some remotes now)\n\nI usually just do a one-off fetch of their URL in such a case, exactly\nbecause I _don't_ want to end up with a bunch of remotes. You can also\nmark them with skipDefaultUpdate if you only care about them\noccasionally (so you can \"git fetch sbeller\" when you care about it, but\nit doesn't slow down your daily \"git fetch\").\n\n-Peff\n"},{"id":"303947","messageId":"CACBZZX6pFbu_CWMG+OTxx8L2OEy4X4RqGfQ_+z4zFOhS-zx8zA@mail.gmail.com","threadId":"44279","inReplyTo":"20161011225942.tvqbbzxglvu7lldi@sigill.intra.peff.net","subject":"Re: Make `git fetch --all` parallel?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2016-10-11T23:16:20Z","receivedAt":"2016-10-11T23:16:46Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Wed, Oct 12, 2016 at 12:59 AM, Jeff King wrote:\n\n> I'm not altogether convinced that parallel fetch would be that much\n> faster, though.\n\nI have local aliases to use GNU parallel for stuff like this, on my\ngit.git which has accumulated 17 remotes:\n\n $ time parallel -j1 'git fetch {}' ::: $(git remote)\n real 0m18.265s\n $ time parallel -j8 'git fetch {}' ::: $(git remote)\n real 0m2.957s\n\nIn that case I didn't have any new objects to fetch, but just doing\nthe negotiation in parallel was a lot faster.\n\nSo there's big wins in some cases.\n"},{"id":"303948","messageId":"CAGZ79kaKOiy-HJboaujXXc66P6CLupteDw4JyPOGetREfz_q_Q@mail.gmail.com","threadId":"44279","inReplyTo":"20161011225942.tvqbbzxglvu7lldi@sigill.intra.peff.net","subject":"Re: Make `git fetch --all` parallel?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-11T23:18:15Z","receivedAt":"2016-10-11T23:25:42Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Oct 11, 2016 at 3:59 PM, Jeff King wrote:\n> On Tue, Oct 11, 2016 at 03:50:36PM -0700, Stefan Beller wrote:\n>\n>> I agree. Though even for implementing the \"dumb\" case of fetching\n>> objects twice we'd have to take care of some racing issues, I would assume.\n>>\n>> Why did you put a \"sleep 2\" below?\n>> * a slow start to better spread load locally? (keep the workstation responsive?)\n>> * a slow start to have different fetches in a different phase of the\n>> fetch protocol?\n>> * avoiding some subtle race?\n>>\n>> At the very least we would need a similar thing as Jeff recently sent for the\n>> push case with objects quarantined and then made available in one go?\n>\n> I don't think so. The object database is perfectly happy with multiple\n> simultaneous writers, and nothing impacts the have/wants until actual\n> refs are written. Quarantining objects before the refs are written is an\n> orthogonal concept.\n\nIf a remote advertises its tips, we'd need to look these up (clientside) to\ndecide if we have them, and I do not think we'd do that via a reachability\ncheck, but via direct lookup in the object data base? So I do not quite\nunderstand, what we gain from the atomic ref writes in e.g. remote/origin/.\n\n\n> I'm not altogether convinced that parallel fetch would be that much\n> faster, though.\n\nOk, time to present data... Let's assume a degenerate case first:\n\"up-to-date with all remotes\" because that is easy to reproduce.\n\nI have 14 remotes currently:\n\n$ time git fetch --all\nreal 0m18.016s\nuser 0m2.027s\nsys 0m1.235s\n\n$ time git config --get-regexp remote.*.url |awk '{print $2}' |xargs\n-P 14 -I % git fetch %\nreal 0m5.168s\nuser 0m2.312s\nsys 0m1.167s\n\nA factor of >3, so I suspect there is improvement ;)\n\nWell just as Ævar pointed out, there is some improvement.\n\n>\n> I usually just do a one-off fetch of their URL in such a case, exactly\n> because I _don't_ want to end up with a bunch of remotes. You can also\n> mark them with skipDefaultUpdate if you only care about them\n> occasionally (so you can \"git fetch sbeller\" when you care about it, but\n> it doesn't slow down your daily \"git fetch\").\n\nAnd I assume you don't want the remotes because it takes time to fetch and not\nbecause your disk space is expensive. ;)\n\n>\n> -Peff\n"},{"id":"303958","messageId":"20161012013428.swxmrbyxv2wo37xf@sigill.intra.peff.net","threadId":"44279","inReplyTo":"CAGZ79kaKOiy-HJboaujXXc66P6CLupteDw4JyPOGetREfz_q_Q@mail.gmail.com","subject":"Re: Make `git fetch --all` parallel?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-10-12T01:34:28Z","receivedAt":"2016-10-12T01:34:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 11, 2016 at 04:18:15PM -0700, Stefan Beller wrote:\n\n> >> At the very least we would need a similar thing as Jeff recently sent for the\n> >> push case with objects quarantined and then made available in one go?\n> >\n> > I don't think so. The object database is perfectly happy with multiple\n> > simultaneous writers, and nothing impacts the have/wants until actual\n> > refs are written. Quarantining objects before the refs are written is an\n> > orthogonal concept.\n> \n> If a remote advertises its tips, we'd need to look these up (clientside) to\n> decide if we have them, and I do not think we'd do that via a reachability\n> check, but via direct lookup in the object data base? So I do not quite\n> understand, what we gain from the atomic ref writes in e.g. remote/origin/.\n\nIt's been a while since I've dug into the fetch protocol. But I think we\ncover the \"do we have the objects already\" check via quickfetch(), which\ndoes do a reachability check, And then we advertise our \"have\" commits\nby walking backwards from our ref tips, so everything there is\nreachable.\n\nAnything else would be questionable, especially under older versions of\ngit, as we promise only to have a complete graph for objects reachable\nfrom the refs. Older versions of git would happily truncate unreachable\nhistory based on the 2-week prune expiration period.\n\n> > I'm not altogether convinced that parallel fetch would be that much\n> > faster, though.\n> \n> Ok, time to present data... Let's assume a degenerate case first:\n> \"up-to-date with all remotes\" because that is easy to reproduce.\n> \n> I have 14 remotes currently:\n> \n> $ time git fetch --all\n> real 0m18.016s\n> user 0m2.027s\n> sys 0m1.235s\n> \n> $ time git config --get-regexp remote.*.url |awk '{print $2}' |xargs\n> -P 14 -I % git fetch %\n> real 0m5.168s\n> user 0m2.312s\n> sys 0m1.167s\n\nSo first, thank you (and Ævar) for providing real numbers. It's clear\nthat I was talking nonsense.\n\nSecond, I wonder where all that time is going. Clearly there's an\nend-to-end latency issue, but I'm not sure where it is. Is it startup\ntime for git-fetch? Is it in getting and processing the ref\nadvertisement from the other side? What I'm wondering is if there are\nopportunities to speed up the serial case (but nobody really cared\nbefore because it doesn't matter unless you're doing 14 of them back to\nback).\n\n> > I usually just do a one-off fetch of their URL in such a case, exactly\n> > because I _don't_ want to end up with a bunch of remotes. You can also\n> > mark them with skipDefaultUpdate if you only care about them\n> > occasionally (so you can \"git fetch sbeller\" when you care about it, but\n> > it doesn't slow down your daily \"git fetch\").\n> \n> And I assume you don't want the remotes because it takes time to fetch and not\n> because your disk space is expensive. ;)\n\nThat, and it clogs the ref namespace. You can mostly ignore the extra\nrefs, but they show up in the \"git checkout ...\" DWIM, for example.\n\n-Peff\n"},{"id":"303960","messageId":"20161012015224.g2eb24jexepeewob@sigill.intra.peff.net","threadId":"44279","inReplyTo":"20161012013428.swxmrbyxv2wo37xf@sigill.intra.peff.net","subject":"Re: Make `git fetch --all` parallel?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-10-12T01:52:25Z","receivedAt":"2016-10-12T01:53:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 11, 2016 at 09:34:28PM -0400, Jeff King wrote:\n\n> > Ok, time to present data... Let's assume a degenerate case first:\n> > \"up-to-date with all remotes\" because that is easy to reproduce.\n> > \n> > I have 14 remotes currently:\n> > \n> > $ time git fetch --all\n> > real 0m18.016s\n> > user 0m2.027s\n> > sys 0m1.235s\n> > \n> > $ time git config --get-regexp remote.*.url |awk '{print $2}' |xargs\n> > -P 14 -I % git fetch %\n> > real 0m5.168s\n> > user 0m2.312s\n> > sys 0m1.167s\n> \n> So first, thank you (and Ævar) for providing real numbers. It's clear\n> that I was talking nonsense.\n> \n> Second, I wonder where all that time is going. Clearly there's an\n> end-to-end latency issue, but I'm not sure where it is. Is it startup\n> time for git-fetch? Is it in getting and processing the ref\n> advertisement from the other side? What I'm wondering is if there are\n> opportunities to speed up the serial case (but nobody really cared\n> before because it doesn't matter unless you're doing 14 of them back to\n> back).\n\nHmm. I think it really might be just network latency. Here's my fetch\ntime:\n\n $ git config remote.origin.url\n git://github.com/gitster/git.git\n\n $ time git fetch origin\n real 0m0.183s\n user 0m0.072s\n sys 0m0.008s\n\n14 of those in a row shouldn't take more than about 2.5 seconds, which\nis still twice as fast as your parallel case. So what's going on?\n\nOne is that I live about a hundred miles from GitHub's data center, and\nmy ping time there is ~13ms. The other side of the country, let alone\nEurope, is going to be noticeably slower just for the TCP handshake.\n\nThe second is that git:// is really cheap and simple. git-over-ssh is\nover twice as slow:\n\n $ time git fetch git@github.com:gitster/git\n ...\n real 0m0.432s\n user 0m0.100s\n sys 0m0.032s\n\nHTTP fares better than I would have thought, but is also slower:\n\n $ time git fetch https://github.com/gitster/git\n ...\n real 0m0.258s\n user 0m0.080s\n sys 0m0.032s\n\n-Peff\n"},{"id":"303965","messageId":"CAGZ79kbvdEUj5JKQX2MJ9THG03Bwfr0nSE5z-VPTmE3H+6TY2g@mail.gmail.com","threadId":"44279","inReplyTo":"20161012015224.g2eb24jexepeewob@sigill.intra.peff.net","subject":"Re: Make `git fetch --all` parallel?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-12T06:47:55Z","receivedAt":"2016-10-12T06:48:18Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Oct 11, 2016 at 6:52 PM, Jeff King wrote:\n> On Tue, Oct 11, 2016 at 09:34:28PM -0400, Jeff King wrote:\n>\n>> > Ok, time to present data... Let's assume a degenerate case first:\n>> > \"up-to-date with all remotes\" because that is easy to reproduce.\n>> >\n>> > I have 14 remotes currently:\n>> >\n>> > $ time git fetch --all\n>> > real 0m18.016s\n>> > user 0m2.027s\n>> > sys 0m1.235s\n>> >\n>> > $ time git config --get-regexp remote.*.url |awk '{print $2}' |xargs\n>> > -P 14 -I % git fetch %\n>> > real 0m5.168s\n>> > user 0m2.312s\n>> > sys 0m1.167s\n>>\n>> So first, thank you (and Ævar) for providing real numbers. It's clear\n>> that I was talking nonsense.\n>>\n>> Second, I wonder where all that time is going. Clearly there's an\n>> end-to-end latency issue, but I'm not sure where it is. Is it startup\n>> time for git-fetch? Is it in getting and processing the ref\n>> advertisement from the other side? What I'm wondering is if there are\n>> opportunities to speed up the serial case (but nobody really cared\n>> before because it doesn't matter unless you're doing 14 of them back to\n>> back).\n>\n> Hmm. I think it really might be just network latency. Here's my fetch\n> time:\n>\n> $ git config remote.origin.url\n> git://github.com/gitster/git.git\n>\n> $ time git fetch origin\n> real 0m0.183s\n> user 0m0.072s\n> sys 0m0.008s\n>\n> 14 of those in a row shouldn't take more than about 2.5 seconds, which\n> is still twice as fast as your parallel case. So what's going on?\n>\n> One is that I live about a hundred miles from GitHub's data center, and\n> my ping time there is ~13ms. The other side of the country, let alone\n> Europe, is going to be noticeably slower just for the TCP handshake.\n>\n> The second is that git:// is really cheap and simple. git-over-ssh is\n> over twice as slow:\n>\n> $ time git fetch git@github.com:gitster/git\n> ...\n> real 0m0.432s\n> user 0m0.100s\n> sys 0m0.032s\n>\n> HTTP fares better than I would have thought, but is also slower:\n>\n> $ time git fetch https://github.com/gitster/git\n> ...\n> real 0m0.258s\n> user 0m0.080s\n> sys 0m0.032s\n>\n> -Peff\n\nWell 9/14 are https for me, the rest is git://\nAlso 9/14 (but a different set) is github, the rest is\neither internal or kernel.org.\n\nFetching from github (https) is only 0.9s from here\n(SF bay area, I'm not in Europe any more ;) )\n\nI would have expected to have a speedup\nof roughly 2 + latency gains. Factor 2 because\nin the current state of affairs either the client or the\nremote is working, i.e. the other sie is idle/waiting, so\nfactor 2 seemed reasonable (and ofc the latency), so I\nwas a bit surprised to see a higher yield.\n"}]}