{"thread":{"id":"2474","subject":"Cloning speed comparison, round II","startedAt":"2005-11-12T15:53:02Z","lastAt":"2005-11-12T22:03:03Z","messageCount":8,"participants":["Petr Baudis","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"11683","messageId":"20051112155302.GD30496@pasky.or.cz","threadId":"2474","inReplyTo":null,"subject":"Cloning speed comparison, round II","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-11-12T15:53:02Z","receivedAt":"2005-11-12T15:53:02Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":" Hello,\n\n here comes another round of the cloning speed comparison. For the\nprevious round, see\n\n\t6395 F Aug 13 Petr Baudis ( 2.6K) Cloning speed comparison\n\tMessage-ID: <20050813015402.GC20812@pasky.ji.cz>\n\n What do we have here:\n\n\tgit.git: 10267 objects, mostly packed\n\tcogito.git: 99903903 objects, no packs\n\n (But please note that compared to the previous round, there are\nprobably more unpacked objects lying around in the git.git repository\n- actually roughly 1000.)\n\n Latest GIT, latest Cogito, performed by cg-clone to the official\nkernel.org hostnames for each transport. 'real' medians from several\ntries (number of tries usually at least three, based on their variance):\n\n\n rsync git+ssh(*) git(**) http\n\ngit.git 0m45s 0m34s 5m30s 4m01s (++)\n\ncogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)\n\n\n(*) git+ssh was to master.kernel.org, which was under significant load\n from some seemingly runaway gzip process, so that slowed things\n down.\n\n(**) The unpacking was slooooooooow yet the load was quite low. This\n should be investigated, the native git fetching is much slower\n than even HTTP.\n\n(+) There was unusually high variance here - the results range from\n 0m27s to 2m30s. In all other cases the results for all runs were\n very similiar.\n\n(++) Messy - it stalls at 72 objects, then does seemingly nothing for\n a long while, then requests a pack. Huh? Also, I saw the\n\n\tGetting alternates list for http://www.kernel.org/pub/scm/git/git.git/\n\n line about 100 times for the 72 objects (but never after the pack\n is fetched, at least).\n\n\n Conclusion: The GIT protocol has some major problem which is dragging\nit down to ridiculous slowness (isn't it supposed to be as fast as git+ssh\nor even slightly faster?). rsync unfortunately still provides the fastest\nanonymous access. HTTP is better than it was, but especially when fetching\npacks, there is a room for improvement - the mysterious stalls should be\neliminated, and it shouldn't get bogged down by repeated silly requests\nfor alternates lists etc.\n\n Compared to the last round, HTTP is much faster for unpacked object\nstores (thanks to parallel fetching), but much slower for packed object\nstores - however that can be caused by more objects lying unpacked now\nin the git.git repository. git+ssh (back then clone-pack:ssh) is as\nfast as usual - the 0m27s case suggests that it would be much faster\nfor Cogito if master.kernel.org wouldn't be under load.\n\n GIT protocol is the great disappointment. And I kind of hoped that I could\npush the rsync deprecation further because of the GIT protocol.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"11690","messageId":"Pine.LNX.4.64.0511121134170.3263@g5.osdl.org","threadId":"2474","inReplyTo":"20051112155302.GD30496@pasky.or.cz","subject":"Re: Cloning speed comparison, round II","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-12T19:40:11Z","receivedAt":"2005-11-12T19:40:11Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 12 Nov 2005, Petr Baudis wrote:\n>\n> rsync git+ssh(*) git(**) http\n> \n> git.git 0m45s 0m34s 5m30s 4m01s (++)\n> \n> cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)\n> \n> \n> (*) git+ssh was to master.kernel.org, which was under significant load\n> from some seemingly runaway gzip process, so that slowed things\n> down.\n> \n> (**) The unpacking was slooooooooow yet the load was quite low. This\n> should be investigated, the native git fetching is much slower\n> than even HTTP.\n\ngit:// and git+ssh:// should be the exact same protocol, the main \ndifference in this case being the server they go to.\n\nIn the case of (**), the unpacking itself is fast, but it's done as the \nstream of data comes in, so it will appear slow if the server at the other \nend is slow (or the network to that server is slow).\n\nSo I think the difference between your git+ssh and git tests are purely \ndue to the fact that master.kernel.org sees a lot less load (both in CPU \nand in networking) than the public sites, and the time differences have \nnothing to do with the protocol per se.\n\nI suspect master.kernel.org also has a beefier machine with more memory \n(but even if that's not the case, it's simply true that the public \nmachines obviously do mirroring to a lot of other machines, and run things \nlike webgit and just basic serving too).\n\nIf anything, git:// as a protocol is theoretically a bit faster, since the \nlogin procedure is faster and there's no encryption overhead.\n\n\t\t\tLinus\n"},{"id":"11691","messageId":"20051112194646.GE30496@pasky.or.cz","threadId":"2474","inReplyTo":"Pine.LNX.4.64.0511121134170.3263@g5.osdl.org","subject":"Re: Cloning speed comparison, round II","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-11-12T19:46:46Z","receivedAt":"2005-11-12T19:46:46Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sat, Nov 12, 2005 at 08:40:11PM CET, I got a letter\nwhere Linus Torvalds said that...\n> \n> \n> On Sat, 12 Nov 2005, Petr Baudis wrote:\n> >\n> > rsync git+ssh(*) git(**) http\n> > \n> > git.git 0m45s 0m34s 5m30s 4m01s (++)\n> > \n> > cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)\n> > \n> > \n> > (*) git+ssh was to master.kernel.org, which was under significant load\n> > from some seemingly runaway gzip process, so that slowed things\n> > down.\n> > \n> > (**) The unpacking was slooooooooow yet the load was quite low. This\n> > should be investigated, the native git fetching is much slower\n> > than even HTTP.\n\n(BTW, it's obviously not much slower than HTTP - I wrote that after the\nfirst git run, but I screwed up myself at that one.)\n\n> So I think the difference between your git+ssh and git tests are purely \n> due to the fact that master.kernel.org sees a lot less load (both in CPU \n> and in networking) than the public sites, and the time differences have \n> nothing to do with the protocol per se.\n\nWell, at the time of fetching, master.kernel.org with git+ssh had load\nabout ~3.5 and some wild gzip was eating most of the CPU there. So if\nthe git protocol still manages to be TEN times slower while rsync goes\nfull speed from that machine, I would say that this means the git server\nrequires way too much CPU.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"11693","messageId":"Pine.LNX.4.64.0511121159470.3263@g5.osdl.org","threadId":"2474","inReplyTo":"20051112194646.GE30496@pasky.or.cz","subject":"Re: Cloning speed comparison, round II","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-12T20:13:20Z","receivedAt":"2005-11-12T20:13:20Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 12 Nov 2005, Petr Baudis wrote:\n\n> Dear diary, on Sat, Nov 12, 2005 at 08:40:11PM CET, I got a letter\n> where Linus Torvalds said that...\n> > \n> > \n> > On Sat, 12 Nov 2005, Petr Baudis wrote:\n> > >\n> > > rsync git+ssh(*) git(**) http\n> > > \n> > > git.git 0m45s 0m34s 5m30s 4m01s (++)\n> > > \n> > > cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)\n> \n> Well, at the time of fetching, master.kernel.org with git+ssh had load\n> about ~3.5 and some wild gzip was eating most of the CPU there. So if\n> the git protocol still manages to be TEN times slower while rsync goes\n> full speed from that machine, I would say that this means the git server\n> requires way too much CPU.\n\nLook again.\n\nmaster.kernel.org was _faster_ than rsync using the native git protocol, \ndespite being under a load of 3.5.\n\nLook at the numbers: 45 secs for rsync, 34 secs for git protocol to \nmaster.\n\nNow, I don't know which rsync machine you used (you can rsync both from \nmaster and from rync.kernel.org), since you don't say. \n\nNow, it's unquestionably true that rsync can be faster under many \ncircumstances. Most notably when disk IO is really slow, since the native \ngit protocol will do a lot more synchronous operations, since it actually \ntests what it is doing.\n\nBut I _guarantee_ you that rsync is at least ten times slower than the \nnative git protocol in many circumstances. It can't handle repacking \n(which is critical for good server performance).\n\nAnd in fact it can't handle totally unpacked directories and small updates \nwell either (the reason I totally stopped doing rsync was because it took \nminutes to go through the whole list of unpacked objects for a small \nupdate, while the native protocol would just fetch the needed objects and \nbe done with it.\n\nSo sometimes rsync is faster, sometimes the git protocol is faster. But \nthe git protocol is _always_ better from a sanity standpoint.\n\nThe things you get with the native git protocol:\n\n - you don't have to trust the other end. If the other end lies about the \n SHA1's of its objects, rsync will never know. It will just download the \n thing, and you may have a corrupt database.\n\n With the git protocol, we just get the objects, and recompute their \n names. The other end can't lie about what their SHA is.\n\n - the rsync protocol totally breaks down with multiple branches. It \n fetches stuff it shouldn't because it doesn't know better.\n\n - the rsync protocol scales with project size, not with change size. This \n works well for small projects, where the changes are usually not all \n that hugely different from the total size of the project, but it really \n sucks for big projects.\n\n - the rsync protocol fundamentally cannot handle two differently packed \n trees well. That doesn't matter if you only track one tree, but it \n matters _hugely_ for people (like me) who pull from tens of different \n trees.\n\nSo the fact is: rsync is often slower, and _always_ less capable. \n\n\t\t\tLinus\n"},{"id":"11695","messageId":"Pine.LNX.4.64.0511121229340.3263@g5.osdl.org","threadId":"2474","inReplyTo":"Pine.LNX.4.64.0511121159470.3263@g5.osdl.org","subject":"Re: Cloning speed comparison, round II","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-12T20:31:32Z","receivedAt":"2005-11-12T20:31:32Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 12 Nov 2005, Linus Torvalds wrote:\n> \n> So the fact is: rsync is often slower, and _always_ less capable. \n\nSide note: a lot of the rsync problems are non-issues for the initial \nclone. \n\nThat initial clone is in fact the only time I think rsync can be a good \nidea, especially if the server end is repacking regularly.\n\nThen the only downside of cloning using rsync is that it will obviously \nalso set up the \"origin\" branch to be updated using rsync, which is sad if \nthere are alternatives.\n\nBut it may be a good idea to first clone using rsync, and then edit the \norigin file to change the rsync into a git-native protocol if one is \navailable.\n\n\t\tLinus\n"},{"id":"11699","messageId":"20051112212030.GH30496@pasky.or.cz","threadId":"2474","inReplyTo":"Pine.LNX.4.64.0511121159470.3263@g5.osdl.org","subject":"Re: Cloning speed comparison, round II","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-11-12T21:20:30Z","receivedAt":"2005-11-12T21:20:30Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sat, Nov 12, 2005 at 09:13:20PM CET, I got a letter\nwhere Linus Torvalds said that...\n> On Sat, 12 Nov 2005, Petr Baudis wrote:\n> \n> > Dear diary, on Sat, Nov 12, 2005 at 08:40:11PM CET, I got a letter\n> > where Linus Torvalds said that...\n> > > \n> > > \n> > > On Sat, 12 Nov 2005, Petr Baudis wrote:\n> > > >\n> > > > rsync git+ssh(*) git(**) http\n> > > > \n> > > > git.git 0m45s 0m34s 5m30s 4m01s (++)\n> > > > \n> > > > cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)\n> > \n> > Well, at the time of fetching, master.kernel.org with git+ssh had load\n> > about ~3.5 and some wild gzip was eating most of the CPU there. So if\n> > the git protocol still manages to be TEN times slower while rsync goes\n> > full speed from that machine, I would say that this means the git server\n> > requires way too much CPU.\n> \n> Look again.\n> \n> master.kernel.org was _faster_ than rsync using the native git protocol, \n> despite being under a load of 3.5.\n> \n> Look at the numbers: 45 secs for rsync, 34 secs for git protocol to \n> master.\n\nThat's git+ssh (not git transport protocol - this DID make a difference\nat first, I know it's hard to believe) on machine A vs. rsync on\nmachine B.\n\n> Now, I don't know which rsync machine you used (you can rsync both from \n> master and from rync.kernel.org), since you don't say. \n\nrsync.kernel.org (\"... official kernel.org hostnames for each transport.\")\n\n\nSo I had this length more detailed benchmark which re-examined git+ssh\nvs. git on master.kernel.org (good that git-daemon binds to\nnon-priviledged port). However, an embarrassing thing turned out - this\nseems to be a network problem, actually.\n\n From kernel.org to machine X, it takes 20s, and from machine X to my\nhome machine, it takes 20s. From kernel.org to my home machine, it takes\n5 minutes. This concerns only the git protocols, HTTP and SSH from my\nhome machine to kernel.org is fast as usual. Even stranger, this is the\nsame for both hera.kernel.org and zeus.kernel.org, while each is at\ntotally different network. Tried many times. Ugh.\n\nWell, so sorry for false alarm.\n\nYour totally puzzled,\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"11700","messageId":"20051112213002.GI30496@pasky.or.cz","threadId":"2474","inReplyTo":"Pine.LNX.4.64.0511121229340.3263@g5.osdl.org","subject":"Re: Cloning speed comparison, round II","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-11-12T21:30:02Z","receivedAt":"2005-11-12T21:30:02Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sat, Nov 12, 2005 at 09:31:32PM CET, I got a letter\nwhere Linus Torvalds said that...\n> On Sat, 12 Nov 2005, Linus Torvalds wrote:\n> > \n> > So the fact is: rsync is often slower, and _always_ less capable. \n> \n> Side note: a lot of the rsync problems are non-issues for the initial \n> clone. \n\nBut based on my \"machine X\" tests, git is as fast as git+ssh (which is\nas it should be), which means it is even slightly faster than rsync,\neven for the initial commit.\n\n> That initial clone is in fact the only time I think rsync can be a good \n> idea, especially if the server end is repacking regularly.\n\nYes, the only advantage of rsync I can see is that you get it all\npacked. Well, I think this only means we are doing something wrong,\nand perhaps we should automatically pack as well.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"11703","messageId":"Pine.LNX.4.64.0511121352500.3263@g5.osdl.org","threadId":"2474","inReplyTo":"20051112212030.GH30496@pasky.or.cz","subject":"Re: Cloning speed comparison, round II","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-12T22:03:03Z","receivedAt":"2005-11-12T22:03:03Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 12 Nov 2005, Petr Baudis wrote:\n> \n> From kernel.org to machine X, it takes 20s, and from machine X to my\n> home machine, it takes 20s. From kernel.org to my home machine, it takes\n> 5 minutes.\n\nThat's a twilight zone moment ;)\n\nThe pack-file writer is even trying to be good about not doing lots of \nsmall writes (it should chunk things up into 8kB chunks) so it should \nactually be a nice network app and send full-sized TCP frames from the \nvery beginning (and nagle should mean that it continues to do so even \naround the chunk boundaries, assuming the server is fast enough at \ngenerating the data).\n\nBut it's entirely possible that one of the paths between the machine has \nsome logic that prioritizes known stream types - gives higher priority to \nhttp/ssh over \"unknown\" protocols. That's a bad thing to do (the whole \npoint of the internet is that the smarts is in the end-points, not in the \nnetwork), but with so much of the packets whizzing by being \nvirus-generated crap, some places apparently do things like that.\n\n\t\t\tLinus\n"}]}