{"thread":{"id":"1488","subject":"Cloning speed comparison","startedAt":"2005-08-13T01:54:02Z","lastAt":"2005-08-15T21:27:46Z","messageCount":11,"participants":["Petr Baudis","Linus Torvalds","H. Peter Anvin","Daniel Barkalow","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"7161","messageId":"20050813015402.GC20812@pasky.ji.cz","threadId":"1488","inReplyTo":null,"subject":"Cloning speed comparison","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-08-13T01:54:02Z","receivedAt":"2005-08-13T01:54:02Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"  Hello,\n\n  I've wondered how slow the protocols other than rsync are, and the\n(well, a bit dubious; especially wrt. caching on the remote side)\nresults are:\n\n\tgit\tclone-pack:ssh\t25s\n\tgit\trsync\t\t27s\n\tgit\thttp-pull\t47s\n\tgit\tdumb-http\t54s\n\tgit\tssh-pull\t660s\n\n\tcogito\tclone-pack:ssh\t35s (!)\n\tcogito\trsync\t\t140s\n\tcogito\tssh-pull\t480s\n\tcogito\thttp-pull\textrapolated to about an hour!\n\tcogito\tdumb-http\tN/A (missing info in the repository)\n\n  (I didn't test the git server protocol, since kernel.org doesn't run\ngit server and I was too lazy to setup one.)\n\n  The git repository contains one big pack, one small pack and few\nstandalone objects (5882 objects in total), while cogito is standalone\nobjects only (9670 objects in total, 8681 reachable).\n\n  The numbers are off by some epsilons, as I didn't bother with multiple\nmeasures, but shouldn't be hugely off for a general comparison. The\nnetwork connection has 2048kbit/s download, the other side was\nwww.kernel.org for HTTP and rsync, and master.kernel.org for ssh.\n\n  Pulling from localhost (128M of RAM, 5M to 30M free - awful, yes):\n\n\tcogito\trsync:ssh\t150s\n\tcogito\tssh-pull\t120s (but didn't complete, see PS)\n\tcogito\thttp-pull\t260s\n\tcogito\tclone-pack:ssh\t340s\n\n  Anyway, clone-pack is a clear winner for networks (but someone should\nre-check that, especially compared to rsync, wrt. server-side file\ncaching); really cool fast, but not very practical for anonymous access.\nAny volunteers for a simple CGI (or gitweb addon) script + HTTP support\nin clone-pack? HTTP is certainly the most suitable protocol for\nanonymous pulls, so it's a shame it's still that sluggish.\n\n  It is so slow here since it has some very ugly access pattern on the\nobjects database and my RAM is full so it does not get cached; even on\nthe servers, it was slower at first - unfortunately, I didn't measure\nthat, so what's in the top table are second accesses. Still, I would\nexpect the big repositories to stay mostly in the server cache, so this\nisn't that big problem for those, I think.\n\n  PS:\n\tWith the latest git version as of time of writing this:\n\t$ time cg-clone git+ssh://pasky@localhost/home/pasky/WWW/dev/git/.g cogito\n\t...\n\tprogress: 5759 objects, 10292457 bytes\n\t$ time cg-clone http://localhost/~pasky/dev/git/.g cogito\n\t...\n\tprogress: 8681 objects, 14881571 bytes\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nIf you want the holes in your knowledge showing up try teaching\nsomeone.  -- Alan Cox\n"},{"id":"7162","messageId":"Pine.LNX.4.58.0508121908180.3295@g5.osdl.org","threadId":"1488","inReplyTo":"20050813015402.GC20812@pasky.ji.cz","subject":"Re: Cloning speed comparison","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-08-13T02:12:26Z","receivedAt":"2005-08-13T02:12:26Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 13 Aug 2005, Petr Baudis wrote:\n> \n>   Anyway, clone-pack is a clear winner for networks (but someone should\n> re-check that, especially compared to rsync, wrt. server-side file\n> caching); really cool fast, but not very practical for anonymous access.\n\ngit-daemon is for the anonymous access case, either started from inetd \n(or any other external \"listen to port, exec service\" thing), or with the \nbuilt-in listening stuff.\n\nIt uses exactly the same protocol and logic as the regular ssh clone-pack \nthing, except it doesn't authenticate the remote end: it only checks that \nthe local end is accepting anonymous pulls by checking whether there is a \n\"git-daemon-export-ok\" file in the git directory.\n\nIn my tests, the git daemon was noticeably faster than ssh, if only \nbecause the authentication actually tends to be a big part of the overhead \nin small pulls.\n\n[ Hey. There's a deer outside my window eating our roses again. Cute ]\n\n\t\t\tLinus\n"},{"id":"7163","messageId":"20050813031025.GE20812@pasky.ji.cz","threadId":"1488","inReplyTo":"Pine.LNX.4.58.0508121908180.3295@g5.osdl.org","subject":"Re: Cloning speed comparison","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-08-13T03:10:25Z","receivedAt":"2005-08-13T03:10:25Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sat, Aug 13, 2005 at 04:12:26AM CEST, I got a letter\nwhere Linus Torvalds <torvalds@osdl.org> told me that...\n> On Sat, 13 Aug 2005, Petr Baudis wrote:\n> > \n> >   Anyway, clone-pack is a clear winner for networks (but someone should\n> > re-check that, especially compared to rsync, wrt. server-side file\n> > caching); really cool fast, but not very practical for anonymous access.\n> \n> git-daemon is for the anonymous access case, either started from inetd \n> (or any other external \"listen to port, exec service\" thing), or with the \n> built-in listening stuff.\n> \n> It uses exactly the same protocol and logic as the regular ssh clone-pack \n> thing, except it doesn't authenticate the remote end: it only checks that \n> the local end is accepting anonymous pulls by checking whether there is a \n> \"git-daemon-export-ok\" file in the git directory.\n> \n> In my tests, the git daemon was noticeably faster than ssh, if only \n> because the authentication actually tends to be a big part of the overhead \n> in small pulls.\n\nOh. Sounds nice, are there plans to run this on kernel.org too? (So far,\n90% of my GIT network activity happens with kernel.org; the rest is with\nmy notebook, and I want to keep that ssh.)\n\nBTW, is the pack protocol flexible enough to be extended to support\npushing? That would be great as well. You might suggest just using ssh,\nbut that (i) requires you to be root on the machine to add new users\n(ii) consequently adds administrative burden (iii) isn't easy to set up\nso that the user has no shell access, shall you want to restrict that.\n\n> [ Hey. There's a deer outside my window eating our roses again. Cute ]\n\nOh, it must be nice in Oregon. I can't imagine anything like that to\nhappen in Czechia unless you live at a solitude or in some lonely tiny\nvillage.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nIf you want the holes in your knowledge showing up try teaching\nsomeone.  -- Alan Cox\n"},{"id":"7165","messageId":"Pine.LNX.4.58.0508122021490.19049@g5.osdl.org","threadId":"1488","inReplyTo":"20050813031025.GE20812@pasky.ji.cz","subject":"Re: Cloning speed comparison","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-08-13T03:28:44Z","receivedAt":"2005-08-13T03:28:44Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 13 Aug 2005, Petr Baudis wrote:\n>\n> Oh. Sounds nice, are there plans to run this on kernel.org too? (So far,\n> 90% of my GIT network activity happens with kernel.org; the rest is with\n> my notebook, and I want to keep that ssh.)\n\nMaybe. I don't know what the status of that is, but the plan was to at \nleast give it a try.\n\n> BTW, is the pack protocol flexible enough to be extended to support\n> pushing?\n\nThe _protocol_ could handle it, but you obviously need some kind of secure \nauthentication, and quite frankly, one of the selling points on git-daemon \nright now is that it's all read-only and very simple and there should be \nno security issues because it will never write anything at all.\n\nSo right now git-daemon only accepts requests from fetch-pack.\n\n> > [ Hey. There's a deer outside my window eating our roses again. Cute ]\n> \n> Oh, it must be nice in Oregon. I can't imagine anything like that to\n> happen in Czechia unless you live at a solitude or in some lonely tiny\n> village.\n\nDeer are really just oversized rats with horns (*). They're cute, though,\nand it's kind of funny looking up from the screen and noticing one\nmunching on the roses just ten feet away. \n\n\t\t\tLinus\n\n(*) Did I mention that biology wasn't one of the things I did at Uni?\n"},{"id":"7170","messageId":"42FD8224.9020700@zytor.com","threadId":"1488","inReplyTo":"Pine.LNX.4.58.0508122021490.19049@g5.osdl.org","subject":"Re: Cloning speed comparison","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-08-13T05:16:20Z","receivedAt":"2005-08-13T05:16:20Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n>>BTW, is the pack protocol flexible enough to be extended to support\n>>pushing?\n> \n> The _protocol_ could handle it, but you obviously need some kind of secure \n> authentication, and quite frankly, one of the selling points on git-daemon \n> right now is that it's all read-only and very simple and there should be \n> no security issues because it will never write anything at all.\n> \n\nRunning it over ssh would be a good way to do authentication...\n\n\t-hpa\n"},{"id":"7171","messageId":"42FD8237.8080400@zytor.com","threadId":"1488","inReplyTo":"20050813031025.GE20812@pasky.ji.cz","subject":"Re: Cloning speed comparison","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-08-13T05:16:39Z","receivedAt":"2005-08-13T05:16:39Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Petr Baudis wrote:\n>>>\n>>In my tests, the git daemon was noticeably faster than ssh, if only \n>>because the authentication actually tends to be a big part of the overhead \n>>in small pulls.\n> \n> Oh. Sounds nice, are there plans to run this on kernel.org too? (So far,\n> 90% of my GIT network activity happens with kernel.org; the rest is with\n> my notebook, and I want to keep that ssh.)\n> \n\nYes, when I get some time...\n\n\t-hpa\n"},{"id":"7172","messageId":"Pine.LNX.4.58.0508122223280.19049@g5.osdl.org","threadId":"1488","inReplyTo":"42FD8224.9020700@zytor.com","subject":"Re: Cloning speed comparison","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-08-13T05:25:21Z","receivedAt":"2005-08-13T05:25:21Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 12 Aug 2005, H. Peter Anvin wrote:\n> \n> Running it over ssh would be a good way to do authentication...\n\nWell, if you have ssh as an option, you don't need git-daemon any more, \nsince the protocol that git-daemon does runs quite well over ssh on its \nown...\n\nThe only point of git-daemon really is when you don't have ssh access (ie\nyou may want to give people a limited interface, but not full ssh). Ie\nas-is, it's only for anonymous reads of a git archive, but it obviously\n_could_ do more.\n\n\t\tLinus\n"},{"id":"7225","messageId":"42FE815F.8050201@zytor.com","threadId":"1488","inReplyTo":"Pine.LNX.4.58.0508122223280.19049@g5.osdl.org","subject":"Re: Cloning speed comparison","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-08-13T23:25:19Z","receivedAt":"2005-08-13T23:25:19Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n> On Fri, 12 Aug 2005, H. Peter Anvin wrote:\n> \n>>Running it over ssh would be a good way to do authentication...\n> \n> \n> Well, if you have ssh as an option, you don't need git-daemon any more, \n> since the protocol that git-daemon does runs quite well over ssh on its \n> own...\n> \n> The only point of git-daemon really is when you don't have ssh access (ie\n> you may want to give people a limited interface, but not full ssh). Ie\n> as-is, it's only for anonymous reads of a git archive, but it obviously\n> _could_ do more.\n> \n\nOkay.  So use git-daemon for the anonymous users, and run the git \nprotocol over ssh for writing.  Seems easy enough for me.\n\n\t-hpa\n"},{"id":"7304","messageId":"Pine.LNX.4.63.0508151321270.12816@iabervon.org","threadId":"1488","inReplyTo":"20050813015402.GC20812@pasky.ji.cz","subject":"Re: Cloning speed comparison","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-08-15T17:50:31Z","receivedAt":"2005-08-15T17:50:31Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sat, 13 Aug 2005, Petr Baudis wrote:\n\n>   Hello,\n> \n>   I've wondered how slow the protocols other than rsync are, and the\n> (well, a bit dubious; especially wrt. caching on the remote side)\n> results are:\n> \n> \tgit\tclone-pack:ssh\t25s\n> \tgit\trsync\t\t27s\n> \tgit\thttp-pull\t47s\n> \tgit\tdumb-http\t54s\n> \tgit\tssh-pull\t660s\n> \n> \tcogito\tclone-pack:ssh\t35s (!)\n> \tcogito\trsync\t\t140s\n> \tcogito\tssh-pull\t480s\n> \tcogito\thttp-pull\textrapolated to about an hour!\n\nI should be able to get http-pull down to the neighborhood of \n(current) ssh-pull; http-pull is that slow (when the source repository \nisn't packed) because it's entirely sequential, rather than overlapping \nrequests like ssh-pull now does.\n\nI should also be able to get ssh-pull down to the area of clone-pack, but \nthat's lower-priority, since there's clone-pack.\n\n(I've written an untested patch for local-pull, which I'll be testing, \ncleaning, and submitting tonight, assuming my newly-arrived monitor \nactually works)\n\n>   PS:\n> \tWith the latest git version as of time of writing this:\n> \t$ time cg-clone git+ssh://pasky@localhost/home/pasky/WWW/dev/git/.g cogito\n> \t...\n> \tprogress: 5759 objects, 10292457 bytes\n> \t$ time cg-clone http://localhost/~pasky/dev/git/.g cogito\n> \t...\n> \tprogress: 8681 objects, 14881571 bytes\n\nI've noticed that ssh connections don't actually disconnect at the end \nwith recent versions of ssh sometimes. In my experience, this occasionally \nhappens with git, but always happens with scp, suggesting that it's an ssh \nbug of some sort; I've also only noticed this with openssh 3.9_p1 with \nsome of Gentoo's -r2 patches.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"7312","messageId":"7vacjjx70u.fsf@assigned-by-dhcp.cox.net","threadId":"1488","inReplyTo":"Pine.LNX.4.63.0508151321270.12816@iabervon.org","subject":"Re: Cloning speed comparison","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-15T20:46:41Z","receivedAt":"2005-08-15T20:46:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> writes:\n\n> I should be able to get http-pull down to the neighborhood of \n> (current) ssh-pull; http-pull is that slow (when the source repository \n> isn't packed) because it's entirely sequential, rather than overlapping \n> requests like ssh-pull now does.\n\nI like those prefetch() and process() code in pull.c very much.\n\nI have been wondering if increasing parallelism more by\nprefetching beyond the immediate parents of the current commit,\nin \"if (get_history)\" part of process_commit().  Maybe it is not\nworth it because doing a commit, its associated tree(s) and its\nparents would already give us enough parallelism already.\n\n> (I've written an untested patch for local-pull, which I'll be testing, \n> cleaning, and submitting tonight, assuming my newly-arrived monitor \n> actually works)\n\nThat is a great news.  Thank you for doing this; looking forward\nto see it, but no rush.  Enjoy your new monitor.\n"},{"id":"7317","messageId":"Pine.LNX.4.63.0508151652290.21501@iabervon.org","threadId":"1488","inReplyTo":"7vacjjx70u.fsf@assigned-by-dhcp.cox.net","subject":"Re: Cloning speed comparison","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-08-15T21:27:46Z","receivedAt":"2005-08-15T21:27:46Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 15 Aug 2005, Junio C Hamano wrote:\n\n> Daniel Barkalow <barkalow@iabervon.org> writes:\n> \n> > I should be able to get http-pull down to the neighborhood of \n> > (current) ssh-pull; http-pull is that slow (when the source repository \n> > isn't packed) because it's entirely sequential, rather than overlapping \n> > requests like ssh-pull now does.\n> \n> I like those prefetch() and process() code in pull.c very much.\n> \n> I have been wondering if increasing parallelism more by\n> prefetching beyond the immediate parents of the current commit,\n> in \"if (get_history)\" part of process_commit().  Maybe it is not\n> worth it because doing a commit, its associated tree(s) and its\n> parents would already give us enough parallelism already.\n\nIt is actually already maxing out the parallelism; it has a FIFO of \nobjects which it needs, and calls prefetch() when it enqueues an object \nand fetch() when it dequeues it. It only cares about the dependancies for \nthis purpose, not the types.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"}]}