Cloning speed comparison

  Hello,

  I've wondered how slow the protocols other than rsync are, and the
(well, a bit dubious; especially wrt. caching on the remote side)
results are:

	git	clone-pack:ssh	25s
	git	rsync		27s
	git	http-pull	47s
	git	dumb-http	54s
	git	ssh-pull	660s

	cogito	clone-pack:ssh	35s (!)
	cogito	rsync		140s
	cogito	ssh-pull	480s
	cogito	http-pull	extrapolated to about an hour!
	cogito	dumb-http	N/A (missing info in the repository)

  (I didn't test the git server protocol, since kernel.org doesn't run
git server and I was too lazy to setup one.)

  The git repository contains one big pack, one small pack and few
standalone objects (5882 objects in total), while cogito is standalone
objects only (9670 objects in total, 8681 reachable).

  The numbers are off by some epsilons, as I didn't bother with multiple
measures, but shouldn't be hugely off for a general comparison. The
network connection has 2048kbit/s download, the other side was
www.kernel.org for HTTP and rsync, and master.kernel.org for ssh.

  Pulling from localhost (128M of RAM, 5M to 30M free - awful, yes):

	cogito	rsync:ssh	150s
	cogito	ssh-pull	120s (but didn't complete, see PS)
	cogito	http-pull	260s
	cogito	clone-pack:ssh	340s

  Anyway, clone-pack is a clear winner for networks (but someone should
re-check that, especially compared to rsync, wrt. server-side file
caching); really cool fast, but not very practical for anonymous access.
Any volunteers for a simple CGI (or gitweb addon) script + HTTP support
in clone-pack? HTTP is certainly the most suitable protocol for
anonymous pulls, so it's a shame it's still that sluggish.

  It is so slow here since it has some very ugly access pattern on the
objects database and my RAM is full so it does not get cached; even on
the servers, it was slower at first - unfortunately, I didn't measure
that, so what's in the top table are second accesses. Still, I would
expect the big repositories to stay mostly in the server cache, so this
isn't that big problem for those, I think.

  PS:
	With the latest git version as of time of writing this:
	$ time cg-clone git+ssh://pasky@localhost/home/pasky/WWW/dev/git/.g cogito
	...
	progress: 5759 objects, 10292457 bytes
	$ time cg-clone http://localhost/~pasky/dev/git/.g cogito
	...
	progress: 8681 objects, 14881571 bytes

On Sat, 13 Aug 2005, Petr Baudis wrote:

git-daemon is for the anonymous access case, either started from inetd (or any other external "listen to port, exec service" thing), or with the built-in listening stuff.

It uses exactly the same protocol and logic as the regular ssh clone-pack thing, except it doesn't authenticate the remote end: it only checks that the local end is accepting anonymous pulls by checking whether there is a "git-daemon-export-ok" file in the git directory.

In my tests, the git daemon was noticeably faster than ssh, if only because the authentication actually tends to be a big part of the overhead in small pulls.

[ Hey. There's a deer outside my window eating our roses again. Cute ]

			Linus

Dear diary, on Sat, Aug 13, 2005 at 04:12:26AM CEST, I got a letter where Linus Torvalds <torvalds@osdl.org> told me that...

Oh. Sounds nice, are there plans to run this on kernel.org too? (So far, 90% of my GIT network activity happens with kernel.org; the rest is with my notebook, and I want to keep that ssh.)

BTW, is the pack protocol flexible enough to be extended to support pushing? That would be great as well. You might suggest just using ssh, but that (i) requires you to be root on the machine to add new users (ii) consequently adds administrative burden (iii) isn't easy to set up so that the user has no shell access, shall you want to restrict that.

Oh, it must be nice in Oregon. I can't imagine anything like that to happen in Czechia unless you live at a solitude or in some lonely tiny village.

On Sat, 13 Aug 2005, Petr Baudis wrote:

Maybe. I don't know what the status of that is, but the plan was to at least give it a try.

The _protocol_ could handle it, but you obviously need some kind of secure authentication, and quite frankly, one of the selling points on git-daemon right now is that it's all read-only and very simple and there should be no security issues because it will never write anything at all.

So right now git-daemon only accepts requests from fetch-pack.

Deer are really just oversized rats with horns (*). They're cute, though, and it's kind of funny looking up from the screen and noticing one munching on the roses just ten feet away.

			Linus

(*) Did I mention that biology wasn't one of the things I did at Uni?

Linus Torvalds wrote:

Running it over ssh would be a good way to do authentication...

	-hpa

On Fri, 12 Aug 2005, H. Peter Anvin wrote:

Well, if you have ssh as an option, you don't need git-daemon any more, since the protocol that git-daemon does runs quite well over ssh on its own...

The only point of git-daemon really is when you don't have ssh access (ie you may want to give people a limited interface, but not full ssh). Ie as-is, it's only for anonymous reads of a git archive, but it obviously _could_ do more.

		Linus

Linus Torvalds wrote:

Okay. So use git-daemon for the anonymous users, and run the git protocol over ssh for writing. Seems easy enough for me.

	-hpa

Petr Baudis wrote:

Yes, when I get some time...

	-hpa

On Sat, 13 Aug 2005, Petr Baudis wrote:

I should be able to get http-pull down to the neighborhood of (current) ssh-pull; http-pull is that slow (when the source repository isn't packed) because it's entirely sequential, rather than overlapping requests like ssh-pull now does.

I should also be able to get ssh-pull down to the area of clone-pack, but that's lower-priority, since there's clone-pack.

(I've written an untested patch for local-pull, which I'll be testing, cleaning, and submitting tonight, assuming my newly-arrived monitor actually works)

I've noticed that ssh connections don't actually disconnect at the end with recent versions of ssh sometimes. In my experience, this occasionally happens with git, but always happens with scp, suggesting that it's an ssh bug of some sort; I've also only noticed this with openssh 3.9_p1 with some of Gentoo's -r2 patches.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow <barkalow@iabervon.org> writes:

I like those prefetch() and process() code in pull.c very much.

I have been wondering if increasing parallelism more by prefetching beyond the immediate parents of the current commit, in "if (get_history)" part of process_commit(). Maybe it is not worth it because doing a commit, its associated tree(s) and its parents would already give us enough parallelism already.

That is a great news. Thank you for doing this; looking forward to see it, but no rush. Enjoy your new monitor.

On Mon, 15 Aug 2005, Junio C Hamano wrote:

It is actually already maxing out the parallelism; it has a FIFO of objects which it needs, and calls prefetch() when it enqueues an object and fetch() when it dequeues it. It only cares about the dependancies for this purpose, not the types.

	-Daniel
*This .sig left intentionally blank*