Re: git-fetch per-repository speed issues

From: Linus Torvalds <torvalds@osdl.org>
Date: Jul 3, 2006, 23:14 UTC
Message-ID: <Pine.LNX.4.64.0607031603290.12404@g5.osdl.org>
In-Reply-To: <1151949764.4723.51.camel@neko.keithp.com>

On Mon, 3 Jul 2006, Keith Packard wrote:

> 
> With git, we'd prefer to use the git protocol instead of rsync for the
> usual pack-related reasons, but that is limited to a single repository
> at a time.

Well, you could use multiple branches in the same repository, even if they are totally unrealated. That would allow you to fetch them all in one go.

One way to do that is to just name the branches hierarcially have one repo, but then call the branches something like

	libXrandr/master
	libXrandr/develop
	Xorg/master
	Xorg/develop
	..

Show 10 quoted lines

> And, it's painfully slow, even when the repository is up to
> date:
> 
> $ cd lib/libXrandr
> $ time git-fetch origin
> ...
> 
> real    0m17.035s
> user    0m2.584s
> sys     0m0.576s

That's _seriously_ wrong. If everything is up-to-date, a fetch should be basically zero-cost. That's especially true with the anonymous git protocol, which doesn't have any connection validation overhead (for the ssh protocol, the cost is usually the ssh login).

But there may well be some bug there.

Look at this:

	[torvalds@g5 git]$ time git fetch git://git.kernel.org/pub/scm/git/git.git 
	
	real    0m0.431s
	user    0m0.036s
	sys     0m0.024s

and that's over my DSL line, not some studly network thing.

Basically, a repo that is up-to-date should do a "git fetch" about as quickly as it does a "git ls-remote". Which in turn really shouldn't be doing much anything at all, apart from the connect itself:

	[torvalds@g5 git]$ time git ls-remote master.kernel.org:/pub/scm/git/git.git > /dev/null 
	
	real    0m1.758s
	user    0m0.188s
	sys     0m0.024s
	[torvalds@g5 git]$ time git ls-remote git://git.kernel.org/pub/scm/git/git.git > /dev/null 
	
	real    0m0.431s
	user    0m0.056s
	sys     0m0.016s

(note how the ssh connection is much slower - it actually ends up doing all the ssh back-and-forth).

Can you try from different hosts? One problem may be the remote end just trying to do reverse DNS lookups for xinetd or whatever?

Also, one thing to try is to just do

	strace -Ttt git-peek-remote ...

which shows where the time is going (I selected "git-peek-remote", because that's a simple program).

		Linus