Re: git-fetch per-repository speed issues

From: Linus Torvalds <torvalds@osdl.org>
Date: Jul 4, 2006, 03:21 UTC
Message-ID: <Pine.LNX.4.64.0607032008590.12404@g5.osdl.org>
In-Reply-To: <1151973438.4723.70.camel@neko.keithp.com>

On Mon, 3 Jul 2006, Keith Packard wrote:

Show 8 quoted lines

> On Mon, 2006-07-03 at 16:14 -0700, Linus Torvalds wrote:
> > 
> > Well, you could use multiple branches in the same repository, even if they 
> > are totally unrealated. That would allow you to fetch them all in one go.
> 
> I'd like to avoid this; the hope is that most people won't ever need to
> look at most repositories; it would be somewhat like having glibc in the
> same repo as the kernel...

Sure, understood. I'm just saying that if you want to fetch in one go, it's one possibility.

However, your setup has something else seriously wrong.

> Yeah, I tried with the git protocol and it's a few seconds faster (about
> 14 seconds instead of 17). Ick.

That's -still- about 13 seconds too much.

> I think it might have something to do with the number of heads we're
> tracking.

It really shouldn't matter. You get all the heads in one go with a single connection, so if 32 heads takes 32 times longer, there's something wrong.

Show 5 quoted lines

> > Also, one thing to try is to just do
> > 
> > 	strace -Ttt git-peek-remote ...
> 
> That's plenty fast, 0.410 seconds, with nothing ugly in the strace.

Ok, a "git fetch" really shouldn't take any longer than a single connection. However, the fact that you have 32 heads, and it takes pretty close to _exactly_ 32 times 0.410 seconds (32*0.410s = 13.1s) makes me suspect that "git fetch" is just broken and fetches one branch at a time.

Which would be just stupid.

But look as I might, I see only that one "git-fetch-pack" in git-fetch.sh that should trigger. Once. Not 32 times. But your timings sure sound like it's doing a _lot_ more than it should.

Junio, any ideas?

Keithp, can you try this trivial patch? It _should_ say something like

	Fetching
	refs/heads/master
	refs/heads/...
	refs/heads/...
	...
	refs/heads/... from git://..../...

and more importantly, it should say so only once.

And then it should leave a "fetch.trace" file in your working directory, which should show where that _one_ thing spends its time.

		Linus

----

diff --git a/git-fetch.sh b/git-fetch.sh
index 48818f8..4739202 100755
--- a/git-fetch.sh
+++ b/git-fetch.sh
@@ -339,6 +339,8 @@ fetch_main () {
     ( : subshell because we muck with IFS
       IFS=" 	$LF"
       (
+	  echo "Fetching $rref from $remote" >&2
+	  strace -o fetch.trace -Ttt \
 	  git-fetch-pack $exec $keep --thin "$remote" $rref || echo failed "$remote"
       ) |
       while read sha1 remote_name