Re: What to expect after 0.99.8

From: Linus Torvalds <torvalds@osdl.org>
Date: Oct 3, 2005, 23:16 UTC
Message-ID: <Pine.LNX.4.64.0510031606550.31407@g5.osdl.org>
In-Reply-To: <7v1x32l0gz.fsf@assigned-by-dhcp.cox.net>

On Mon, 3 Oct 2005, Junio C Hamano wrote:

Show 10 quoted lines

> 
> This reminds me of one patch:
> 
>     From: Dan Aloni <da-x@monatomic.org>
>     Subject: [PATCH] Fix git+ssh's indefinite halts during long fetches
>     Date: Sat, 1 Oct 2005 21:39:42 +0300
>     Message-ID: <20051001183942.GA2099@localdomain>
> 
> I'd appreciate it if you had a chance to take a look at it and
> comment on it.

I personally hate it.

It adds horrible patches to fairly core stuff, all because the prefetching is not limited.

As far as I can tell, it should be much easier to just limit the prefetching to some reasonable limit (say, a few objects deep), which guarantees that the prefetching doesn't fill up the write queues on the fetching side.

It's not like prefetching improves performance once you get to the point where you can stream. I suspect having more than two or three objects "in flight" really only helps with

 - lots of small objects
 - high latency
 - high bandwidth

and the thing is, high latency together with high bandwidth is really quite uncommon - usually high latency goes along with _low_ bandwidth (the one exception is things like satellite links, which can have latencies in the seconds, even with good throughput).

It should be pretty easy to benchmark, but my _suspicion_ is that limiting the read-ahead to even just five is likely to get you 99% of the way, and that the performance impact of going higher is very limited.

(It might need some extra code to make the synchronous receiving side re-start the prefetching if the prefetching has stopped after a few entries - but at that point the extra code is where it is supposed to be, rather than having core code work around problems in the fetching. I also suspect that the prefetch limiting can happily be done in the generic "pull" code, rather than separately for each protocol, so it would need to be done in just one place).

		Linus