threads / rfc / 2140

[RFC] Timeouts on HTTP requests

Subject: [RFC] Timeouts on HTTP requests

## tl;dr

4 messages between Oct 18, 2005 and Oct 19, 2005.

replies: 3people: 3as markdown or json

Nick Hengeveld· Oct 18, 2005, 23:51 UTC · lore

Our QA department today checked what would happen if the network connection went away completely in the middle of an HTTP transfer. It looks as though the answer is that git-http-fetch sits there forever waiting for CURL to return something.

I'm thinking of taking advantage of CURL's capability of aborting a request if the transfer rate drops below a threshold for a specified length of time using a new pair of environment variables and/or config file settings:

GIT_HTTP_LOW_SPEED_LIMIT/http.lowspeedlimit GIT_HTTP_LOW_SPEED_TIME/http.lowspeedtime

Does this make sense, and if so should there be defaults if nothing is specified?

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.
Junio C Hamano· Oct 19, 2005, 06:02 UTC · re: Nick Hengeveld · lore

Re: [RFC] Timeouts on HTTP requests

Nick Hengeveld <nickh@reactrix.com> writes:
> Our QA department today checked what would happen if the network connection
> went away completely in the middle of an HTTP transfer.  It looks as though
> the answer is that git-http-fetch sits there forever waiting for CURL to
> return something.
Ouch.
Show 9 quoted lines
> I'm thinking of taking advantage of CURL's capability of aborting a request
> if the transfer rate drops below a threshold for a specified length of time
> using a new pair of environment variables and/or config file settings:
>
> GIT_HTTP_LOW_SPEED_LIMIT/http.lowspeedlimit
> GIT_HTTP_LOW_SPEED_TIME/http.lowspeedtime
>
> Does this make sense, and if so should there be defaults if nothing is
> specified?

I suspect these would be quite different between DSL and localnet, so I doubt if there is a reasonable default value to quick give-up.

On the other hand, having _no_ activity for say 30 seconds would indicate a dead link on either modem or localnet.

BTW, I've been thinking about giving defaults by shipping templates/config (i.e. no compile-time defaults). One trick I found cute is to have "clone.keeppack = 1" in the templates to be applied for any newly built repository, especially now kernel.org has git-daemon enabled.

Petr Baudis· Oct 19, 2005, 11:09 UTC · re: Junio C Hamano · lore

Re: [RFC] Timeouts on HTTP requests

Dear diary, on Wed, Oct 19, 2005 at 08:02:14AM CEST, I got a letter where Junio C Hamano <junkio@cox.net> told me that...

Show 25 quoted lines
> Nick Hengeveld <nickh@reactrix.com> writes:
> 
> > Our QA department today checked what would happen if the network connection
> > went away completely in the middle of an HTTP transfer.  It looks as though
> > the answer is that git-http-fetch sits there forever waiting for CURL to
> > return something.
> 
> Ouch.
> 
> > I'm thinking of taking advantage of CURL's capability of aborting a request
> > if the transfer rate drops below a threshold for a specified length of time
> > using a new pair of environment variables and/or config file settings:
> >
> > GIT_HTTP_LOW_SPEED_LIMIT/http.lowspeedlimit
> > GIT_HTTP_LOW_SPEED_TIME/http.lowspeedtime
> >
> > Does this make sense, and if so should there be defaults if nothing is
> > specified?
> 
> I suspect these would be quite different between DSL and
> localnet, so I doubt if there is a reasonable default value to
> quick give-up.
> 
> On the other hand, having _no_ activity for say 30 seconds would
> indicate a dead link on either modem or localnet.

I agree that we should definitely use timeout instead of some low speed limit - can't curl do that?

But 30 seconds is too little (on bad links, I've seen TCP connections stalled for much longer), I would use at least 120 seconds. Or perhaps 30 seconds, but retry three times or so.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Nick Hengeveld· Oct 19, 2005, 15:34 UTC · re: Petr Baudis · lore

Re: [RFC] Timeouts on HTTP requests

On Wed, Oct 19, 2005 at 01:09:10PM +0200, Petr Baudis wrote:
Show 5 quoted lines
> > On the other hand, having _no_ activity for say 30 seconds would
> > indicate a dead link on either modem or localnet.
> 
> I agree that we should definitely use timeout instead of some low speed
> limit - can't curl do that?

Curl lets you set timeouts for the connect and the overall request, but neither of these are related to activity. In our case, some of the files we transfer can be quite large and need to transfer over a 56K modem line. As long as data is moving, we're happy; but we need the transfer to abort if the connection drops and data stops moving completely.

> But 30 seconds is too little (on bad links, I've seen TCP connections
> stalled for much longer), I would use at least 120 seconds. Or perhaps
> 30 seconds, but retry three times or so.

I've been testing with low speed limit of 1 and low speed time of 300 - if data transfer stays below 1 byte/sec for 5 minutes, curl aborts the transfer.

I like Junio's suggestion regarding defaults though, don't compile them in but let default config templates take care of it.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

← back to recent threads