Re: [BUG?] git http connection reuse

From: Kyle J. McKay <mackyle@gmail.com>
Date: Feb 16, 2014, 07:16 UTC
Message-ID: <377DF3F4-1F9B-4CB0-A9BF-658E561F4349@gmail.com>
In-Reply-To: <20140216040538.GA4754@sigill.intra.peff.net>

On Feb 15, 2014, at 20:05, Jeff King wrote:

Show 7 quoted lines

> I've noticed that git does not reuse http connections when fetching,  
> and
> I'm trying to figure out why. It seems related to our use of two curl
> features:
>
>  1. If we use the curl_multi interface (even though we are doing the
>     requests sequentially), we do not get reuse.

Take a look at the curl_multi_setopt page [1]. I think these explain the curl_multi issue:

Show 26 quoted lines

> CURLMOPT_PIPELINING
>
> Pass a long set to 1 to enable or 0 to disable. Enabling pipelining  
> on a multi handle will make it attempt to perform HTTP Pipelining as  
> far as possible for transfers using this handle. This means that if  
> you add a second request that can use an already existing  
> connection, the second request will be "piped" on the same  
> connection rather than being executed in parallel. (Added in 7.16.0)
>
>
> CURLMOPT_MAX_TOTAL_CONNECTIONS
>
> Pass a long. The set number will be used as the maximum amount of  
> simultaneously open connections in total. For each new session,  
> libcurl will open a new connection up to the limit set by  
> CURLMOPT_MAX_TOTAL_CONNECTIONS. When the limit is reached, the  
> sessions will be pending until there are available connections. If  
> CURLMOPT_PIPELINING is 1, libcurl will try to pipeline if the host  
> is capable of it.
>
> The default value is 0, which means that there is no limit. However,  
> for backwards compatibility, setting it to 0 when  
> CURLMOPT_PIPELINING is 1 will not be treated as unlimited. Instead  
> it will open only 1 connection and try to pipeline on it.
>
> (Added in 7.30.0)

If pipelining is off (the default) and total connections is not 1 it sounds to me from the description above that the requests will be executed on separate connections until the maximum number of connections is in use and then there might be some reuse. That might not be what's actually going on with multi, but that's my guess and I think setting CURLMOPT_PIPELINING to to 1 and then also setting CURLMOPT_MAX_TOTAL_CONNECTIONS to a non-zero value might be what you want.

>  2. If we set CURLOPT_HTTPAUTH to CURLAUTH_ANY, we do not get reuse.

[snip]

> My curl version is 7.35.0, if that makes a difference.

And that is explained by this from the CHANGES file:

Show 16 quoted lines

> Daniel Stenberg (7 Jan 2014)
> - ConnectionExists: fix NTLM check for new connection
>
>   When the requested authentication bitmask includes NTLM, we cannot
>   re-use a connection for another username/password as we then risk
>   re-using NTLM (connection-based auth).
>
>   This has the unfortunate downside that if you include NTLM as a  
> possible
>   auth, you cannot re-use connections for other usernames/passwords  
> even
>   if NTLM doesn't end up the auth type used.
>
>   Reported-by: Paras S
>   Patched-by: Paras S
>   Bug: http://curl.haxx.se/mail/lib-2014-01/0046.html

Looks like you're just lucky as that above change first appears in 7.35.0. But it seems there are some patches for older versions so they might be affected as well [2].

Adjusting your test program to leave the first connection set to CURLAUTH_ANY, but then setting the second one to CURLAUTH_ANY with the two NTLM bits turned off (CURLAUTH_NTLM and CURLAUTH_NTLM_WB) results in the connection being reused for me. :)

With the older cURL versions I already had installed, the second connection is always reused. I didn't see the non-reuse with your sample code until I fetched 7.35.0.

--Kyle

[1] http://curl.haxx.se/libcurl/c/curl_multi_setopt.html [2] http://curl.haxx.se/docs/adv_20140129.html