{"thread":{"id":"41992","subject":"Default authentication over https?","startedAt":"2016-04-11T16:04:02Z","lastAt":"2016-04-15T23:02:17Z","messageCount":7,"participants":["Isaac Levy","Jeff King","Daniel Stenberg","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"283141","messageId":"CAPf1peAW11hZpN6_ztA62tcu6mgCfV3VwwjjtXT5yySUPD9Qpw@mail.gmail.com","threadId":"41992","inReplyTo":null,"subject":"Default authentication over https?","fromName":"Isaac Levy","fromEmail":"isaac.r.levy@gmail.com","sentAt":"2016-04-11T16:04:02Z","receivedAt":"2016-04-11T16:04:02Z","isPatch":false,"sender":{"key":"isaac.r.levy@gmail.com","avatar":null},"body":"Hi all,\n\nI use a git server which requires authentication over https. Git seems\ndetermined to always try an unauthenticated request first, slowing\ndown operations by a couple seconds.\n\nIs there a way to configure git to default to authenticated requests?  Thanks!\n\nRegards,\nIsaac Levy\n"},{"id":"283427","messageId":"20160413223613.GB10011@sigill.intra.peff.net","threadId":"41992","inReplyTo":"CAPf1peAW11hZpN6_ztA62tcu6mgCfV3VwwjjtXT5yySUPD9Qpw@mail.gmail.com","subject":"Re: Default authentication over https?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-04-13T22:36:13Z","receivedAt":"2016-04-13T22:36:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 11, 2016 at 12:04:02PM -0400, Isaac Levy wrote:\n\n> I use a git server which requires authentication over https. Git seems\n> determined to always try an unauthenticated request first, slowing\n> down operations by a couple seconds.\n> \n> Is there a way to configure git to default to authenticated requests?  Thanks!\n\nNo, there isn't. Very old versions of git would ask for the password if\nyou provided a username, but since v1.7.8 we only do so in response to\nan HTTP 401. The code is still there to do the \"proactive\" asking, but\nit's not wired up to any particular config option.\n\nHowever, I don't think even that would give you what you want. Because I\nthink that even if we provide a credential, curl will make an initial\nrequest (presumably to find out which auth type it should use, but that\nis just a guess). I don't know if there is a way to convince curl to\nstick the credential in the first request (if my guess is right, then\nperhaps by setting the auth type explicitly, or even by sticking in our\nown Authorization header).\n\nSo I think the answer for now is \"no\", but it might be possible (and not\neven that hard) to do with a patch.\n\n-Peff\n"},{"id":"283442","messageId":"alpine.DEB.2.20.1604141142190.23813@tvnag.unkk.fr","threadId":"41992","inReplyTo":"20160413223613.GB10011@sigill.intra.peff.net","subject":"Re: Default authentication over https?","fromName":"Daniel Stenberg","fromEmail":"daniel@haxx.se","sentAt":"2016-04-14T09:46:09Z","receivedAt":"2016-04-14T09:46:09Z","isPatch":false,"sender":{"key":"daniel@haxx.se","avatar":"https://gravatar.com/avatar/69fdca87edd17cee21ca2e79fc2ff671d644603c3dc27167430f3cd3dbab7ba8?d=mp&s=160"},"body":"On Wed, 13 Apr 2016, Jeff King wrote:\n\n> However, I don't think even that would give you what you want. Because I\n> think that even if we provide a credential, curl will make an initial\n> request (presumably to find out which auth type it should use, but that\n> is just a guess). I don't know if there is a way to convince curl to\n> stick the credential in the first request\n\ncurl supports this. but then you must do exactly that: tell libcurl to use \nthat single auth method only. It will of course make it fail if you select the \nwrong method etc.\n\nThe unauthenticated first request is both to probe for which methods the \nserver wants, but also works for the case when users provide credentials \nwithout the server actually ending up asking for them...\n\n-- \n\n  / daniel.haxx.se\n"},{"id":"283490","messageId":"CAPf1peDf_YbDsaz1ykvzKLkdUWtSqrHT7yrgfUgjE4R1eS4r6g@mail.gmail.com","threadId":"41992","inReplyTo":"alpine.DEB.2.20.1604141142190.23813@tvnag.unkk.fr","subject":"Re: Default authentication over https?","fromName":"Isaac Levy","fromEmail":"isaac.r.levy@gmail.com","sentAt":"2016-04-14T21:32:16Z","receivedAt":"2016-04-14T21:32:16Z","isPatch":false,"sender":{"key":"isaac.r.levy@gmail.com","avatar":null},"body":"After the authenticated request, curl says it's keeping the connection\nopen, but the next fetch seems to do two handshakes again.  The\nunauthenticated request closes the connection, so the 2nd handshake is\nforced, but I'm not sure why subsequent git fetches still do\nhandshakes.  I did a bit of sleuthing w/ source, GIT_CURL_VERBOSE and\nwireshark.\n\nWe're using GitHub enterprise -- it'd just be nice if there was a\nbetter way to configure for super fast fetches.  ssh with cached\nconnections does avoid the SSL this overhead -- though as I recall git\nprotocols over ssh are less performant.\n\nFinally I also checked out the persistent-https contrib section as a\nworkaround but couldn't get it to work. Is that project dead?\n\nThank you for your replies and support.\n\nRegards,\nIsaac\n\nOn Thu, Apr 14, 2016 at 5:46 AM, Daniel Stenberg <daniel@haxx.se> wrote:\n> On Wed, 13 Apr 2016, Jeff King wrote:\n>\n>> However, I don't think even that would give you what you want. Because I\n>> think that even if we provide a credential, curl will make an initial\n>> request (presumably to find out which auth type it should use, but that\n>> is just a guess). I don't know if there is a way to convince curl to\n>> stick the credential in the first request\n>\n>\n> curl supports this. but then you must do exactly that: tell libcurl to use\n> that single auth method only. It will of course make it fail if you select\n> the wrong method etc.\n>\n> The unauthenticated first request is both to probe for which methods the\n> server wants, but also works for the case when users provide credentials\n> without the server actually ending up asking for them...\n>\n> --\n>\n>  / daniel.haxx.se\n"},{"id":"283606","messageId":"20160415222120.GA24777@sigill.intra.peff.net","threadId":"41992","inReplyTo":"CAPf1peDf_YbDsaz1ykvzKLkdUWtSqrHT7yrgfUgjE4R1eS4r6g@mail.gmail.com","subject":"Re: Default authentication over https?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-04-15T22:21:20Z","receivedAt":"2016-04-15T22:21:20Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 14, 2016 at 05:32:16PM -0400, Isaac Levy wrote:\n\n> After the authenticated request, curl says it's keeping the connection\n> open, but the next fetch seems to do two handshakes again.  The\n> unauthenticated request closes the connection, so the 2nd handshake is\n> forced, but I'm not sure why subsequent git fetches still do\n> handshakes.  I did a bit of sleuthing w/ source, GIT_CURL_VERBOSE and\n> wireshark.\n> \n> We're using GitHub enterprise -- it'd just be nice if there was a\n> better way to configure for super fast fetches.  ssh with cached\n> connections does avoid the SSL this overhead -- though as I recall git\n> protocols over ssh are less performant.\n\nIt's the opposite; generally git is more efficient over ssh, because we\nhave a true full-duplex connection, and we can do everything over a\nsingle session (though setup for the ssh session may or may not be\nfaster than SSL).\n\nHere's what I observed just for the initial contact. If I instrument git\nlike this:\n\ndiff --git a/http.c b/http.c\nindex 4304b80..9bedad7 100644\n--- a/http.c\n+++ b/http.c\n@@ -1422,6 +1422,7 @@ static int http_request(const char *url,\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, \"gzip\");\n \n+\twarning(\"running one curl slot...\");\n \tret = run_one_slot(slot, &results);\n \n \tif (options && options->content_type) {\n\nand run:\n\n  GIT_CURL_VERBOSE=1 git ls-remote 2>&1 | egrep '< HTTP|> |^Auth|^warn'\n\nI get three requests:\n\n  warning: running one curl slot...\n  > GET /my/repo/info/refs?service=git-upload-pack HTTP/1.1\n  < HTTP/1.1 401 Authorization Required\n  warning: running one curl slot...\n  > GET /my/repo/info/refs?service=git-upload-pack HTTP/1.1\n  < HTTP/1.1 401 Authorization Required\n  > GET /my/repo/info/refs?service=git-upload-pack HTTP/1.1\n  Authorization: Basic ...\n  < HTTP/1.1 200 OK\n\nThe first request is us (git) asking curl to hit the URL, and curl\nreturns the 401 from the server to us. At that point we'll prompt the\nuser (or the credential helper) for the password, and then we'll give\nthat to curl and ask it to make the request again. Curl will do the\nprobe with the 401, because we haven't set an auth type, and then follow\nup (without returning the 401 to git) with the right credentials.\n\nI think we can take that down to _two_ requests pretty easily. We know\nin the very first request that the server told us something like:\n\n  < WWW-Authenticate: Basic realm=\"GitHub\"\n\nbut curl doesn't remember that. However, we should be able to pull it\nout of the old request and feed it into the new one. That would save the\nsecond request, which is just a probe.\n\nBut ideally we'd have this down to one request. I think we could do\nsomething like this:\n\ndiff --git a/http.c b/http.c\nindex 9bedad7..7e5009d 100644\n--- a/http.c\n+++ b/http.c\n@@ -870,6 +870,11 @@ struct active_request_slot *get_active_slot(void)\n #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);\n #endif\n+\n+\t/* XXX should be configurable via http.* or whatever */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);\n+\tcredential_fill(&http_auth);\n+\n \tif (http_auth.password || curl_empty_auth)\n \t\tinit_curl_http_auth(slot->curl);\n \n\nMy request then looks like:\n\n  warning: running one curl slot...\n  > GET /my/repo/info/refs?service=git-upload-pack HTTP/1.1\n  Authorization: Basic ...\n  < HTTP/1.1 200 OK\n\nwith just a single round-trip.\n\nObviously the hard-coding there is wrong, but the mental model here is\nthat the user would do something like:\n\n  git config http.https://github.com.auth basic\n\nto say \"I know I want to do Basic auth for this host\", and that would\ntrigger git to tell that to curl, and to prompt for the password\npreemptively.  The obvious downside is that it requires manual\nconfiguration, and some clue about \"Basic\" versus other auth types.\n\nI didn't trace the protocol further, but I suspect that curl ends up\ndoing that probe for _each_ request, because each time we hand it only\nthe credential without telling it that we know the server wants \"Basic\"\nauth.\n\nSo that tries to keep the number of requests down in the first place. As\nfar as reusing connections for the connections we _do_ make, I'm not\nsure. I do see curl claiming to leave the connection up:\n\n  * Connection #0 to host github.com left intact\n\nbut then we seem to make a separate one for the next request:\n\n  * Hostname github.com was found in DNS cache\n  *   Trying 192.30.252.120...\n  * Connected to github.com (192.30.252.120) port 443 (#1)\n\nI'm not sure why that is. We do get SSL reuse:\n\n  * SSL re-using session ID\n  * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256\n\nwhich is good. And then of the two requests curl makes later, the second\none has:\n\n  * Re-using existing connection! (#1) with host github.com\n\nwhich is good. So I'm not sure why we don't reuse connection #0 from the\nvery first request. Perhaps it's something in the way we are setting up\nthe curl handle.\n\n> Finally I also checked out the persistent-https contrib section as a\n> workaround but couldn't get it to work. Is that project dead?\n\nI don't know. It certainly hasn't seen a lot of activity lately, but\nGoogle folks might still be using it (I think the original impetus was\nfor fetching of many repos from the same server for the Android\nproject). I don't know that we've seen any mention of wide use on the\nlist.\n\n-Peff\n"},{"id":"283608","messageId":"20160415224334.GA32160@sigill.intra.peff.net","threadId":"41992","inReplyTo":"20160415222120.GA24777@sigill.intra.peff.net","subject":"Re: Default authentication over https?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-04-15T22:43:35Z","receivedAt":"2016-04-15T22:43:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 15, 2016 at 06:21:20PM -0400, Jeff King wrote:\n\n> I think we can take that down to _two_ requests pretty easily. We know\n> in the very first request that the server told us something like:\n> \n>   < WWW-Authenticate: Basic realm=\"GitHub\"\n> \n> but curl doesn't remember that. However, we should be able to pull it\n> out of the old request and feed it into the new one. That would save the\n> second request, which is just a probe.\n\nHmm. Looks like we already pull this out of the curl result for other\nreasons, but we never feed it back in to the next request. So if I do\nthis:\n\ndiff --git a/http.c b/http.c\nindex 9bedad7..add9bf2 100644\n--- a/http.c\n+++ b/http.c\n@@ -1132,6 +1132,8 @@ static int handle_curl_result(struct slot_results *results)\n \t\t\treturn HTTP_NOAUTH;\n \t\t} else {\n #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY\n+\t\t\tif (results->auth_avail)\n+\t\t\t\thttp_auth_methods = results->auth_avail;\n \t\t\thttp_auth_methods &= ~CURLAUTH_GSSNEGOTIATE;\n #endif\n \t\t\treturn HTTP_REAUTH;\n\nthat drops my test case down to two requests: once to find out that we\nneed auth via the 401, and then we feed curl sufficient information to\ndo the followup in a single request (the GSSNEGOTIATE thing there is a\nhack from 4dbe664, which we can ignore for now).\n\nInterestingly, curl _does_ reuse the connection this time. I'm still not\nsure why it didn't in the original case. But this means the whole thing\nis happening over a single TCP session, which is good (and I didn't have\nto change my config at all).\n\n-Peff\n"},{"id":"283714","messageId":"20160415230217.GC140502@vauxhall.crustytoothpaste.net","threadId":"41992","inReplyTo":"20160415224334.GA32160@sigill.intra.peff.net","subject":"Re: Default authentication over https?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2016-04-15T23:02:17Z","receivedAt":"2016-04-15T23:02:17Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Apr 15, 2016 at 06:43:35PM -0400, Jeff King wrote:\n> Hmm. Looks like we already pull this out of the curl result for other\n> reasons, but we never feed it back in to the next request. So if I do\n> this:\n> \n> diff --git a/http.c b/http.c\n> index 9bedad7..add9bf2 100644\n> --- a/http.c\n> +++ b/http.c\n> @@ -1132,6 +1132,8 @@ static int handle_curl_result(struct slot_results *results)\n>  \t\t\treturn HTTP_NOAUTH;\n>  \t\t} else {\n>  #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY\n> +\t\t\tif (results->auth_avail)\n> +\t\t\t\thttp_auth_methods = results->auth_avail;\n>  \t\t\thttp_auth_methods &= ~CURLAUTH_GSSNEGOTIATE;\n>  #endif\n>  \t\t\treturn HTTP_REAUTH;\n> \n> that drops my test case down to two requests: once to find out that we\n> need auth via the 401, and then we feed curl sufficient information to\n> do the followup in a single request (the GSSNEGOTIATE thing there is a\n> hack from 4dbe664, which we can ignore for now).\n> \n> Interestingly, curl _does_ reuse the connection this time. I'm still not\n> sure why it didn't in the original case. But this means the whole thing\n> is happening over a single TCP session, which is good (and I didn't have\n> to change my config at all).\n\nThis looks fine from my point of view.  GSS-Negotiate and Digest are\ngoing to require at least one round-trip because they need the token or\nnonce to authenticate.  Basic shouldn't, though.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"}]}