{"thread":{"id":"31692","subject":"git smart-http do not authent to allow git ls-remote to be called anonymously","startedAt":"2012-09-30T14:35:35Z","lastAt":"2012-10-31T12:01:12Z","messageCount":7,"participants":["乙酸鋰","Shawn Pearce","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"200213","messageId":"CAHtLG6Q+XO=LhnKw4hhwtOe2ROeDN1Kg=JN5GTQqdvYjk-Sv4g@mail.gmail.com","threadId":"31692","inReplyTo":null,"subject":"git smart-http do not authent to allow git ls-remote to be called anonymously","fromName":"乙酸鋰","fromEmail":"ch3cooli@gmail.com","sentAt":"2012-09-30T14:35:35Z","receivedAt":"2012-09-30T14:35:35Z","isPatch":false,"sender":{"key":"ch3cooli@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2804967?v=4"},"body":"Hi,\n\nI use smart-http on Apache.\nIf nothing to be pushed / pulled, I do not want password to be\nsupplied. And allow git ls-remote to run without password\n\n*.git/info/refs?service=git-upload-pack\n*.git/info/refs?service=git-receive-pack\n\nI only need authentication on\n\n*.git/git-upload-pack\n*.git/git-receive-pack\n\n/etc/apache/httpd.conf\n\n<LocationMatch \"^/git/.*/git-(upload|receive)-pack$\">\n    AuthType Basic\n    AuthName \"staff only\"\n    AuthUserFile /etc/apache/apache.pwd\n    Require valid-user\n</LocationMatch>\n\nHowever this does not work. It does not ask for password at all.\n\nI use Ubuntu 10.04, Apache 2.2.14, Git 1.7.11.3.\n\nDirectory structure: any depth (more than 1 subdir) of path from /git\nto .git folders\n\nCould you advise how to configure this? Is this a bug?\n\nRegards,\nch3cooli\n"},{"id":"200227","messageId":"CAJo=hJuBHCMBv5FTh6dSkvGR=XCtKn2BRgWgTJwt6kDs49-iEg@mail.gmail.com","threadId":"31692","inReplyTo":"CAHtLG6Q+XO=LhnKw4hhwtOe2ROeDN1Kg=JN5GTQqdvYjk-Sv4g@mail.gmail.com","subject":"Re: git smart-http do not authent to allow git ls-remote to be called anonymously","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2012-10-01T00:53:40Z","receivedAt":"2012-10-01T00:53:40Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sun, Sep 30, 2012 at 7:35 AM, 乙酸鋰 <ch3cooli@gmail.com> wrote:\n> I use smart-http on Apache.\n> If nothing to be pushed / pulled, I do not want password to be\n> supplied. And allow git ls-remote to run without password\n>\n> *.git/info/refs?service=git-upload-pack\n> *.git/info/refs?service=git-receive-pack\n>\n> I only need authentication on\n>\n> *.git/git-upload-pack\n> *.git/git-receive-pack\n>\n> /etc/apache/httpd.conf\n>\n> <LocationMatch \"^/git/.*/git-(upload|receive)-pack$\">\n>     AuthType Basic\n>     AuthName \"staff only\"\n>     AuthUserFile /etc/apache/apache.pwd\n>     Require valid-user\n> </LocationMatch>\n>\n> However this does not work. It does not ask for password at all.\n\nThis sounds like a bug in your Apache configuration. I would verify it\nprompts for a password as expected before worrying about the Git\nclient:\n\n  curl -v http://localhost/git/blah/git-upload-pack\n\nshould fail with a 401 requesting access to \"staff only\". Once this is\nworking, git will present authorization as necessary during the\n/git-upload-pack|git-receive-pack calls.\n"},{"id":"200229","messageId":"20121001010935.GA18987@sigill.intra.peff.net","threadId":"31692","inReplyTo":"CAHtLG6Q+XO=LhnKw4hhwtOe2ROeDN1Kg=JN5GTQqdvYjk-Sv4g@mail.gmail.com","subject":"Re: git smart-http do not authent to allow git ls-remote to be called anonymously","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-10-01T01:09:35Z","receivedAt":"2012-10-01T01:09:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Sep 30, 2012 at 10:35:35PM +0800, 乙酸鋰 wrote:\n\n> I use smart-http on Apache.\n> If nothing to be pushed / pulled, I do not want password to be\n> supplied. And allow git ls-remote to run without password\n> \n> *.git/info/refs?service=git-upload-pack\n> *.git/info/refs?service=git-receive-pack\n> \n> I only need authentication on\n> \n> *.git/git-upload-pack\n> *.git/git-receive-pack\n> \n> /etc/apache/httpd.conf\n> \n> <LocationMatch \"^/git/.*/git-(upload|receive)-pack$\">\n>     AuthType Basic\n>     AuthName \"staff only\"\n>     AuthUserFile /etc/apache/apache.pwd\n>     Require valid-user\n> </LocationMatch>\n> \n> However this does not work. It does not ask for password at all.\n\nWhat is \"it\" in the final sentence? Does the server not tell the git\nclient that authorization is required, and actually serve the request?\nIf so, then that is a bug in your apache config.\n\nOr is it that the server tells git that it needs authorization, but git\ndoes not prompt, and instead just fails with \"Authentication failed\". In\nthat case, the issue is that you need a newer git client. Traditionally\nthe client expected to handle authentication during the initial\n\"info/refs\" request. I added support for handling authentication during\nlater requests in commit b81401c, which is in git v1.7.11.7 and\nv1.7.12.1.\n\nYou should reconsider whether this is what you really want, though. With\nthe configuration you showed, anyone will be able to get a list of all\nrefs and their sha1s. So they would know all your branch names, and they\ncould even potentially find out what's in your branches by making\noffline guesses and comparing them to your branch sha1s (the feasibility\nof this would depend on exactly what you're storing).\n\n-Peff\n"},{"id":"201152","messageId":"20121014063022.GA14812@sigill.intra.peff.net","threadId":"31692","inReplyTo":"CAHtLG6T=hFsSy=ScRK2cYBoUcmAG_tsg12FiFMTvzpHGmPTzfg@mail.gmail.com","subject":"Re: git smart-http do not authent to allow git ls-remote to be called anonymously","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-10-14T06:30:22Z","receivedAt":"2012-10-14T06:30:22Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"[re-adding git@vger; please keep discussion on-list]\n\nOn Sun, Oct 14, 2012 at 01:29:13PM +0800, 乙酸鋰 wrote:\n\n> Sorry, it does not serve the request. It returns http 401.\n> But if I add the username and password as a part of the URL, it succeeds.\n\nIn that case, then you probably need to upgrade your client version of\ngit, as I mentioned here:\n\n> >> Or is it that the server tells git that it needs authorization, but git\n> >> does not prompt, and instead just fails with \"Authentication failed\". In\n> >> that case, the issue is that you need a newer git client. Traditionally\n> >> the client expected to handle authentication during the initial\n> >> \"info/refs\" request. I added support for handling authentication during\n> >> later requests in commit b81401c, which is in git v1.7.11.7 and\n> >> v1.7.12.1.\n\n-Peff\n"},{"id":"202300","messageId":"20121031115346.GC30879@sigill.intra.peff.net","threadId":"31692","inReplyTo":"CAHtLG6SsMgwquaD_Rd0YvR9Ues-u1ktEdOXC2MAfEM9Naac=5g@mail.gmail.com","subject":"Re: git smart-http do not authent to allow git ls-remote to be called anonymously","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-10-31T11:53:46Z","receivedAt":"2012-10-31T11:53:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"[+cc git@vger; please keep discussion on the list]\n\nOn Sun, Oct 28, 2012 at 01:26:51PM +0800, 乙酸鋰 wrote:\n\n> > POST /git/Cat1/SubCat1/xsp.git/git-upload-pack HTTP/1.1\n> User-Agent: git/1.8.0\n> Host: localhost\n> Accept-Encoding: gzip\n> Content-Type: application/x-git-upload-pack-request\n> Accept: application/x-git-upload-pack-result\n> Content-Length: 190\n> \n> * The requested URL returned error: 401\n> * Closing connection #0\n> Username for 'http://localhost': user\n> Password for 'http://user@localhost':\n> fatal: The remote end hung up unexpectedly\n\nOK, I see what is going on. The code in b81401c to retry POST requests\ndoes not handle gzipped contents, and upload-pack tends to gzip what it\nsends.\n\nYour apache configuration is not really something that we ever intended\nto support, and I am a little dubious of the security tradeoff being\nmade. But it is actually pretty easy for us to support, and it\neliminates a special case from the code, so I am tempted to do so.\n\nThe following patch series (on top of the current master, as they\nrequire some cleanup that did not make it into 1.8.0) seems to fix it\nfor me.\n\n  [1/2]: remote-curl: hoist gzip buffer size to top of post_rpc\n  [2/2]: remote-curl: retry failed requests for auth even with gzip\n\n-Peff\n"},{"id":"202301","messageId":"20121031115522.GA21011@sigill.intra.peff.net","threadId":"31692","inReplyTo":"20121031115346.GC30879@sigill.intra.peff.net","subject":"[PATCH 1/2] remote-curl: hoist gzip buffer size to top of post_rpc","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-10-31T11:55:22Z","receivedAt":"2012-10-31T11:55:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"When we gzip the post data for a smart-http rpc request, we\ncompute the gzip body and its size inside the \"use_gzip\"\nconditional. We keep track of the body after the conditional\nends, but not the size. Let's remember both, which will\nenable us to retry failed gzip requests in a future patch.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nThis is a tiny change conceptually, but the name change makes the diff\nquite noisy. Thus I pulled it into a separate patch from 2/2.\n\n remote-curl.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex aefafd3..10cd47d 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -413,6 +413,7 @@ static int post_rpc(struct rpc_state *rpc)\n \tstruct curl_slist *headers = NULL;\n \tint use_gzip = rpc->gzip_request;\n \tchar *gzip_body = NULL;\n+\tsize_t gzip_size;\n \tint err, large_request = 0;\n \n \t/* Try to load the entire request, if we can fit it into the\n@@ -478,19 +479,18 @@ retry:\n \t\t * we can try to deflate it ourselves, this may save on.\n \t\t * the transfer time.\n \t\t */\n-\t\tsize_t size;\n \t\tgit_zstream stream;\n \t\tint ret;\n \n \t\tmemset(&stream, 0, sizeof(stream));\n \t\tgit_deflate_init_gzip(&stream, Z_BEST_COMPRESSION);\n-\t\tsize = git_deflate_bound(&stream, rpc->len);\n-\t\tgzip_body = xmalloc(size);\n+\t\tgzip_size = git_deflate_bound(&stream, rpc->len);\n+\t\tgzip_body = xmalloc(gzip_size);\n \n \t\tstream.next_in = (unsigned char *)rpc->buf;\n \t\tstream.avail_in = rpc->len;\n \t\tstream.next_out = (unsigned char *)gzip_body;\n-\t\tstream.avail_out = size;\n+\t\tstream.avail_out = gzip_size;\n \n \t\tret = git_deflate(&stream, Z_FINISH);\n \t\tif (ret != Z_STREAM_END)\n@@ -500,16 +500,16 @@ retry:\n \t\tif (ret != Z_OK)\n \t\t\tdie(\"cannot deflate request; zlib end error %d\", ret);\n \n-\t\tsize = stream.total_out;\n+\t\tgzip_size = stream.total_out;\n \n \t\theaders = curl_slist_append(headers, \"Content-Encoding: gzip\");\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);\n-\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, size);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, gzip_size);\n \n \t\tif (options.verbosity > 1) {\n \t\t\tfprintf(stderr, \"POST %s (gzip %lu to %lu bytes)\\n\",\n \t\t\t\trpc->service_name,\n-\t\t\t\t(unsigned long)rpc->len, (unsigned long)size);\n+\t\t\t\t(unsigned long)rpc->len, (unsigned long)gzip_size);\n \t\t\tfflush(stderr);\n \t\t}\n \t} else {\n-- \n1.8.0.207.gdf2154c\n"},{"id":"202302","messageId":"20121031120112.GB21011@sigill.intra.peff.net","threadId":"31692","inReplyTo":"20121031115346.GC30879@sigill.intra.peff.net","subject":"[PATCH 2/2] remote-curl: retry failed requests for auth even with gzip","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-10-31T12:01:12Z","receivedAt":"2012-10-31T12:01:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Commit b81401c taught the post_rpc function to retry the\nhttp request after prompting for credentials. However, it\ndid not handle two cases:\n\n  1. If we have a large request, we do not retry. That's OK,\n     since we would have sent a probe (with retry) already.\n\n  2. If we are gzipping the request, we do not retry. That\n     was considered OK, because the intended use was for\n     push (e.g., listing refs is OK, but actually pushing\n     objects is not), and we never gzip on push.\n\nThis patch teaches post_rpc to retry even a gzipped request.\nThis has two advantages:\n\n  1. It is possible to configure a \"half-auth\" state for\n     fetching, where the set of refs and their sha1s are\n     advertised, but one cannot actually fetch objects.\n\n     This is not a recommended configuration, as it leaks\n     some information about what is in the repository (e.g.,\n     an attacker can try brute-forcing possible content in\n     your repository and checking whether it matches your\n     branch sha1). However, it can be slightly more\n     convenient, since a no-op fetch will not require a\n     password at all.\n\n  2. It future-proofs us should we decide to ever gzip more\n     requests.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nI doubt we would ever want to gzip push requests. The bulk of the data\nis objects, which are already compressed. In theory we could gzip the\nnon-pack parts, but that would first mean splitting them into a separate\nrequest. The tiny bit of savings are almost certainly not worth the\ncomplexity.\n\nBut the future-proofing might still be valuable if we ever modify the\nprotocol to have a new phase.\n\n remote-curl.c           | 11 ++++++++++-\n t/lib-httpd/apache.conf |  7 +++++++\n t/t5551-http-fetch.sh   | 15 +++++++++++++++\n 3 files changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 10cd47d..fac2bef 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -474,6 +474,15 @@ retry:\n \t\t\tfflush(stderr);\n \t\t}\n \n+\t} else if (gzip_body) {\n+\t\t/*\n+\t\t * If we are looping to retry authentication, then the previous\n+\t\t * run will have set up the headers and gzip buffer already,\n+\t\t * and we just need to send it.\n+\t\t */\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, gzip_size);\n+\n \t} else if (use_gzip && 1024 < rpc->len) {\n \t\t/* The client backend isn't giving us compressed data so\n \t\t * we can try to deflate it ourselves, this may save on.\n@@ -530,7 +539,7 @@ retry:\n \tcurl_easy_setopt(slot->curl, CURLOPT_FILE, rpc);\n \n \terr = run_slot(slot);\n-\tif (err == HTTP_REAUTH && !large_request && !use_gzip)\n+\tif (err == HTTP_REAUTH && !large_request)\n \t\tgoto retry;\n \tif (err != HTTP_OK)\n \t\terr = -1;\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex ec8618d..15a3c71 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -96,6 +96,13 @@ SSLEngine On\n \tRequire valid-user\n </LocationMatch>\n \n+<LocationMatch \"^/auth-fetch/.*/git-upload-pack$\">\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+</LocationMatch>\n+\n <IfDefine DAV>\n \tLoadModule dav_module modules/mod_dav.so\n \tLoadModule dav_fs_module modules/mod_dav_fs.so\ndiff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh\nindex 7380f2a..5f174da 100755\n--- a/t/t5551-http-fetch.sh\n+++ b/t/t5551-http-fetch.sh\n@@ -129,6 +129,21 @@ test_expect_success 'clone from auth-only-for-push repository' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone from auth-only-for-objects repository' '\n+\techo two >expect &&\n+\tset_askpass user@host &&\n+\tgit clone --bare \"$HTTPD_URL/auth-fetch/smart/repo.git\" half-auth &&\n+\texpect_askpass both user@host &&\n+\tgit --git-dir=half-auth log -1 --format=%s >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'no-op half-auth fetch does not require a password' '\n+\tset_askpass wrong &&\n+\tgit --git-dir=half-auth fetch &&\n+\texpect_askpass none\n+'\n+\n test -n \"$GIT_TEST_LONG\" && test_set_prereq EXPENSIVE\n \n test_expect_success EXPENSIVE 'create 50,000 tags in the repo' '\n-- \n1.8.0.207.gdf2154c\n"}]}