{"thread":{"id":"54228","subject":"Git doesn't honor NO_PROXY environment variable while cloning","startedAt":"2020-09-11T12:16:21Z","lastAt":"2020-09-11T19:08:22Z","messageCount":6,"participants":["Ondrej Pohorelsky","Jeff King","Daniel Stenberg","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"405401","messageId":"CA+B51BGRuLfF7FpiK93Wih0XhsC7rJLGjkF2CzrEsUkBEif+jw@mail.gmail.com","threadId":"54228","inReplyTo":null,"subject":"Git doesn't honor NO_PROXY environment variable while cloning","fromName":"Ondrej Pohorelsky","fromEmail":"opohorel@redhat.com","sentAt":"2020-09-11T12:15:59Z","receivedAt":"2020-09-11T12:16:21Z","isPatch":false,"sender":{"key":"opohorel@redhat.com","avatar":"https://avatars.githubusercontent.com/u/35430604?v=4"},"body":"Hi,\n\nwe've got a bug[0] reported in Red Hat Bugzilla. It seems like Git\ndoesn't honor NO_PROXY environment variable while cloning repositories\n\nReporter has provided these steps to reproduce using Git version 2.18.4:\n\n1. Start a ubi8 container with podman:\n```\n$ podman run --rm -it registry.access.redhat.com/ubi8/ubi /bin/bash\n```\n2. Install git in the container - `# yum install -y git`\n3. Add \"bad\" _PROXY environment variables:\n```\n# export HTTP_PROXY=http://user:pass@bad.proxy\n# export HTTPS_PROXY=https://user:pass@bad.proxy\n```\n4. Add Github as an ignorable proxy domain:\n```\n# export NO_PROXY=github.com\n```\n5. Clone a github repo:\n```\n# mkdir -p /tmp/clone-test\n# cd /tmp/clone-test\n# git clone https://github.com/sclorg/nodejs-ex.git\n# echo $?\n```\n\nActual results:\n```\n# git clone https://github.com/sclorg/nodejs-ex.git\nCloning into 'nodejs-ex'...\n# echo $?\n128\n```\n\nExpected results:\n\nGit clone succeeds, bypassing the proxy.\n\n\n\nHowever I've found out that this possible issue is present even in\nnewer versions e.g. 2.28.0.\n\nThere is a workaround. You can rewrite proxy to be blank in .gitconfig\nfor specific websites.\n```\n[http \"https://github.com/\"]\n        proxy = \"\"\n```\n\nIs this an issue or expected behaviour? And if it is expected\nbehaviour, then why?\n\nBest regards,\nOndřej Pohořelský\n\n\n[0]https://bugzilla.redhat.com/show_bug.cgi?id=1875639\n\n"},{"id":"405409","messageId":"20200911135928.GA1986935@coredump.intra.peff.net","threadId":"54228","inReplyTo":"CA+B51BGRuLfF7FpiK93Wih0XhsC7rJLGjkF2CzrEsUkBEif+jw@mail.gmail.com","subject":"Re: Git doesn't honor NO_PROXY environment variable while cloning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-09-11T13:59:28Z","receivedAt":"2020-09-11T15:01:47Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"[tl;dr This looks to me like a problem in libcurl. I've included my\n whole line of reasoning below. Folks interested in libcurl might want\n to jump straight to the backtrace].\n\nOn Fri, Sep 11, 2020 at 02:15:59PM +0200, Ondrej Pohorelsky wrote:\n\n> we've got a bug[0] reported in Red Hat Bugzilla. It seems like Git\n> doesn't honor NO_PROXY environment variable while cloning repositories\n\nIt's not so much \"doesn't honor NO_PROXY\" as \"NO_PROXY seems to trigger\nanother bug\". Here's a slightly simplified recipe:\n\n  [set up our bogus proxy]\n  $ export HTTPS_PROXY=https://user:pass@bad.proxy\n\n  [so far so good; we expect this to complain about the proxy]\n  $ git clone https://github.com/git/git\n  Cloning into 'git'...\n  fatal: unable to access 'https://github.com/git/git/': Could not resolve proxy: bad.proxy\n\n  [now let's set NO_PROXY]\n  $ export NO_PROXY=github.com\n  $ git clone https://github.com/git/git\n  Cloning into 'git'...\n  $ echo $?\n  128\n\nSo we are looking at the variable, but then we exit with failure (and\nwith nothing to stderr!). That was with v2.28 above. But let's try it\nwith the current tip of master:\n\n  $ git clone https://github.com/git/git\n  Cloning into 'git'...\n  error: git-remote-https died of signal 11\n\nAh, that's interesting. The switch in behavior is due to 675df192c5\n(transport-helper: do not run git-remote-ext etc. in dashed form,\n2020-08-26). Before that patch the transport code ran git-remote-https\ndirectly. If it returned a non-zero exit code, we just assumed it had\nprinted its own error to stdout and exited. But after that patch we run\n\"git remote-https\", and the git.c wrapper has code to complain about\nsegfaults.\n\nSo that's a little digression from our real bug. But it points us in the\nright direction: remote-https is segfaulting. So here's a more direct\nreproduction:\n\n  export HTTPS_PROXY=https://user:pass@bad.proxy\n  export NO_PROXY=github.com\n  echo list | ./git-remote-https https://github.com/git/git\n\nwhich results in a segfault. And we can run that process under gdb to\nget a backtrace:\n\n  echo list >input\n  gdb -ex 'set args https://github.com/git/git <input' \\\n      -ex 'run' \\\n      -ex 'bt' \\\n      ./git-remote-https\n\nwhich yields (after install debug symbols for system curl, etc):\n\n  #0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65\n  #1  0x00007ffff7da6934 in __GI___inet_pton (af=af@entry=2, src=src@entry=0x0, dst=dst@entry=0x7fffffffd600)\n      at inet_pton.c:69\n\nWell, the issue is clear enough. Somebody is passing NULL to\ninet_pton(). But who?\n\n  #2  0x00007ffff7f88a08 in ossl_connect_step1 (conn=conn@entry=0x5555558275b0, sockindex=sockindex@entry=0)\n      at vtls/openssl.c:3136\n  #3  0x00007ffff7f8a62f in ossl_connect_common (conn=0x5555558275b0, sockindex=0, nonblocking=true, \n      done=0x555555827bea) at vtls/openssl.c:3985\n  #4  0x00007ffff7f8b43b in Curl_ssl_connect_nonblocking (conn=conn@entry=0x5555558275b0, sockindex=sockindex@entry=0, \n      done=done@entry=0x555555827bea) at vtls/vtls.c:331\n  #5  0x00007ffff7f552db in https_proxy_connect (sockindex=0, conn=0x5555558275b0) at http_proxy.c:57\n  #6  Curl_proxy_connect (conn=0x5555558275b0, sockindex=0) at http_proxy.c:77\n  #7  0x00007ffff7f4c148 in Curl_http_connect (conn=0x5555558275b0, done=0x7fffffffdc58) at http.c:1397\n  #8  0x00007ffff7f5ffa6 in multi_runsingle (multi=0x555555811240, now=..., data=0x555555821c90) at multi.c:1802\n  #9  0x00007ffff7f610e1 in curl_multi_perform (multi=0x555555811240, running_handles=0x7fffffffddc8) at multi.c:2420\n\nUh oh, it's deep within libcurl. It may be that we're passing it bogus\ndata, but I don't think so. The rest of our backtrace shows what Git is\ndoing:\n\n  #10 0x0000555555566f41 in step_active_slots () at http.c:1458\n  #11 0x0000555555566f9c in run_active_slot (slot=0x555555814370) at http.c:1479\n  #12 0x000055555556762e in run_one_slot (slot=0x555555814370, results=0x7fffffffe050) at http.c:1680\n  #13 0x000055555556813b in http_request (\n      url=0x555555812ae0 \"https://github.com/git/git/info/refs?service=git-upload-pack\", result=0x7fffffffe210, \n      target=0, options=0x7fffffffe160) at http.c:1964\n  #14 0x0000555555568371 in http_request_reauth (\n      url=0x555555812ae0 \"https://github.com/git/git/info/refs?service=git-upload-pack\", result=0x7fffffffe210, \n      target=0, options=0x7fffffffe160) at http.c:2040\n  #15 0x0000555555568514 in http_get_strbuf (\n      url=0x555555812ae0 \"https://github.com/git/git/info/refs?service=git-upload-pack\", result=0x7fffffffe210, \n      options=0x7fffffffe160) at http.c:2088\n  #16 0x00005555555605fb in discover_refs (service=0x5555556f73fb \"git-upload-pack\", for_push=0) at remote-curl.c:484\n  [...]\n\nSo that's the very first http request we make of libcurl. And I don't\nthink we'll have regained control via any callbacks, etc. It's possible\nwe've screwed up passing the NO_PROXY variable somehow, but we don't\nreally do anything interesting with it:\n\n  $ git grep -A2 NO_PROXY\n  http.c:         var_override(&curl_no_proxy, getenv(\"NO_PROXY\"));\n  http.c-         var_override(&curl_no_proxy, getenv(\"no_proxy\"));\n  http.c-         curl_easy_setopt(result, CURLOPT_NOPROXY, curl_no_proxy);\n\nAnd if I comment out that curl_easy_setopt() line, we still hit the\nproblem (because modern libcurl will read NO_PROXY itself).\n\nSo it seems at first glance that this is a curl bug. It's triggering via\nssl routines, so let's try plain http:\n\n  $ export http_proxy=http://user:pass@proxy.example.com\n  $ export NO_PROXY=example.com\n  $ git clone https://example.com\n  Cloning into 'example.com'...\n  fatal: repository 'http://example.com/' not found\n\nNo segfault, and it seems to have actually contacted the server (and you\ncan verify that with GIT_CURL_VERBOSE=1 in the environment).\n\nWe can't do quite the same test with github.com because it will redirect\nhttp to https. But its result is interesting, too:\n\n  $ export NO_PROXY=github.com\n  $ git clone http://github.com/git/git foo\n  Cloning into 'foo'...\n  warning: redirecting to https://github.com/git/git/\n  remote: Enumerating objects: 292855, done.\n  [...etc...]\n\nSo it _does_ work after the redirect. It only fails if the initial\nrequest is https.\n\nSo I dunno. This seems like a libcurl bug, but it's possible we're feeding\nit data wrong somehow.\n\n-Peff\n"},{"id":"405412","messageId":"alpine.DEB.2.20.2009111729530.6227@tvnag.unkk.fr","threadId":"54228","inReplyTo":"20200911135928.GA1986935@coredump.intra.peff.net","subject":"Re: Git doesn't honor NO_PROXY environment variable while cloning","fromName":"Daniel Stenberg","fromEmail":"daniel@haxx.se","sentAt":"2020-09-11T15:31:16Z","receivedAt":"2020-09-11T16:12:07Z","isPatch":false,"sender":{"key":"daniel@haxx.se","avatar":"https://gravatar.com/avatar/69fdca87edd17cee21ca2e79fc2ff671d644603c3dc27167430f3cd3dbab7ba8?d=mp&s=160"},"body":"On Fri, 11 Sep 2020, Jeff King wrote:\n\n> So I dunno. This seems like a libcurl bug, but it's possible we're feeding \n> it data wrong somehow.\n\nI didn't check very closely, but it certainly sounds like this one:\n\n   https://github.com/curl/curl/pull/5902\n\nFixed in curl master in this commit (8 days ago):\n\n   https://github.com/curl/curl/commit/3eff1c5092e542819ac7e6454a70c94b36ab\n\n-- \n\n  / daniel.haxx.se\n"},{"id":"405415","messageId":"20200911164656.GA2641000@coredump.intra.peff.net","threadId":"54228","inReplyTo":"alpine.DEB.2.20.2009111729530.6227@tvnag.unkk.fr","subject":"Re: Git doesn't honor NO_PROXY environment variable while cloning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-09-11T16:46:56Z","receivedAt":"2020-09-11T16:48:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 11, 2020 at 05:31:16PM +0200, Daniel Stenberg wrote:\n\n> On Fri, 11 Sep 2020, Jeff King wrote:\n> \n> > So I dunno. This seems like a libcurl bug, but it's possible we're\n> > feeding it data wrong somehow.\n> \n> I didn't check very closely, but it certainly sounds like this one:\n> \n>   https://github.com/curl/curl/pull/5902\n\nYou know, I almost cc'd you, but I didn't want to bug you until I was\nmore sure it was a curl issue. But here you are anyway. :)\n\nThat indeed looks a lot like the same issue. And building the tip of\nlibcurl's master and linking git against it makes the problem go away.\nSo that is almost certainly it.\n\nThanks for the quick response!\n\n-Peff\n"},{"id":"405417","messageId":"20200911140119.GA2072036@coredump.intra.peff.net","threadId":"54228","inReplyTo":"20200911135928.GA1986935@coredump.intra.peff.net","subject":"Re: Git doesn't honor NO_PROXY environment variable while cloning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-09-11T14:01:19Z","receivedAt":"2020-09-11T17:08:21Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 11, 2020 at 09:59:28AM -0400, Jeff King wrote:\n\n> So I dunno. This seems like a libcurl bug, but it's possible we're feeding\n> it data wrong somehow.\n\nI forgot to mention: I'm using libcurl 7.72.0 (the current in debian\nunstable). But it also reproduces on my debian stable machine with\nlibcurl 7.64.0.\n\n-Peff\n"},{"id":"405441","messageId":"xmqqo8mcnnc0.fsf@gitster.c.googlers.com","threadId":"54228","inReplyTo":"20200911164656.GA2641000@coredump.intra.peff.net","subject":"Re: Git doesn't honor NO_PROXY environment variable while cloning","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-11T19:08:15Z","receivedAt":"2020-09-11T19:08:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Fri, Sep 11, 2020 at 05:31:16PM +0200, Daniel Stenberg wrote:\n>\n>> On Fri, 11 Sep 2020, Jeff King wrote:\n>> \n>> > So I dunno. This seems like a libcurl bug, but it's possible we're\n>> > feeding it data wrong somehow.\n>> \n>> I didn't check very closely, but it certainly sounds like this one:\n>> \n>>   https://github.com/curl/curl/pull/5902\n>\n> You know, I almost cc'd you, but I didn't want to bug you until I was\n> more sure it was a curl issue. But here you are anyway. :)\n>\n> That indeed looks a lot like the same issue. And building the tip of\n> libcurl's master and linking git against it makes the problem go away.\n> So that is almost certainly it.\n>\n> Thanks for the quick response!\n\nThanks, both.\n"}]}