Git doesn't honor NO_PROXY environment variable while cloning

Hi,

we've got a bug[0] reported in Red Hat Bugzilla. It seems like Git doesn't honor NO_PROXY environment variable while cloning repositories

Reporter has provided these steps to reproduce using Git version 2.18.4:

1. Start a ubi8 container with podman:
```
$ podman run --rm -it registry.access.redhat.com/ubi8/ubi /bin/bash
```
2. Install git in the container - `# yum install -y git`
3. Add "bad" _PROXY environment variables:
```
# export HTTP_PROXY=http://user:pass@bad.proxy
# export HTTPS_PROXY=https://user:pass@bad.proxy
```
4. Add Github as an ignorable proxy domain:
```
# export NO_PROXY=github.com
```
5. Clone a github repo:
```
# mkdir -p /tmp/clone-test
# cd /tmp/clone-test
# git clone https://github.com/sclorg/nodejs-ex.git
# echo $?
```

Actual results: ``` # git clone https://github.com/sclorg/nodejs-ex.git Cloning into 'nodejs-ex'... # echo $? 128 ```

Expected results:

Git clone succeeds, bypassing the proxy.

However I've found out that this possible issue is present even in newer versions e.g. 2.28.0.

There is a workaround. You can rewrite proxy to be blank in .gitconfig
for specific websites.
```
[http "https://github.com/"]
        proxy = ""
```

Is this an issue or expected behaviour? And if it is expected behaviour, then why?

Best regards, Ondřej Pohořelský

[0]https://bugzilla.redhat.com/show_bug.cgi?id=1875639

[tl;dr This looks to me like a problem in libcurl. I've included my
 whole line of reasoning below. Folks interested in libcurl might want
 to jump straight to the backtrace].

On Fri, Sep 11, 2020 at 02:15:59PM +0200, Ondrej Pohorelsky wrote:

It's not so much "doesn't honor NO_PROXY" as "NO_PROXY seems to trigger another bug". Here's a slightly simplified recipe:

  [set up our bogus proxy]
  $ export HTTPS_PROXY=https://user:pass@bad.proxy

  [so far so good; we expect this to complain about the proxy]
  $ git clone https://github.com/git/git
  Cloning into 'git'...
  fatal: unable to access 'https://github.com/git/git/': Could not resolve proxy: bad.proxy

  [now let's set NO_PROXY]
  $ export NO_PROXY=github.com
  $ git clone https://github.com/git/git
  Cloning into 'git'...
  $ echo $?
  128

So we are looking at the variable, but then we exit with failure (and with nothing to stderr!). That was with v2.28 above. But let's try it with the current tip of master:

  $ git clone https://github.com/git/git
  Cloning into 'git'...
  error: git-remote-https died of signal 11

Ah, that's interesting. The switch in behavior is due to 675df192c5 (transport-helper: do not run git-remote-ext etc. in dashed form, 2020-08-26). Before that patch the transport code ran git-remote-https directly. If it returned a non-zero exit code, we just assumed it had printed its own error to stdout and exited. But after that patch we run "git remote-https", and the git.c wrapper has code to complain about segfaults.

So that's a little digression from our real bug. But it points us in the right direction: remote-https is segfaulting. So here's a more direct reproduction:

  export HTTPS_PROXY=https://user:pass@bad.proxy
  export NO_PROXY=github.com
  echo list | ./git-remote-https https://github.com/git/git

which results in a segfault. And we can run that process under gdb to get a backtrace:

  echo list >input
  gdb -ex 'set args https://github.com/git/git <input' \
      -ex 'run' \
      -ex 'bt' \
      ./git-remote-https

which yields (after install debug symbols for system curl, etc):

  #0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
  #1  0x00007ffff7da6934 in __GI___inet_pton (af=af@entry=2, src=src@entry=0x0, dst=dst@entry=0x7fffffffd600)
      at inet_pton.c:69

Well, the issue is clear enough. Somebody is passing NULL to inet_pton(). But who?

  #2  0x00007ffff7f88a08 in ossl_connect_step1 (conn=conn@entry=0x5555558275b0, sockindex=sockindex@entry=0)
      at vtls/openssl.c:3136
  #3  0x00007ffff7f8a62f in ossl_connect_common (conn=0x5555558275b0, sockindex=0, nonblocking=true, 
      done=0x555555827bea) at vtls/openssl.c:3985
  #4  0x00007ffff7f8b43b in Curl_ssl_connect_nonblocking (conn=conn@entry=0x5555558275b0, sockindex=sockindex@entry=0, 
      done=done@entry=0x555555827bea) at vtls/vtls.c:331
  #5  0x00007ffff7f552db in https_proxy_connect (sockindex=0, conn=0x5555558275b0) at http_proxy.c:57
  #6  Curl_proxy_connect (conn=0x5555558275b0, sockindex=0) at http_proxy.c:77
  #7  0x00007ffff7f4c148 in Curl_http_connect (conn=0x5555558275b0, done=0x7fffffffdc58) at http.c:1397
  #8  0x00007ffff7f5ffa6 in multi_runsingle (multi=0x555555811240, now=..., data=0x555555821c90) at multi.c:1802
  #9  0x00007ffff7f610e1 in curl_multi_perform (multi=0x555555811240, running_handles=0x7fffffffddc8) at multi.c:2420

Uh oh, it's deep within libcurl. It may be that we're passing it bogus data, but I don't think so. The rest of our backtrace shows what Git is doing:

  #10 0x0000555555566f41 in step_active_slots () at http.c:1458
  #11 0x0000555555566f9c in run_active_slot (slot=0x555555814370) at http.c:1479
  #12 0x000055555556762e in run_one_slot (slot=0x555555814370, results=0x7fffffffe050) at http.c:1680
  #13 0x000055555556813b in http_request (
      url=0x555555812ae0 "https://github.com/git/git/info/refs?service=git-upload-pack", result=0x7fffffffe210, 
      target=0, options=0x7fffffffe160) at http.c:1964
  #14 0x0000555555568371 in http_request_reauth (
      url=0x555555812ae0 "https://github.com/git/git/info/refs?service=git-upload-pack", result=0x7fffffffe210, 
      target=0, options=0x7fffffffe160) at http.c:2040
  #15 0x0000555555568514 in http_get_strbuf (
      url=0x555555812ae0 "https://github.com/git/git/info/refs?service=git-upload-pack", result=0x7fffffffe210, 
      options=0x7fffffffe160) at http.c:2088
  #16 0x00005555555605fb in discover_refs (service=0x5555556f73fb "git-upload-pack", for_push=0) at remote-curl.c:484
  [...]

So that's the very first http request we make of libcurl. And I don't think we'll have regained control via any callbacks, etc. It's possible we've screwed up passing the NO_PROXY variable somehow, but we don't really do anything interesting with it:

  $ git grep -A2 NO_PROXY
  http.c:         var_override(&curl_no_proxy, getenv("NO_PROXY"));
  http.c-         var_override(&curl_no_proxy, getenv("no_proxy"));
  http.c-         curl_easy_setopt(result, CURLOPT_NOPROXY, curl_no_proxy);

And if I comment out that curl_easy_setopt() line, we still hit the problem (because modern libcurl will read NO_PROXY itself).

So it seems at first glance that this is a curl bug. It's triggering via ssl routines, so let's try plain http:

  $ export http_proxy=http://user:pass@proxy.example.com
  $ export NO_PROXY=example.com
  $ git clone https://example.com
  Cloning into 'example.com'...
  fatal: repository 'http://example.com/' not found

No segfault, and it seems to have actually contacted the server (and you can verify that with GIT_CURL_VERBOSE=1 in the environment).

We can't do quite the same test with github.com because it will redirect http to https. But its result is interesting, too:

  $ export NO_PROXY=github.com
  $ git clone http://github.com/git/git foo
  Cloning into 'foo'...
  warning: redirecting to https://github.com/git/git/
  remote: Enumerating objects: 292855, done.
  [...etc...]

So it _does_ work after the redirect. It only fails if the initial request is https.

So I dunno. This seems like a libcurl bug, but it's possible we're feeding it data wrong somehow.

-Peff

On Fri, 11 Sep 2020, Jeff King wrote:

I didn't check very closely, but it certainly sounds like this one:

   https://github.com/curl/curl/pull/5902

Fixed in curl master in this commit (8 days ago):

   https://github.com/curl/curl/commit/3eff1c5092e542819ac7e6454a70c94b36ab

On Fri, Sep 11, 2020 at 05:31:16PM +0200, Daniel Stenberg wrote:

You know, I almost cc'd you, but I didn't want to bug you until I was more sure it was a curl issue. But here you are anyway. :)

That indeed looks a lot like the same issue. And building the tip of libcurl's master and linking git against it makes the problem go away. So that is almost certainly it.

Thanks for the quick response!

-Peff

Jeff King <peff@peff.net> writes:

Thanks, both.

On Fri, Sep 11, 2020 at 09:59:28AM -0400, Jeff King wrote:

I forgot to mention: I'm using libcurl 7.72.0 (the current in debian unstable). But it also reproduces on my debian stable machine with libcurl 7.64.0.

-Peff