# Git 1.7.5 problem with HTTPS 13 messages from 2011-11-15 to 2011-11-17. Participants: Dmitry Smirnov, Shawn Pearce, Junio C Hamano, Tay Ray Chuan, Daniel Stenberg, Haitao Li, Jonathan Nieder. Thread: https://gitlist.dev/t/28940 ## Dmitry Smirnov, 2011-11-15 13:52 Subject: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CACf55T6BGds_D%3Dnbb8G%3Dm%2BJwr%2BbHFruCs-Q0%2BFOO%2BWXitXEJ-g%40mail.gmail.com ``` Hi, I have problems with downloading Android code from android.googlesource.com. The error says: fatal: branch stable is not signed I was trying to figure out what happens and finally came to conclusion that this is a problem of the git. When I try to clone the git itself using https (https://git.kernel.org/pub/scm/git/git.git) I'm getting the follwing error: warning: remote HEAD refers to nonexistent ref, unable to checkout. If I use the http URL (http://git.kernel.org/pub/scm/git/git.git) I can clone with no problems. I was also considering that the problem is caused by proxy. But when I tried to clone the same git source from another host via the same proxy, it works pretty good. The difference is the git version: on the first host it is 1.7.5.4 (comes with Ubuntu 11.10), on the second - 1.7.0.4 I was trying to collect some tcpdump and it shows the follwoing sequence 15 1.962132 X.X.X.X Y.Y.Y.Y HTTP 204 CONNECT git.kernel.org:443 HTTP/1.1 17 3.687364 Y.Y.Y.Y X.X.X.X HTTP 105 HTTP/1.0 200 Connection established 19 3.764793 X.X.X.X Y.Y.Y.Y TLSv1 208 Client Hello 21 3.815135 X.X.X.X Y.Y.Y.Y TLSv1 215 Ignored Unknown Record 23 4.045326 Y.Y.Y.Y X.X.X.X TLSv1 2239 Server Hello, Certificate, Server Key Exchange, Server Hello Done 25 4.055059 Y.Y.Y.Y X.X.X.X TLSv1 73 Alert (Level: Fatal, Description: Protocol Version) As you can see, session seems terminated with fatal alert. It looks like client tries to use TLS 1.2 but server accepts only TLS 1.0. But this is just assumption. Also, it is no clear what is that Ignored Unknown Record. But it contains uncripted text: GET /pub/scm/git/git.git/info/refs?service=git-upload-pack HTTP/1.1 User-Agent: git/1.7.5.4 Host: git.kernel.org Accept: */* Pragma: no-cache So, any suggesstions? Dmitry ``` ## Shawn Pearce, 2011-11-15 15:03 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CAJo%3DhJvdstr39suGMwxNoT%2B_cKThxsEYHi96eqja7HuHyPmhWA%40mail.gmail.com In-Reply-To: ``` On Tue, Nov 15, 2011 at 05:52, Dmitry Smirnov wrote: > I have problems with downloading Android code from android.googlesource.com. > > The error says: fatal: branch stable is not signed > > I was trying to figure out what happens and finally came to conclusion > that this is a problem of the git. Not likely. This is an error printed by the "repo" tool used by Android. It typically indicates the repo command you are executing is pointing to a URL that may be a local mirror and contain additional patches in it that were not signed by me. I would suggest starting over by downloading repo per [1] and using that script to start the process. [1] http://source.android.com/source/downloading.html > When I try to clone the git itself using https > (https://git.kernel.org/pub/scm/git/git.git) I'm getting the follwing > error: > warning: remote HEAD refers to nonexistent ref, unable to checkout. > > If I use the http URL (http://git.kernel.org/pub/scm/git/git.git) I > can clone with no problems. This may be a problem with the git.kernel.org HTTP server. It is probably not a problem with Git itself. ``` ## Junio C Hamano, 2011-11-15 23:10 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: <7v1ut9uglb.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v1ut9uglb.fsf%40alter.siamese.dyndns.org In-Reply-To: ``` Shawn Pearce writes: > On Tue, Nov 15, 2011 at 05:52, Dmitry Smirnov wrote: > >> If I use the http URL (http://git.kernel.org/pub/scm/git/git.git) I >> can clone with no problems. > > This may be a problem with the git.kernel.org HTTP server. It is > probably not a problem with Git itself. Not likely to be an issue with git.kernel.org either. I am puzzled. $ git ls-remote https://git.kernel.org/pub/scm/git/git.git | grep -e HEAD -e master bc1bbe0c19a6ff39522b4fa3259f34150e308e1f HEAD bc1bbe0c19a6ff39522b4fa3259f34150e308e1f refs/heads/master $ rungit v1.7.5.4 ls-remote https://git.kernel.org/pub/scm/git/git.git | grep -e HEAD -e master bc1bbe0c19a6ff39522b4fa3259f34150e308e1f HEAD bc1bbe0c19a6ff39522b4fa3259f34150e308e1f refs/heads/master The only thing that immediately comes to mind is the rare smart-http breakage in 1.7.5 but that was plugged in 1.7.5.1, so... ``` ## Dmitry Smirnov, 2011-11-16 07:04 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CACf55T5KPs5P76DWs_N4na5nV%2BML9DGP84gFiKt0TvjijGE25Q%40mail.gmail.com In-Reply-To: ``` 2011/11/15 Shawn Pearce : > I would suggest starting over by downloading repo per [1] and using > that script to start the process. Cloning of the repo.git repository produces the same error warning: remote HEAD refers to nonexistent ref, unable to checkout. That is why I tried to use another repository... ``` ## Dmitry Smirnov, 2011-11-16 07:11 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CACf55T6SRAfdOP1%2BqQdjeFv13B%3DG8w%2BDR-GCSEz%3D6swFhLJzcw%40mail.gmail.com In-Reply-To: <7v1ut9uglb.fsf@alter.siamese.dyndns.org> ``` 2011/11/16 Junio C Hamano : > $ git ls-remote https://git.kernel.org/pub/scm/git/git.git | >  grep -e HEAD -e master in my case this command produce no output. What if problem is caused by curl or TLS lib (libcurl-gnutls?) which is used by my git? Is there any to log something from git-remote-https ? Dmitry ``` ## Junio C Hamano, 2011-11-16 07:18 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: <7vmxbwr0vu.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vmxbwr0vu.fsf%40alter.siamese.dyndns.org In-Reply-To: ``` Dmitry Smirnov writes: > 2011/11/16 Junio C Hamano : >> $ git ls-remote https://git.kernel.org/pub/scm/git/git.git | >>  grep -e HEAD -e master > > in my case this command produce no output. > > What if problem is caused by curl or TLS lib (libcurl-gnutls?) which > is used by my git? Yeah, this is sounding very likely that this is an issue at the SSL/TLS layer underneath whatever Git speaks. ``` ## Tay Ray Chuan, 2011-11-16 07:34 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CALUzUxrM8o1uahQgSFUuvZ0mSPxG_zVQ9awOantRM2A8kkbbtA%40mail.gmail.com In-Reply-To: ``` On Wed, Nov 16, 2011 at 3:11 PM, Dmitry Smirnov wrote: > What if problem is caused by curl or TLS lib (libcurl-gnutls?) which > is used by my git? Is there any to log something from git-remote-https > ? You can run git with GIT_CURL_VERBOSE set, like this GIT_CURL_VERBOSE=1 git ls-remote ... -- Cheers, Ray Chuan ``` ## Dmitry Smirnov, 2011-11-16 08:51 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CACf55T5cp1ko45DCufcRXm%3DEeZd1-x%2BaYasvbzjDXkQH31u5VA%40mail.gmail.com In-Reply-To: ``` Thanks. I had collected two logs (for clone and ls-remote, attached). Unfortunately, I cannot see, why problem occurs. The only indication is * Connection #0 seems to be dead! Is it possible that curl sends the request in plain text? And according to tcpdump, why git/curl sends the request before Server Hello? 2011/11/16 Tay Ray Chuan : > On Wed, Nov 16, 2011 at 3:11 PM, Dmitry Smirnov wrote: >> What if problem is caused by curl or TLS lib (libcurl-gnutls?) which >> is used by my git? Is there any to log something from git-remote-https >> ? > > You can run git with GIT_CURL_VERBOSE set, like this > >  GIT_CURL_VERBOSE=1 git ls-remote ... > > -- > Cheers, > Ray Chuan > dsmirnov@dsmirnov-ubuntu2:~/projects/tmp$ GIT_CURL_VERBOSE=1 GIT_TRACE=true git clone --verbose https://git.kernel.org/pub/scm/git/git.git trace: built-in: git 'clone' '--verbose' 'https://git.kernel.org/pub/scm/git/git.git' Cloning into git... trace: run_command: 'git-remote-https' 'origin' 'https://git.kernel.org/pub/scm/git/git.git' * Couldn't find host git.kernel.org in the .netrc file; using defaults * About to connect() to proxy proxy.yyyyy.yy port 3128 (#0) * Trying Y.Y.Y.Y... * Connected to proxy.yyyyy.yy (Y.Y.Y.Y) port 3128 (#0) * Establish HTTP proxy tunnel to git.kernel.org:443 > CONNECT git.kernel.org:443 HTTP/1.1 Host: git.kernel.org:443 User-Agent: git/1.7.5.4 Proxy-Connection: Keep-Alive Pragma: no-cache < HTTP/1.0 200 Connection established < * Proxy replied OK to CONNECT request * found 157 certificates in /etc/ssl/certs/ca-certificates.crt > GET /pub/scm/git/git.git/info/refs?service=git-upload-pack HTTP/1.1 User-Agent: git/1.7.5.4 Host: git.kernel.org Accept: */* Pragma: no-cache * Connection #0 to host proxy.yyyyy.yy left intact * Couldn't find host git.kernel.org in the .netrc file; using defaults * Connection #0 seems to be dead! * Closing connection #0 * About to connect() to proxy proxy.yyyyy.yy port 3128 (#0) * Trying Y.Y.Y.Y... * Connected to proxy.yyyyy.yy (Y.Y.Y.Y) port 3128 (#0) * Establish HTTP proxy tunnel to git.kernel.org:443 > CONNECT git.kernel.org:443 HTTP/1.1 Host: git.kernel.org:443 User-Agent: git/1.7.5.4 Proxy-Connection: Keep-Alive Pragma: no-cache < HTTP/1.0 200 Connection established < * Proxy replied OK to CONNECT request * found 157 certificates in /etc/ssl/certs/ca-certificates.crt > GET /pub/scm/git/git.git/HEAD HTTP/1.1 User-Agent: git/1.7.5.4 Host: git.kernel.org Accept: */* Pragma: no-cache * Connection #0 to host proxy.yyyyy.yy left intact warning: remote HEAD refers to nonexistent ref, unable to checkout. dsmirnov@dsmirnov-ubuntu2:~/projects/tmp$ GIT_CURL_VERBOSE=1 GIT_TRACE=true git ls-remote https://git.kernel.org/pub/scm/git/git.git trace: built-in: git 'ls-remote' 'https://git.kernel.org/pub/scm/git/git.git' trace: run_command: 'git-remote-https' 'https://git.kernel.org/pub/scm/git/git.git' 'https://git.kernel.org/pub/scm/git/git.git' * Couldn't find host git.kernel.org in the .netrc file; using defaults * About to connect() to proxy proxy.yyyyy.yy port 3128 (#0) * Trying Y.Y.Y.Y... * Connected to proxy.yyyyy.yy (Y.Y.Y.Y) port 3128 (#0) * Establish HTTP proxy tunnel to git.kernel.org:443 > CONNECT git.kernel.org:443 HTTP/1.1 Host: git.kernel.org:443 User-Agent: git/1.7.5.4 Proxy-Connection: Keep-Alive Pragma: no-cache < HTTP/1.0 200 Connection established < * Proxy replied OK to CONNECT request * found 157 certificates in /etc/ssl/certs/ca-certificates.crt > GET /pub/scm/git/git.git/info/refs?service=git-upload-pack HTTP/1.1 User-Agent: git/1.7.5.4 Host: git.kernel.org Accept: */* Pragma: no-cache * Connection #0 to host proxy.yyyyy.yy left intact * Couldn't find host git.kernel.org in the .netrc file; using defaults * Connection #0 seems to be dead! * Closing connection #0 * About to connect() to proxy proxy.yyyyy.yy port 3128 (#0) * Trying Y.Y.Y.Y... * Connected to proxy.yyyyy.yy (Y.Y.Y.Y) port 3128 (#0) * Establish HTTP proxy tunnel to git.kernel.org:443 > CONNECT git.kernel.org:443 HTTP/1.1 Host: git.kernel.org:443 User-Agent: git/1.7.5.4 Proxy-Connection: Keep-Alive Pragma: no-cache < HTTP/1.0 200 Connection established < * Proxy replied OK to CONNECT request * found 157 certificates in /etc/ssl/certs/ca-certificates.crt > GET /pub/scm/git/git.git/HEAD HTTP/1.1 User-Agent: git/1.7.5.4 Host: git.kernel.org Accept: */* Pragma: no-cache * Connection #0 to host proxy.yyyyy.yy left intact 0000000000000000000000000000000000000000  0000000000000000000000000000000000000000 Cape 0000000000000000000000000000000000000000 UUS10U ca00000000000000000000000000000000000000 UUS10U 0000000000000000000000000000000000000 *�H� 0000000000000000000000000000000000000 *�H� 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 (��v�K\�M�d�����H���oo��D����]��g�I�Fo�1���﷋��_^U#��2o����� ``` ## Daniel Stenberg, 2011-11-16 09:13 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.00.1111161008430.19479%40tvnag.unkk.fr In-Reply-To: ``` On Wed, 16 Nov 2011, Dmitry Smirnov wrote: > Unfortunately, I cannot see, why problem occurs. The only indication is > * Connection #0 seems to be dead! That means libcurl wanted to re-use an existing connection, but it seems to have died in the mean time and therefore it has to create a new one and reconnect instead. I suppose that is the first indication that something isn't quite right. > Is it possible that curl sends the request in plain text? I'd say that isn't very likely and you could easily snoop on the network to figure that out for sure. > And according to tcpdump, why git/curl sends the request before Server > Hello? curl will send the HTTP request once the TLS negotiation has completed as told by the TLS library. I believe you said you're using GnuTLS, are you using a recent version? This is not a transfer layer (curl/HTTPS) bug I recognize, but I can of course not rule out that there's a bug somewhere in there! -- / daniel.haxx.se ``` ## Dmitry Smirnov, 2011-11-16 10:10 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CACf55T6rUiE8Pm3s16oFgJ1ueTAdOwOz_%2BXptE-tZM1z9fcq%2Bw%40mail.gmail.com In-Reply-To: ``` > I'd say that isn't very likely and you could easily snoop on the network to> figure that out for sure In the very first message I wrote that there is strange tcpdump record: 21 3.815135 X.X.X.X Y.Y.Y.Y TLSv1 215 Ignored Unknown Record In this record there is some binary dump followed by an uncripted text: GET /pub/scm/git/git.git/info/refs?service=git-upload-pack HTTP/1.1 User-Agent: git/1.7.5.4 Host: git.kernel.org Accept: */* Pragma: no-cache This packet is recorded before negotiation complete, so I'm wondering who is guilty: git or curl? What Git is providing to libcurl? Can I log it? > curl will send the HTTP request once the TLS negotiation has completed as > told by the TLS library. I believe you said you're using GnuTLS, are you > using a recent version? I'm using the version that comes with Ubuntu 11.10. ``` ## Haitao Li, 2011-11-16 10:32 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CABk5KLiiNSwf_-FxdWZON_zxVuLLGbFfaQhzuC8dxmYDXEDA6g%40mail.gmail.com In-Reply-To: ``` > I was also considering that the problem is caused by proxy. But when I > tried to clone the same git source from another host via the same > proxy, it works pretty good. The difference is the git version: on the > first host it is 1.7.5.4 (comes with Ubuntu 11.10), on the second - > 1.7.0.4 The proxy may have some impact. I see exactly the same error only behind a proxy on my laptop running Ubuntu 11.10 with libgnutls26/2.10.5-1ubuntu3. The same laptop works fine at home without proxy. I have another machine (Ubuntu 11.04 git/1.7.4.1 libgnutls26/2.8.6-1ubuntu2) works fine behind the same proxy. ``` ## Jonathan Nieder, 2011-11-16 22:28 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: <20111116222850.GA17750@elie.hsd1.il.comcast.net> URL: https://gitlist.dev/e/20111116222850.GA17750%40elie.hsd1.il.comcast.net In-Reply-To: ``` Dmitry Smirnov wrote: > What Git is providing to libcurl? Can I log it? ltrace can help. ``` ## Dmitry Smirnov, 2011-11-17 06:36 Subject: Re: Git 1.7.5 problem with HTTPS Message-ID: URL: https://gitlist.dev/e/CACf55T6nHTv3knUo0PvmqqPO7JXKf%2Bo5H5ESLoPr9b%3DZW5kudA%40mail.gmail.com In-Reply-To: ``` I had fixed the problem by manually installing the most recent version of the libcurl3-gnutls for Ubuntu (from precise): http://packages.ubuntu.com/precise/libcurl3-gnutls It will require also most recent libgnutls: http://packages.ubuntu.com/precise/libgnutls26 Dmitry ```