threads / discuss / 22319

problem cloning via http since v1.6.6-rc0

Subject: problem cloning via http since v1.6.6-rc0

## tl;dr

28 messages between Jan 21, 2010 and Nov 3, 2010.

replies: 27people: 8as markdown or json

Yaroslav Halchenko· Jan 21, 2010, 00:47 UTC · lore
Dear Git Developers,

Some users of our project started recently to complain that they could not clone the repository via http (git:// wasn't a choice due to heavy firewalling) and because http:// was used as a protocol to get sources in some distributions (e.g. macports).

Cloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0. I haven't done full bisection since that repository is relatively bulky and poor server is quite loaded anyways, so I thought you just would get a clue without going brute-force. But here are the details: in case of failing operation, I immediately get failure:

$> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' warning: templates not found /home/yoh/share/git-core/templates Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/ trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

on the server, 1.6.3.3 version of git was used to run git update-server-info.

Thanks in advance
-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
Tay Ray Chuan· Jan 21, 2010, 01:34 UTC · re: Yaroslav Halchenko · lore

Re: problem cloning via http since v1.6.6-rc0

Hi,

On Thu, Jan 21, 2010 at 8:47 AM, Yaroslav Halchenko <debian@onerussian.com> wrote:

> Cloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0.
this sounds like around the time the smart http protocol was introduced.
> fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?
>
> on the server, 1.6.3.3 version of git was used to run git
> update-server-info.

hmm, are you using the WebDAV-flavour or the smart http protocol to host the repository?

-- 
Cheers,
Ray Chuan
Tay Ray Chuan· Jan 21, 2010, 01:36 UTC · re: Yaroslav Halchenko · lore

Re: problem cloning via http since v1.6.6-rc0

On Thu, Jan 21, 2010 at 8:47 AM, Yaroslav Halchenko <debian@onerussian.com> wrote:

Show 9 quoted lines
> $> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git
> trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> warning: templates not found /home/yoh/share/git-core/templates
> Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
> trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?
oh, and by the way, could you also run this again with GIT_CURL_VERBOSE=1?
-- 
Cheers,
Ray Chuan
Yaroslav Halchenko· Jan 21, 2010, 02:33 UTC · re: Tay Ray Chuan · lore

Re: problem cloning via http since v1.6.6-rc0

$> GIT_CURL_VERBOSE=1 GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git 
trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
warning: templates not found /home/yoh/share/git-core/templates
Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
trace: run_command: 'remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git' 'remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git-remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: run_command: 'git-remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
* Couldn't find host git.debian.org in the .netrc file; using defaults
* About to connect() to git.debian.org port 80 (#0)
*   Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)
> GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.6.6.267.g5b159
Host: git.debian.org
Accept: */*
Pragma: no-cache
* The requested URL returned error: 404
* Closing connection #0
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

as for smart vs DAV -- don't see any smart alias handling in apache configuration (I have/had no clue about some smart http in git, just looked at apache template and saw smart aliases -- is there smth else to check within webserver config?)

On Thu, 21 Jan 2010, Tay Ray Chuan wrote:
Show 11 quoted lines
> On Thu, Jan 21, 2010 at 8:47 AM, Yaroslav Halchenko
> <debian@onerussian.com> wrote:
> > $> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git
> > trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> > warning: templates not found /home/yoh/share/git-core/templates
> > Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
> > trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> > trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> > trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> > trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> > fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?
> oh, and by the way, could you also run this again with GIT_CURL_VERBOSE=1?
-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
Tay Ray Chuan· Jan 21, 2010, 04:01 UTC · re: Yaroslav Halchenko · lore

Re: problem cloning via http since v1.6.6-rc0

Hi,

On Thu, Jan 21, 2010 at 10:33 AM, Yaroslav Halchenko <debian@onerussian.com> wrote:

Show 10 quoted lines
> *   Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)
>> GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1
> User-Agent: git/1.6.6.267.g5b159
> Host: git.debian.org
> Accept: */*
> Pragma: no-cache
>
> * The requested URL returned error: 404
> * Closing connection #0
> fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

I don't think git's at fault here, as we're getting a 404 Not Found. Could you check that the repository (the one the url points to, after taking into any url rewriting) is a bare one, ie. has structure

  pkg-exppsy/
  |-pymvpa.git
    |-objects/
    |-info/
    |-refs/
    |-...
rather than the non-bare
  pkg-exppsy/
  |-pymvpa.git
    |-.git
      |-objects/
      |-info/
      |-refs/
      |-...
?
> as for smart vs DAV -- don't see any smart alias handling in apache
> configuration (I have/had no clue about some smart http in git, just looked at
> apache template and saw smart aliases -- is there smth else to check within
> webserver config?)

If that's the case, I don't think it's related to your problem. (Btw, "smart" refers to the http protocol that git can use to sync your repo, via a CGI program on the server, instead of WebDAV. See git-http-backend(1) for details.)

-- 
Cheers,
Ray Chuan
Yaroslav Halchenko· Jan 21, 2010, 04:38 UTC · re: Tay Ray Chuan · lore

Re: problem cloning via http since v1.6.6-rc0

On Thu, 21 Jan 2010, Tay Ray Chuan wrote:
> > fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?
> I don't think git's at fault here, as we're getting a 404 Not Found.

khe khe, pardon me, but if git can't talk to its nephew (ie its own repository which was created with earlier version), whenever another nephew (ie git of earlier version) can talk to it, I do consider it to be git's fault ;)

let me zoom in onto difference in communication between two different versions :
*   Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)
> GET /git/pkg-exppsy/pymvpa.git/info/refs HTTP/1.1
User-Agent: git/1.6.5
Host: git.debian.org
Accept: */*
Pragma: no-cache
< HTTP/1.1 200 OK
whenever, once again, for 1.6.6 it looked much shorter:
*   Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)
> GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.6.6.267.g5b159
Host: git.debian.org
Accept: */*
Pragma: no-cache
* The requested URL returned error: 404
* Closing connection #0
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?
> Could you check that the repository (the one the url points to, after
> taking into any url rewriting) is a bare one, ie. has structure

yes - it is bare... I was 101% sure (isn't that a convention to have .git suffix for directories with bare repositories), but just to make sure:

$> cd /srv/git.debian.org/git/pkg-exppsy/pymvpa.git/
total 76
 8 branches/   8 config   8 description   8 HEAD   8 hooks/   8 info/   8 objects/  12 packed-refs   8 refs/
> If that's the case, I don't think it's related to your problem. (Btw,
> "smart" refers to the http protocol that git can use to sync your
> repo, via a CGI program on the server, instead of WebDAV. See
> git-http-backend(1) for details.)
thanks for info, I did know about this beastie.

I see no references to git-http-backend in apache config -- so indeed should not be the case... but from the git-http-backend description:

,---
| By default, only the `upload-pack` service is enabled, which serves
| 'git-fetch-pack' and 'git-ls-remote' clients, which are invoked from
| 'git-fetch', 'git-pull', and 'git-clone'.
`---

so, it looks like 1.6.6 for some reason decided to assume that it is "smart" http whenever it is not? is that the case here?

-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
Ilari Liusvaara· Jan 21, 2010, 05:08 UTC · re: Yaroslav Halchenko · lore

Re: problem cloning via http since v1.6.6-rc0

On Wed, Jan 20, 2010 at 07:47:56PM -0500, Yaroslav Halchenko wrote:
Added spearce to cc.
Show 22 quoted lines
> Dear Git Developers,
> 
> Some users of our project started recently to complain that they could not
> clone the repository via http (git:// wasn't a choice due to heavy firewalling)
> and because http:// was used as a protocol to get sources in some distributions
> (e.g. macports).
> 
> Cloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0.
> I haven't done full bisection since that repository is relatively bulky and
> poor server is quite loaded anyways, so I thought you just would get a clue
> without going brute-force.  But here are the details:  in case of failing
> operation, I immediately get failure:
> 
> $> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git
> trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> warning: templates not found /home/yoh/share/git-core/templates
> Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
> trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
> fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?
Looks like remote-curl (which handles http) issues request for:
'.../info/refs?service=git-upload-pack'

And expects that if there is no smart HTTP server there for the request to be interpretted as:

'.../info/refs'

(i.e. webserver would ignore the query). This isn't true for git.debian.org. Requesting the latter works (and the data formatting looks sane), but the former is 404. This causes the fetch to fail.

-Ilari
Tay Ray Chuan· Jan 21, 2010, 06:47 UTC · re: Ilari Liusvaara · lore

Re: problem cloning via http since v1.6.6-rc0

Hi,

On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara <ilari.liusvaara@elisanet.fi> wrote:

Show 12 quoted lines
> Looks like remote-curl (which handles http) issues request for:
>
> '.../info/refs?service=git-upload-pack'
>
> And expects that if there is no smart HTTP server there for the request to be
> interpretted as:
>
> '.../info/refs'
>
> (i.e. webserver would ignore the query). This isn't true for git.debian.org.
> Requesting the latter works (and the data formatting looks sane), but the
> former is 404. This causes the fetch to fail.

afaik, putting a "?var1=val1&var2=...." still makes it a normal GET request, even if the url requested is just a plain file and not some cgi handler that uses those variables/values.

-- Cheers, Ray Chuan

Tay Ray Chuan· Jan 21, 2010, 07:51 UTC · re: Tay Ray Chuan · lore

Re: problem cloning via http since v1.6.6-rc0

Hi,

On Thu, 21 Jan 2010 14:47:37 +0800 Tay Ray Chuan <rctay89@gmail.com> wrote:

Show 18 quoted lines
> On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara
> <ilari.liusvaara@elisanet.fi> wrote:
> > Looks like remote-curl (which handles http) issues request for:
> >
> > '.../info/refs?service=git-upload-pack'
> >
> > And expects that if there is no smart HTTP server there for the request to be
> > interpretted as:
> >
> > '.../info/refs'
> >
> > (i.e. webserver would ignore the query). This isn't true for git.debian.org.
> > Requesting the latter works (and the data formatting looks sane), but the
> > former is 404. This causes the fetch to fail.
> 
> afaik, putting a "?var1=val1&var2=...." still makes it a normal GET
> request, even if the url requested is just a plain file and not some
> cgi handler that uses those variables/values.

Yaroslav, sorry for making you run in circles - it really is git's fault (sorta).

In recent versions of git, we were sending out the GET request for info/refs with a query string (?serivce=<service name>). I'm not sure why, but your server is not playing nice when the query string is appended.

Could you try this patch and see if it solves the issue? I manage to clone your repo successfully with it.

-- 
Cheers,
Ray Chuan

-->8--
Subject: [PATCH] http/remote-curl: coddle picky servers

When "info/refs" is a static file and not behind a CGI handler, some
servers may not handle a GET request for it with a query string
appended (eg. "?foo=bar") properly.

If such a request fails, retry it sans the query string, and also
discount the possibility of using the "smart" protocol (since no
service is specified with "?service=<service name>").

Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
---
 remote-curl.c |   18 ++++++++++++++++--
 1 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/remote-curl.c b/remote-curl.c
index 1361006..a904164 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -102,7 +102,7 @@ static struct discovery* discover_refs(const char *service)
 	struct strbuf buffer = STRBUF_INIT;
 	struct discovery *last = last_discovery;
 	char *refs_url;
-	int http_ret, is_http = 0;
+	int http_ret, is_http = 0, proto_git_candidate = 1;
 
 	if (last && !strcmp(service, last->service))
 		return last;
@@ -121,6 +121,19 @@ static struct discovery* discover_refs(const char *service)
 
 	init_walker();
 	http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+
+	/* try again with "plain" url (no ? or & appended) */
+	if (http_ret != HTTP_OK) {
+		free(refs_url);
+		strbuf_reset(&buffer);
+
+		proto_git_candidate = 0;
+		strbuf_addf(&buffer, "%s/info/refs", url);
+		refs_url = strbuf_detach(&buffer, NULL);
+
+		http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+	}
+
 	switch (http_ret) {
 	case HTTP_OK:
 		break;
@@ -137,7 +150,8 @@ static struct discovery* discover_refs(const char *service)
 	last->buf_alloc = strbuf_detach(&buffer, &last->len);
 	last->buf = last->buf_alloc;
 
-	if (is_http && 5 <= last->len && last->buf[4] == '#') {
+	if (is_http && proto_git_candidate
+		&& 5 <= last->len && last->buf[4] == '#') {
 		/* smart HTTP response; validate that the service
 		 * pkt-line matches our request.
 		 */
-- 
1.6.6.1.337.g96bc8
Yaroslav Halchenko· Jan 21, 2010, 14:00 UTC · re: Tay Ray Chuan · lore

Re: problem cloning via http since v1.6.6-rc0

Hi Tay Ray,

That patch works fine for me ;) I only hope it would get accepted into bugfix and next dev release (I guess it might annoy some of apache admins a bit due to increase of their errors.log now even for well maintained repositories, but well -- that is life ;-) )

Thanks! Yarik

On Thu, 21 Jan 2010, Tay Ray Chuan wrote:
> > afaik, putting a "?var1=val1&var2=...." still makes it a normal GET
> > request, even if the url requested is just a plain file and not some
> > cgi handler that uses those variables/values.
> Yaroslav, sorry for making you run in circles - it really is git's
> fault (sorta).
> In recent versions of git, we were sending out the GET request for
> info/refs with a query string (?serivce=<service name>). I'm not sure
> why, but your server is not playing nice when the query string is
> appended.
> Could you try this patch and see if it solves the issue? I manage to
> clone your repo successfully with it.
-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]
Tay Ray Chuan· Jan 21, 2010, 14:41 UTC · re: Yaroslav Halchenko · lore

[PATCH] http/remote-curl: coddle picky servers

When "info/refs" is a static file and not behind a CGI handler, some servers may not handle a GET request for it with a query string appended (eg. "?foo=bar") properly.

If such a request fails, retry it sans the query string. In addition, ensure that the "smart" http protocol is not used (a service has to be specified with "?service=<service name>" to be conformant).

Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Reported-and-tested-by: Yaroslav Halchenko <debian@onerussian.com>
---
 remote-curl.c |   18 ++++++++++++++++--
 1 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index 1361006..a904164 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -102,7 +102,7 @@ static struct discovery* discover_refs(const char *service)
 	struct strbuf buffer = STRBUF_INIT;
 	struct discovery *last = last_discovery;
 	char *refs_url;
-	int http_ret, is_http = 0;
+	int http_ret, is_http = 0, proto_git_candidate = 1;

 	if (last && !strcmp(service, last->service))
 		return last;
@@ -121,6 +121,19 @@ static struct discovery* discover_refs(const char *service)

 	init_walker();
 	http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+
+	/* try again with "plain" url (no ? or & appended) */
+	if (http_ret != HTTP_OK) {
+		free(refs_url);
+		strbuf_reset(&buffer);
+
+		proto_git_candidate = 0;
+		strbuf_addf(&buffer, "%s/info/refs", url);
+		refs_url = strbuf_detach(&buffer, NULL);
+
+		http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+	}
+
 	switch (http_ret) {
 	case HTTP_OK:
 		break;
@@ -137,7 +150,8 @@ static struct discovery* discover_refs(const char *service)
 	last->buf_alloc = strbuf_detach(&buffer, &last->len);
 	last->buf = last->buf_alloc;

-	if (is_http && 5 <= last->len && last->buf[4] == '#') {
+	if (is_http && proto_git_candidate
+		&& 5 <= last->len && last->buf[4] == '#') {
 		/* smart HTTP response; validate that the service
 		 * pkt-line matches our request.
 		 */
--
1.6.6.1.337.g96bc8
Shawn O. Pearce· Jan 21, 2010, 15:56 UTC · re: Tay Ray Chuan · lore

Re: [PATCH] http/remote-curl: coddle picky servers

Tay Ray Chuan <rctay89@gmail.com> wrote:
Show 10 quoted lines
> When "info/refs" is a static file and not behind a CGI handler, some
> servers may not handle a GET request for it with a query string
> appended (eg. "?foo=bar") properly.
> 
> If such a request fails, retry it sans the query string. In addition,
> ensure that the "smart" http protocol is not used (a service has to be
> specified with "?service=<service name>" to be conformant).
> 
> Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
> Reported-and-tested-by: Yaroslav Halchenko <debian@onerussian.com>
*grumble* stupid Apache *grumble*
Acked-by: Shawn O. Pearce <spearce@spearce.org>
-- 
Shawn.
Mike Hommey· Jan 21, 2010, 16:07 UTC · re: Shawn O. Pearce · lore

Re: [PATCH] http/remote-curl: coddle picky servers

On Thu, Jan 21, 2010 at 07:56:37AM -0800, Shawn O. Pearce wrote:
Show 13 quoted lines
> Tay Ray Chuan <rctay89@gmail.com> wrote:
> > When "info/refs" is a static file and not behind a CGI handler, some
> > servers may not handle a GET request for it with a query string
> > appended (eg. "?foo=bar") properly.
> > 
> > If such a request fails, retry it sans the query string. In addition,
> > ensure that the "smart" http protocol is not used (a service has to be
> > specified with "?service=<service name>" to be conformant).
> > 
> > Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
> > Reported-and-tested-by: Yaroslav Halchenko <debian@onerussian.com>
> 
> *grumble* stupid Apache *grumble*
stupid Apache... configuration.

Check the error message you get on http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack:

The requested URL /gitweb.cgigit/pkg-exppsy/pymvpa.git/info/refs was not found on this server.

Look closely at the start of the requested URL: /gitweb.cgi... It comes from this rule:

RewriteCond %{QUERY_STRING} ^(.+)$ RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]

which is global to the virtual host.

Anyways, while git.debian.org can certainly be fixed for that, other servers may want to do some different things with urls with parameters.

Mike
Michael S. Tsirkin· Jan 21, 2010, 16:10 UTC · re: Mike Hommey · lore

git fetch -v not at all verbose?

Hi! On many of my trees (with linux kernel), git fetch is slower than git clone. Even more annoyingly, it would hang sometimes for tens of minutes without any output, even if -v is supplied.

stracing it shows a ton of lines like the following: 16324 read(10, "ACK 4bbdfe65d23014f539fec4227260"..., 51) = 51 16324 read(10, "0037", 4) = 4 16324 read(10, "ACK 322c06560fa314b04a6302ea03c0"..., 51) = 51 16324 read(10, "0037", 4) = 4 16324 read(10, "ACK 848ea2043b128b5947851866a114"..., 51) = 51 16324 read(10, "0037", 4) = 4

Is there some way to make got fetch show progress at this stage, or even better, can it be made faster somehow?

Thanks!
-- 
MST
Shawn O. Pearce· Jan 21, 2010, 16:18 UTC · re: Michael S. Tsirkin · lore

Re: git fetch -v not at all verbose?

"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On many of my trees (with linux kernel), git fetch is slower than git clone.
> Even more annoyingly, it would hang sometimes for tens of minutes without any
> output, even if -v is supplied.
Ouch.  I think -v -v boosts the output to be ever more verbose,
and might actually show you something.
Show 7 quoted lines
> stracing it shows a ton of lines like the following:
> 16324 read(10, "ACK 4bbdfe65d23014f539fec4227260"..., 51) = 51
> 16324 read(10, "0037", 4)               = 4
> 16324 read(10, "ACK 322c06560fa314b04a6302ea03c0"..., 51) = 51
> 16324 read(10, "0037", 4)               = 4
> 16324 read(10, "ACK 848ea2043b128b5947851866a114"..., 51) = 51
> 16324 read(10, "0037", 4)               = 4
That's the peers trying to determine a common base.
 
> Is there some way to make got fetch show progress at this stage,
> or even better, can it be made faster somehow?
We shouldn't need to show progress here, we should just be faster.

Given the symptom, it sounds to me like your local repository is some 1,000s of commits ahead of the remote repository you are fetching from. Is that true?

Are you fetching from a configured remote that has tracking branches, or are you fetching through a one-shot URL pasted onto the command line?

-- 
Shawn.
Michael S. Tsirkin· Jan 21, 2010, 16:35 UTC · re: Shawn O. Pearce · lore

Re: git fetch -v not at all verbose?

On Thu, Jan 21, 2010 at 08:18:58AM -0800, Shawn O. Pearce wrote:
Show 26 quoted lines
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On many of my trees (with linux kernel), git fetch is slower than git clone.
> > Even more annoyingly, it would hang sometimes for tens of minutes without any
> > output, even if -v is supplied.
> 
> Ouch.  I think -v -v boosts the output to be ever more verbose,
> and might actually show you something.
>  
> > stracing it shows a ton of lines like the following:
> > 16324 read(10, "ACK 4bbdfe65d23014f539fec4227260"..., 51) = 51
> > 16324 read(10, "0037", 4)               = 4
> > 16324 read(10, "ACK 322c06560fa314b04a6302ea03c0"..., 51) = 51
> > 16324 read(10, "0037", 4)               = 4
> > 16324 read(10, "ACK 848ea2043b128b5947851866a114"..., 51) = 51
> > 16324 read(10, "0037", 4)               = 4
> 
> That's the peers trying to determine a common base.
>  
> > Is there some way to make got fetch show progress at this stage,
> > or even better, can it be made faster somehow?
> 
> We shouldn't need to show progress here, we should just be faster.
> 
> Given the symptom, it sounds to me like your local repository
> is some 1,000s of commits ahead of the remote repository you are
> fetching from.  Is that true?

Hmm, no, but what is true is that I fetched several remotes that diverged significantly into the same local repository. Would that have same effect?

> Are you fetching from a configured remote that has tracking branches,
> or are you fetching through a one-shot URL pasted onto the command
> line?
Configured remote.
> -- 
> Shawn.
Shawn O. Pearce· Jan 21, 2010, 16:57 UTC · re: Michael S. Tsirkin · lore

Re: git fetch -v not at all verbose?

"Michael S. Tsirkin" <mst@redhat.com> wrote:
Show 5 quoted lines
> On Thu, Jan 21, 2010 at 08:18:58AM -0800, Shawn O. Pearce wrote:
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On many of my trees (with linux kernel), git fetch is slower than git clone.
> > > Even more annoyingly, it would hang sometimes for tens of minutes without any
> > > output, even if -v is supplied.
...
Show 7 quoted lines
> > Given the symptom, it sounds to me like your local repository
> > is some 1,000s of commits ahead of the remote repository you are
> > fetching from.  Is that true?
> 
> Hmm, no, but what is true is that I fetched several remotes
> that diverged significantly into the same local repository.
> Would that have same effect?
Yes.
Show 5 quoted lines
> > Are you fetching from a configured remote that has tracking branches,
> > or are you fetching through a one-shot URL pasted onto the command
> > line?
> 
> Configured remote.

Hmm. I wonder if we should try to shortcut the commit walking in a case like this and just feed the tracking branches we already have.

-- 
Shawn.
Michael S. Tsirkin· Jan 21, 2010, 17:30 UTC · re: Shawn O. Pearce · lore

Re: git fetch -v not at all verbose?

On Thu, Jan 21, 2010 at 08:57:37AM -0800, Shawn O. Pearce wrote:
Show 25 quoted lines
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Thu, Jan 21, 2010 at 08:18:58AM -0800, Shawn O. Pearce wrote:
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > On many of my trees (with linux kernel), git fetch is slower than git clone.
> > > > Even more annoyingly, it would hang sometimes for tens of minutes without any
> > > > output, even if -v is supplied.
> ...
> > > Given the symptom, it sounds to me like your local repository
> > > is some 1,000s of commits ahead of the remote repository you are
> > > fetching from.  Is that true?
> > 
> > Hmm, no, but what is true is that I fetched several remotes
> > that diverged significantly into the same local repository.
> > Would that have same effect?
> 
> Yes.
> 
> > > Are you fetching from a configured remote that has tracking branches,
> > > or are you fetching through a one-shot URL pasted onto the command
> > > line?
> > 
> > Configured remote.
> 
> Hmm.  I wonder if we should try to shortcut the commit walking in
> a case like this and just feed the tracking branches we already have.

Or for the case of 1,000s of commits ahead, git could try to implement a heuristic to reduce the number of commits sent. Currently all commits are sent in order, correct? How about binary search like what git bisect does?

> -- 
> Shawn.
Thomas Rast· Jan 21, 2010, 17:47 UTC · re: Michael S. Tsirkin · lore

Re: git fetch -v not at all verbose?

On Thursday 21 January 2010 18:30:10 Michael S. Tsirkin wrote:
Show 5 quoted lines
> On Thu, Jan 21, 2010 at 08:57:37AM -0800, Shawn O. Pearce wrote:
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > Hmm, no, but what is true is that I fetched several remotes
> > > that diverged significantly into the same local repository.
> > > Would that have same effect?
[...]
Show 7 quoted lines
> > Hmm.  I wonder if we should try to shortcut the commit walking in
> > a case like this and just feed the tracking branches we already have.
> 
> Or for the case of 1,000s of commits ahead, git could try to implement a
> heuristic to reduce the number of commits sent. Currently all commits
> are sent in order, correct?  How about binary search like what git
> bisect does?

I had a patch for this ages ago (that combines exponential-stride backwards search and later bisection), but it was shot down on grounds of not working at times and code convolution and I forgot about it...

I can give this another shot, but it seems most of the code has moved due to the transport handlers changes, so I'll first have to read into it again.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Junio C Hamano· Jan 21, 2010, 17:42 UTC · re: Shawn O. Pearce · lore

Re: git fetch -v not at all verbose?

"Shawn O. Pearce" <spearce@spearce.org> writes:
Show 8 quoted lines
>> > Are you fetching from a configured remote that has tracking branches,
>> > or are you fetching through a one-shot URL pasted onto the command
>> > line?
>> 
>> Configured remote.
>
> Hmm.  I wonder if we should try to shortcut the commit walking in
> a case like this and just feed the tracking branches we already have.

You mean that the main culprit is the presense of thousdands of commits that fetcher has obtained through the other remotes (and his own) that the uploader makes fetcher walk all the way, in the false hope that there might be a commit among them that is closer to the commits being fetched than the ones at the tip of tracking branch the fetcher has for this uploader currently?

And the solution might be to tell only about the tips of remote tracking branches fetcher has obtained from this particular uploader, not about other remote tracking bracnesh it got from others or his own local branches (which may have merged from other remotes)?

It is a clever idea but I suspect it may not work well in practice. For example, suppose a project is two-tier, say, with top-level and subsystem repositories, the former of which regularly merge from the latter, and you are a participant primarily working on the subsystem. You fetch daily from the subsystem repository, but weekly from the top-level.

Now, when you fetch from the top-level, the remote tracking refs you have for it are much more stale than your other refs. The top-level would have acquired a lot more commits from the same subsystem repository since you fetched from there the last time, and you already have many of them through your daily fetch from the subsystem repository. To minimize the transfer in such a case, the fetcher does want to tell the uploader that it has those commits from the same subsystem repository, so that the commit walker can stop at a recent merge into the top-level from the subsystem repository.

There was a discussion about updating the commit walk exchange to bisect the history (skip and try a much older one to see if it is reachable, but to avoid overshooting, step back and see if a newer one is still common). It would be a lot more work and needs to be implemented as a new protocol capability, but I think it is the right way to go in the longer term.

Michael S. Tsirkin· Nov 3, 2010, 09:52 UTC · re: Junio C Hamano · lore

Re: git fetch -v not at all verbose?

On Thu, Jan 21, 2010 at 09:42:36AM -0800, Junio C Hamano wrote:
Show 44 quoted lines
> "Shawn O. Pearce" <spearce@spearce.org> writes:
> 
> >> > Are you fetching from a configured remote that has tracking branches,
> >> > or are you fetching through a one-shot URL pasted onto the command
> >> > line?
> >> 
> >> Configured remote.
> >
> > Hmm.  I wonder if we should try to shortcut the commit walking in
> > a case like this and just feed the tracking branches we already have.
> 
> You mean that the main culprit is the presense of thousdands of commits
> that fetcher has obtained through the other remotes (and his own) that the
> uploader makes fetcher walk all the way, in the false hope that there
> might be a commit among them that is closer to the commits being fetched
> than the ones at the tip of tracking branch the fetcher has for this
> uploader currently?
> 
> And the solution might be to tell only about the tips of remote tracking
> branches fetcher has obtained from this particular uploader, not about
> other remote tracking bracnesh it got from others or his own local
> branches (which may have merged from other remotes)?
> 
> It is a clever idea but I suspect it may not work well in practice.  For
> example, suppose a project is two-tier, say, with top-level and subsystem
> repositories, the former of which regularly merge from the latter, and you
> are a participant primarily working on the subsystem.  You fetch daily
> from the subsystem repository, but weekly from the top-level.
> 
> Now, when you fetch from the top-level, the remote tracking refs you have
> for it are much more stale than your other refs.  The top-level would have
> acquired a lot more commits from the same subsystem repository since you
> fetched from there the last time, and you already have many of them
> through your daily fetch from the subsystem repository.  To minimize the
> transfer in such a case, the fetcher does want to tell the uploader that
> it has those commits from the same subsystem repository, so that the
> commit walker can stop at a recent merge into the top-level from the
> subsystem repository.
> 
> There was a discussion about updating the commit walk exchange to bisect
> the history (skip and try a much older one to see if it is reachable, but
> to avoid overshooting, step back and see if a newer one is still common).
> It would be a lot more work and needs to be implemented as a new protocol
> capability, but I think it is the right way to go in the longer term.

I thought about this some more: it seems that nothing in pack-protocol.txt dictates that client has to send have lines in order. The whole logic would be on client side.

So a new capability will be there just in case we find a use for a server-side optimization later on, we don't need the client to behave differently in any way when this capability is enabled/disabled. Right?

-- 
MST
Junio C Hamano· Nov 3, 2010, 16:14 UTC · re: Michael S. Tsirkin · lore

Re: git fetch -v not at all verbose?

"Michael S. Tsirkin" <mst@redhat.com> writes:
Show 11 quoted lines
> On Thu, Jan 21, 2010 at 09:42:36AM -0800, Junio C Hamano wrote:
> ...
>> There was a discussion about updating the commit walk exchange to bisect
>> the history (skip and try a much older one to see if it is reachable, but
>> to avoid overshooting, step back and see if a newer one is still common).
>> It would be a lot more work and needs to be implemented as a new protocol
>> capability, but I think it is the right way to go in the longer term.
>
> I thought about this some more: it seems that nothing in
> pack-protocol.txt dictates that client has to send have
> lines in order. The whole logic would be on client side.

The current protocol may not require any order for it to function correctly in the sense that the sent pack will contain everything that is necessary, but it does require that commits on a lineage to be sent from near tip to near root if you want to have a _good_ common ancestor to be found.

If the downloader sends older commits first without sending some new ones, the uploader can say "Ok, I know about that old one you told me you have, so we could use that as a common commit" [*1*]. But there is no way for it to continue the sentence with "... but I cannot tell if other ones you told me you have that I know nothing about are all directly connected to that common one we just found (in which case that common one is the best we can do), or you have newer ones than the common commit that I also have but you omitted from the listing (in other words, if you didn't omit them, we could have found a better common commit). Could you please back up a bit and let us see if we can do better with newer ones?" with the current protocol exchange.

The downloader _could_, upon seeing an ACK to a commit that is an ancestor of commits that it skipped, try sending these skipped commits, without telling the uploader that it what it is doing. But the uploader will unilaterally decide when it thinks it has heard enough, after giving an ACK back in the original protocol, or after finding enough common ancestors to cover all the tips requested with WANTs, so I suspect that you may not have a chance to play such a game without an explicit protocol extension.

[Footnote]

*1* That is what an ACK means. In an multi-ack exchange, it also tells the downloader there is no point to give any ancestors of that commit, but allows the downloader to continue sending commits from other lineage.

Tay Ray Chuan· Jan 21, 2010, 16:20 UTC · re: Mike Hommey · lore

Re: [PATCH] http/remote-curl: coddle picky servers

Hi,
On Fri, Jan 22, 2010 at 12:07 AM, Mike Hommey <mh@glandium.org> wrote:
Show 10 quoted lines
> Look closely at the start of the requested URL: /gitweb.cgi...
> It comes from this rule:
>
> RewriteCond %{QUERY_STRING} ^(.+)$
> RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]
>
> which is global to the virtual host.
>
> Anyways, while git.debian.org can certainly be fixed for that, other
> servers may want to do some different things with urls with parameters.
heh, I was suspecting some URL rewriting was going on.
Is this an issue that should be fixed in gitweb?
(added John 'Warthog9' Hawley to the Cc list, perhaps he might know.)
-- 
Cheers,
Ray Chuan
Shawn O. Pearce· Jan 21, 2010, 16:24 UTC · re: Tay Ray Chuan · lore

Re: [PATCH] http/remote-curl: coddle picky servers

Tay Ray Chuan <rctay89@gmail.com> wrote:
Show 15 quoted lines
> On Fri, Jan 22, 2010 at 12:07 AM, Mike Hommey <mh@glandium.org> wrote:
> > Look closely at the start of the requested URL: /gitweb.cgi...
> > It comes from this rule:
> >
> > RewriteCond %{QUERY_STRING} ^(.+)$
> > RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]
> >
> > which is global to the virtual host.
> >
> > Anyways, while git.debian.org can certainly be fixed for that, other
> > servers may want to do some different things with urls with parameters.
> 
> heh, I was suspecting some URL rewriting was going on.
> 
> Is this an issue that should be fixed in gitweb?

I don't see why it should be. gitweb isn't a service CGI. I find it odd that someone would configure their website to route anything with a query string into gitweb. WTF?

-- 
Shawn.
Mike Hommey· Jan 21, 2010, 16:34 UTC · re: Shawn O. Pearce · lore

Re: [PATCH] http/remote-curl: coddle picky servers

On Thu, Jan 21, 2010 at 08:24:02AM -0800, Shawn O. Pearce wrote:
Show 20 quoted lines
> Tay Ray Chuan <rctay89@gmail.com> wrote:
> > On Fri, Jan 22, 2010 at 12:07 AM, Mike Hommey <mh@glandium.org> wrote:
> > > Look closely at the start of the requested URL: /gitweb.cgi...
> > > It comes from this rule:
> > >
> > > RewriteCond %{QUERY_STRING} ^(.+)$
> > > RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]
> > >
> > > which is global to the virtual host.
> > >
> > > Anyways, while git.debian.org can certainly be fixed for that, other
> > > servers may want to do some different things with urls with parameters.
> > 
> > heh, I was suspecting some URL rewriting was going on.
> > 
> > Is this an issue that should be fixed in gitweb?
> 
> I don't see why it should be.  gitweb isn't a service CGI.  I find
> it odd that someone would configure their website to route anything
> with a query string into gitweb.  WTF?

There was a good reason for it, but the implementation was too broad: The main gitweb list (http://git.debian.org/) is made statically, because it is too long for gitweb to create it in a timely fashion.

So while the main page is made to be a static file, when the request has a query string, which means it's not the main gitweb list, the cgi is used.

Except this rule was also used for unrelated urls.
But as I said in my reply to Tay Ray Chuan, it's fixed.
Mike
Mike Hommey· Jan 21, 2010, 16:34 UTC · re: Tay Ray Chuan · lore

Re: [PATCH] http/remote-curl: coddle picky servers

On Fri, Jan 22, 2010 at 12:20:26AM +0800, Tay Ray Chuan wrote:
Show 17 quoted lines
> Hi,
> 
> On Fri, Jan 22, 2010 at 12:07 AM, Mike Hommey <mh@glandium.org> wrote:
> > Look closely at the start of the requested URL: /gitweb.cgi...
> > It comes from this rule:
> >
> > RewriteCond %{QUERY_STRING} ^(.+)$
> > RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]
> >
> > which is global to the virtual host.
> >
> > Anyways, while git.debian.org can certainly be fixed for that, other
> > servers may want to do some different things with urls with parameters.
> 
> heh, I was suspecting some URL rewriting was going on.
> 
> Is this an issue that should be fixed in gitweb?
Nah, that's just an issue with the config at git.debian.org.
It's fixed already.
Mike
Ilari Liusvaara· Jan 21, 2010, 10:35 UTC · re: Tay Ray Chuan · lore

Re: problem cloning via http since v1.6.6-rc0

On Thu, Jan 21, 2010 at 02:47:37PM +0800, Tay Ray Chuan wrote:
Show 11 quoted lines
> Hi,
> 
> On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara
> >
> > (i.e. webserver would ignore the query). This isn't true for git.debian.org.
> > Requesting the latter works (and the data formatting looks sane), but the
> > former is 404. This causes the fetch to fail.
> 
> afaik, putting a "?var1=val1&var2=...." still makes it a normal GET
> request, even if the url requested is just a plain file and not some
> cgi handler that uses those variables/values.

Yes, it is normal GET (POST would be something else). And wheither it is CGI doesn't come into play for request since client decides wheither to send GET or POST and wheither to include query or not.

Query is just technical name for part between ? and # (or end of HTTP URL), and can be present in any type of request that accepts http:// URL.

As said, code expects query part to be ignored if target is regular file but broke when it didn't get ignored.

-Ilari
Tay Ray Chuan· Jan 21, 2010, 11:36 UTC · re: Ilari Liusvaara · lore

Re: problem cloning via http since v1.6.6-rc0

Hi,

On Thu, Jan 21, 2010 at 6:35 PM, Ilari Liusvaara <ilari.liusvaara@elisanet.fi> wrote:

Show 19 quoted lines
> On Thu, Jan 21, 2010 at 02:47:37PM +0800, Tay Ray Chuan wrote:
>> Hi,
>>
>> On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara
>> >
>> > (i.e. webserver would ignore the query). This isn't true for git.debian.org.
>> > Requesting the latter works (and the data formatting looks sane), but the
>> > former is 404. This causes the fetch to fail.
>>
>> afaik, putting a "?var1=val1&var2=...." still makes it a normal GET
>> request, even if the url requested is just a plain file and not some
>> cgi handler that uses those variables/values.
>
> Yes, it is normal GET (POST would be something else). And wheither it is CGI
> doesn't come into play for request since client decides wheither to send GET
> or POST and wheither to include query or not.
>
> Query is just technical name for part between ? and # (or end of HTTP URL),
> and can be present in any type of request that accepts http:// URL.

yes, indeed, I misread your message. Your idea of the query string affecting the server response didn't strike me then.

-- 
Cheers,
Ray Chuan

← back to recent threads