{"thread":{"id":"22319","subject":"problem cloning via http since v1.6.6-rc0","startedAt":"2010-01-21T00:47:56Z","lastAt":"2010-11-03T16:14:05Z","messageCount":28,"participants":["Yaroslav Halchenko","Tay Ray Chuan","Ilari Liusvaara","Shawn O. Pearce","Mike Hommey","Michael S. Tsirkin","Junio C Hamano","Thomas Rast"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"132252","messageId":"20100121004756.GA18213@onerussian.com","threadId":"22319","inReplyTo":null,"subject":"problem cloning via http since v1.6.6-rc0","fromName":"Yaroslav Halchenko","fromEmail":"debian@onerussian.com","sentAt":"2010-01-21T00:47:56Z","receivedAt":"2010-01-21T00:47:56Z","isPatch":false,"sender":{"key":"debian@onerussian.com","avatar":"https://gravatar.com/avatar/9d2f005048a0274a9c26bc47a51f580e2bf631dfedfdb18370f65e08e4250317?d=mp&s=160"},"body":"Dear Git Developers,\n\nSome users of our project started recently to complain that they could not\nclone the repository via http (git:// wasn't a choice due to heavy firewalling)\nand because http:// was used as a protocol to get sources in some distributions\n(e.g. macports).\n\nCloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0.\nI haven't done full bisection since that repository is relatively bulky and\npoor server is quite loaded anyways, so I thought you just would get a clue\nwithout going brute-force. But here are the details: in case of failing\noperation, I immediately get failure:\n\n$> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git\ntrace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\nwarning: templates not found /home/yoh/share/git-core/templates\nInitialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/\ntrace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\ntrace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\ntrace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\ntrace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\nfatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n\non the server, 1.6.3.3 version of git was used to run git\nupdate-server-info.\n\nThanks in advance\n-- \nYaroslav O. Halchenko\nPostdoctoral Fellow, Department of Psychological and Brain Sciences\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834 Fax: +1 (603) 646-1419\nWWW: http://www.linkedin.com/in/yarik \n"},{"id":"132254","messageId":"be6fef0d1001201734s45ef83fcy32e07f8c213cbe2@mail.gmail.com","threadId":"22319","inReplyTo":"20100121004756.GA18213@onerussian.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-01-21T01:34:55Z","receivedAt":"2010-01-21T01:34:55Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, Jan 21, 2010 at 8:47 AM, Yaroslav Halchenko\n wrote:\n> Cloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0.\n\nthis sounds like around the time the smart http protocol was introduced.\n\n> fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n>\n> on the server, 1.6.3.3 version of git was used to run git\n> update-server-info.\n\nhmm, are you using the WebDAV-flavour or the smart http protocol to\nhost the repository?\n\n-- \nCheers,\nRay Chuan\n"},{"id":"132255","messageId":"be6fef0d1001201736g9160306g51949a5f36d83e14@mail.gmail.com","threadId":"22319","inReplyTo":"20100121004756.GA18213@onerussian.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-01-21T01:36:59Z","receivedAt":"2010-01-21T01:36:59Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"On Thu, Jan 21, 2010 at 8:47 AM, Yaroslav Halchenko\n wrote:\n> $> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git\n> trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> warning: templates not found /home/yoh/share/git-core/templates\n> Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/\n> trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n\noh, and by the way, could you also run this again with GIT_CURL_VERBOSE=1?\n\n-- \nCheers,\nRay Chuan\n"},{"id":"132262","messageId":"20100121023310.GB18213@onerussian.com","threadId":"22319","inReplyTo":"be6fef0d1001201736g9160306g51949a5f36d83e14@mail.gmail.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Yaroslav Halchenko","fromEmail":"debian@onerussian.com","sentAt":"2010-01-21T02:33:10Z","receivedAt":"2010-01-21T02:33:10Z","isPatch":false,"sender":{"key":"debian@onerussian.com","avatar":"https://gravatar.com/avatar/9d2f005048a0274a9c26bc47a51f580e2bf631dfedfdb18370f65e08e4250317?d=mp&s=160"},"body":"$> GIT_CURL_VERBOSE=1 GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git \ntrace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\nwarning: templates not found /home/yoh/share/git-core/templates\nInitialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/\ntrace: run_command: 'remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\ntrace: exec: 'git' 'remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\ntrace: exec: 'git-remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\ntrace: run_command: 'git-remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n* Couldn't find host git.debian.org in the .netrc file; using defaults\n* About to connect() to git.debian.org port 80 (#0)\n* Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)\n> GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1\nUser-Agent: git/1.6.6.267.g5b159\nHost: git.debian.org\nAccept: */*\nPragma: no-cache\n\n* The requested URL returned error: 404\n* Closing connection #0\nfatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n\nas for smart vs DAV -- don't see any smart alias handling in apache\nconfiguration (I have/had no clue about some smart http in git, just looked at\napache template and saw smart aliases -- is there smth else to check within\nwebserver config?)\n\nOn Thu, 21 Jan 2010, Tay Ray Chuan wrote:\n\n> On Thu, Jan 21, 2010 at 8:47 AM, Yaroslav Halchenko\n> wrote:\n> > $> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git\n> > trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> > warning: templates not found /home/yoh/share/git-core/templates\n> > Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/\n> > trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> > trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> > trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> > trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> > fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n\n> oh, and by the way, could you also run this again with GIT_CURL_VERBOSE=1?\n-- \nYaroslav O. Halchenko\nPostdoctoral Fellow, Department of Psychological and Brain Sciences\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834 Fax: +1 (603) 646-1419\nWWW: http://www.linkedin.com/in/yarik \n"},{"id":"132268","messageId":"be6fef0d1001202001h4c794166y4bcf01b42f3ea1bb@mail.gmail.com","threadId":"22319","inReplyTo":"20100121023310.GB18213@onerussian.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-01-21T04:01:35Z","receivedAt":"2010-01-21T04:01:35Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, Jan 21, 2010 at 10:33 AM, Yaroslav Halchenko\n wrote:\n> *   Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)\n>> GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1\n> User-Agent: git/1.6.6.267.g5b159\n> Host: git.debian.org\n> Accept: */*\n> Pragma: no-cache\n>\n> * The requested URL returned error: 404\n> * Closing connection #0\n> fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n\nI don't think git's at fault here, as we're getting a 404 Not Found.\nCould you check that the repository (the one the url points to, after\ntaking into any url rewriting) is a bare one, ie. has structure\n\n pkg-exppsy/\n |-pymvpa.git\n |-objects/\n |-info/\n |-refs/\n |-...\n\nrather than the non-bare\n\n pkg-exppsy/\n |-pymvpa.git\n |-.git\n |-objects/\n |-info/\n |-refs/\n |-...\n\n?\n\n> as for smart vs DAV -- don't see any smart alias handling in apache\n> configuration (I have/had no clue about some smart http in git, just looked at\n> apache template and saw smart aliases -- is there smth else to check within\n> webserver config?)\n\nIf that's the case, I don't think it's related to your problem. (Btw,\n\"smart\" refers to the http protocol that git can use to sync your\nrepo, via a CGI program on the server, instead of WebDAV. See\ngit-http-backend(1) for details.)\n\n-- \nCheers,\nRay Chuan\n"},{"id":"132271","messageId":"20100121043836.GC18213@onerussian.com","threadId":"22319","inReplyTo":"be6fef0d1001202001h4c794166y4bcf01b42f3ea1bb@mail.gmail.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Yaroslav Halchenko","fromEmail":"debian@onerussian.com","sentAt":"2010-01-21T04:38:36Z","receivedAt":"2010-01-21T04:38:36Z","isPatch":false,"sender":{"key":"debian@onerussian.com","avatar":"https://gravatar.com/avatar/9d2f005048a0274a9c26bc47a51f580e2bf631dfedfdb18370f65e08e4250317?d=mp&s=160"},"body":"\nOn Thu, 21 Jan 2010, Tay Ray Chuan wrote:\n> > fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n> I don't think git's at fault here, as we're getting a 404 Not Found.\nkhe khe, pardon me, but if git can't talk to its nephew (ie its own\nrepository which was created with earlier version), whenever another\nnephew (ie git of earlier version) can talk to it, I do consider it to\nbe git's fault ;)\n\nlet me zoom in onto difference in communication between two different versions :\n\n* Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)\n> GET /git/pkg-exppsy/pymvpa.git/info/refs HTTP/1.1\nUser-Agent: git/1.6.5\nHost: git.debian.org\nAccept: */*\nPragma: no-cache\n\n< HTTP/1.1 200 OK\n\nwhenever, once again, for 1.6.6 it looked much shorter:\n\n* Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)\n> GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1\nUser-Agent: git/1.6.6.267.g5b159\nHost: git.debian.org\nAccept: */*\nPragma: no-cache\n\n* The requested URL returned error: 404\n* Closing connection #0\nfatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n\n\n> Could you check that the repository (the one the url points to, after\n> taking into any url rewriting) is a bare one, ie. has structure\nyes - it is bare... I was 101% sure (isn't that a convention to have\n.git suffix for directories with bare repositories), but just to make\nsure:\n\n$> cd /srv/git.debian.org/git/pkg-exppsy/pymvpa.git/\ntotal 76\n 8 branches/ 8 config 8 description 8 HEAD 8 hooks/ 8 info/ 8 objects/ 12 packed-refs 8 refs/\n\n\n> If that's the case, I don't think it's related to your problem. (Btw,\n> \"smart\" refers to the http protocol that git can use to sync your\n> repo, via a CGI program on the server, instead of WebDAV. See\n> git-http-backend(1) for details.)\nthanks for info, I did know about this beastie.\n\nI see no references to git-http-backend in apache config -- so indeed\nshould not be the case... but from the git-http-backend description:\n\n,---\n| By default, only the `upload-pack` service is enabled, which serves\n| 'git-fetch-pack' and 'git-ls-remote' clients, which are invoked from\n| 'git-fetch', 'git-pull', and 'git-clone'.\n`---\n\nso, it looks like 1.6.6 for some reason decided to assume that it is \"smart\"\nhttp whenever it is not? is that the case here?\n\n-- \nYaroslav O. Halchenko\nPostdoctoral Fellow, Department of Psychological and Brain Sciences\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834 Fax: +1 (603) 646-1419\nWWW: http://www.linkedin.com/in/yarik \n"},{"id":"132272","messageId":"20100121050850.GA18896@Knoppix","threadId":"22319","inReplyTo":"20100121004756.GA18213@onerussian.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Ilari Liusvaara","fromEmail":"ilari.liusvaara@elisanet.fi","sentAt":"2010-01-21T05:08:51Z","receivedAt":"2010-01-21T05:08:51Z","isPatch":false,"sender":{"key":"ilari.liusvaara@elisanet.fi","avatar":null},"body":"On Wed, Jan 20, 2010 at 07:47:56PM -0500, Yaroslav Halchenko wrote:\n\nAdded spearce to cc.\n\n> Dear Git Developers,\n> \n> Some users of our project started recently to complain that they could not\n> clone the repository via http (git:// wasn't a choice due to heavy firewalling)\n> and because http:// was used as a protocol to get sources in some distributions\n> (e.g. macports).\n> \n> Cloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0.\n> I haven't done full bisection since that repository is relatively bulky and\n> poor server is quite loaded anyways, so I thought you just would get a clue\n> without going brute-force. But here are the details: in case of failing\n> operation, I immediately get failure:\n> \n> $> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git\n> trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> warning: templates not found /home/yoh/share/git-core/templates\n> Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/\n> trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'\n> fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?\n\nLooks like remote-curl (which handles http) issues request for:\n\n'.../info/refs?service=git-upload-pack'\n\nAnd expects that if there is no smart HTTP server there for the request to be\ninterpretted as:\n\n'.../info/refs'\n\n(i.e. webserver would ignore the query). This isn't true for git.debian.org.\nRequesting the latter works (and the data formatting looks sane), but the\nformer is 404. This causes the fetch to fail.\n\n-Ilari\n"},{"id":"132274","messageId":"be6fef0d1001202247l7467a14ap8181eb3ed830167a@mail.gmail.com","threadId":"22319","inReplyTo":"20100121050850.GA18896@Knoppix","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-01-21T06:47:37Z","receivedAt":"2010-01-21T06:47:37Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara\n wrote:\n> Looks like remote-curl (which handles http) issues request for:\n>\n> '.../info/refs?service=git-upload-pack'\n>\n> And expects that if there is no smart HTTP server there for the request to be\n> interpretted as:\n>\n> '.../info/refs'\n>\n> (i.e. webserver would ignore the query). This isn't true for git.debian.org.\n> Requesting the latter works (and the data formatting looks sane), but the\n> former is 404. This causes the fetch to fail.\n\nafaik, putting a \"?var1=val1&var2=....\" still makes it a normal GET\nrequest, even if the url requested is just a plain file and not some\ncgi handler that uses those variables/values.\n\n--\nCheers,\nRay Chuan\n"},{"id":"132278","messageId":"20100121155136.17b59e8f.rctay89@gmail.com","threadId":"22319","inReplyTo":"be6fef0d1001202247l7467a14ap8181eb3ed830167a@mail.gmail.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-01-21T07:51:36Z","receivedAt":"2010-01-21T07:51:36Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, 21 Jan 2010 14:47:37 +0800\nTay Ray Chuan wrote:\n> On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara\n> wrote:\n> > Looks like remote-curl (which handles http) issues request for:\n> >\n> > '.../info/refs?service=git-upload-pack'\n> >\n> > And expects that if there is no smart HTTP server there for the request to be\n> > interpretted as:\n> >\n> > '.../info/refs'\n> >\n> > (i.e. webserver would ignore the query). This isn't true for git.debian.org.\n> > Requesting the latter works (and the data formatting looks sane), but the\n> > former is 404. This causes the fetch to fail.\n> \n> afaik, putting a \"?var1=val1&var2=....\" still makes it a normal GET\n> request, even if the url requested is just a plain file and not some\n> cgi handler that uses those variables/values.\n\nYaroslav, sorry for making you run in circles - it really is git's\nfault (sorta).\n\nIn recent versions of git, we were sending out the GET request for\ninfo/refs with a query string (?serivce=). I'm not sure\nwhy, but your server is not playing nice when the query string is\nappended.\n\nCould you try this patch and see if it solves the issue? I manage to\nclone your repo successfully with it.\n\n-- \nCheers,\nRay Chuan\n\n-->8--\nSubject: [PATCH] http/remote-curl: coddle picky servers\n\nWhen \"info/refs\" is a static file and not behind a CGI handler, some\nservers may not handle a GET request for it with a query string\nappended (eg. \"?foo=bar\") properly.\n\nIf such a request fails, retry it sans the query string, and also\ndiscount the possibility of using the \"smart\" protocol (since no\nservice is specified with \"?service=\").\n\nSigned-off-by: Tay Ray Chuan \n---\n remote-curl.c | 18 ++++++++++++++++--\n 1 files changed, 16 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 1361006..a904164 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -102,7 +102,7 @@ static struct discovery* discover_refs(const char *service)\n \tstruct strbuf buffer = STRBUF_INIT;\n \tstruct discovery *last = last_discovery;\n \tchar *refs_url;\n-\tint http_ret, is_http = 0;\n+\tint http_ret, is_http = 0, proto_git_candidate = 1;\n \n \tif (last && !strcmp(service, last->service))\n \t\treturn last;\n@@ -121,6 +121,19 @@ static struct discovery* discover_refs(const char *service)\n \n \tinit_walker();\n \thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n+\n+\t/* try again with \"plain\" url (no ? or & appended) */\n+\tif (http_ret != HTTP_OK) {\n+\t\tfree(refs_url);\n+\t\tstrbuf_reset(&buffer);\n+\n+\t\tproto_git_candidate = 0;\n+\t\tstrbuf_addf(&buffer, \"%s/info/refs\", url);\n+\t\trefs_url = strbuf_detach(&buffer, NULL);\n+\n+\t\thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n+\t}\n+\n \tswitch (http_ret) {\n \tcase HTTP_OK:\n \t\tbreak;\n@@ -137,7 +150,8 @@ static struct discovery* discover_refs(const char *service)\n \tlast->buf_alloc = strbuf_detach(&buffer, &last->len);\n \tlast->buf = last->buf_alloc;\n \n-\tif (is_http && 5 <= last->len && last->buf[4] == '#') {\n+\tif (is_http && proto_git_candidate\n+\t\t&& 5 <= last->len && last->buf[4] == '#') {\n \t\t/* smart HTTP response; validate that the service\n \t\t * pkt-line matches our request.\n \t\t */\n-- \n1.6.6.1.337.g96bc8\n"},{"id":"132293","messageId":"20100121103500.GA19285@Knoppix","threadId":"22319","inReplyTo":"be6fef0d1001202247l7467a14ap8181eb3ed830167a@mail.gmail.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Ilari Liusvaara","fromEmail":"ilari.liusvaara@elisanet.fi","sentAt":"2010-01-21T10:35:01Z","receivedAt":"2010-01-21T10:35:01Z","isPatch":false,"sender":{"key":"ilari.liusvaara@elisanet.fi","avatar":null},"body":"On Thu, Jan 21, 2010 at 02:47:37PM +0800, Tay Ray Chuan wrote:\n> Hi,\n> \n> On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara\n> >\n> > (i.e. webserver would ignore the query). This isn't true for git.debian.org.\n> > Requesting the latter works (and the data formatting looks sane), but the\n> > former is 404. This causes the fetch to fail.\n> \n> afaik, putting a \"?var1=val1&var2=....\" still makes it a normal GET\n> request, even if the url requested is just a plain file and not some\n> cgi handler that uses those variables/values.\n\nYes, it is normal GET (POST would be something else). And wheither it is CGI\ndoesn't come into play for request since client decides wheither to send GET\nor POST and wheither to include query or not.\n\nQuery is just technical name for part between ? and # (or end of HTTP URL),\nand can be present in any type of request that accepts http:// URL.\n\nAs said, code expects query part to be ignored if target is regular file\nbut broke when it didn't get ignored.\n\n-Ilari\n"},{"id":"132294","messageId":"be6fef0d1001210336i56605a37tfbede92cab794d76@mail.gmail.com","threadId":"22319","inReplyTo":"20100121103500.GA19285@Knoppix","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-01-21T11:36:51Z","receivedAt":"2010-01-21T11:36:51Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, Jan 21, 2010 at 6:35 PM, Ilari Liusvaara\n wrote:\n> On Thu, Jan 21, 2010 at 02:47:37PM +0800, Tay Ray Chuan wrote:\n>> Hi,\n>>\n>> On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara\n>> >\n>> > (i.e. webserver would ignore the query). This isn't true for git.debian.org.\n>> > Requesting the latter works (and the data formatting looks sane), but the\n>> > former is 404. This causes the fetch to fail.\n>>\n>> afaik, putting a \"?var1=val1&var2=....\" still makes it a normal GET\n>> request, even if the url requested is just a plain file and not some\n>> cgi handler that uses those variables/values.\n>\n> Yes, it is normal GET (POST would be something else). And wheither it is CGI\n> doesn't come into play for request since client decides wheither to send GET\n> or POST and wheither to include query or not.\n>\n> Query is just technical name for part between ? and # (or end of HTTP URL),\n> and can be present in any type of request that accepts http:// URL.\n\nyes, indeed, I misread your message. Your idea of the query string\naffecting the server response didn't strike me then.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"132298","messageId":"20100121140054.GH18213@onerussian.com","threadId":"22319","inReplyTo":"20100121155136.17b59e8f.rctay89@gmail.com","subject":"Re: problem cloning via http since v1.6.6-rc0","fromName":"Yaroslav Halchenko","fromEmail":"debian@onerussian.com","sentAt":"2010-01-21T14:00:55Z","receivedAt":"2010-01-21T14:00:55Z","isPatch":false,"sender":{"key":"debian@onerussian.com","avatar":"https://gravatar.com/avatar/9d2f005048a0274a9c26bc47a51f580e2bf631dfedfdb18370f65e08e4250317?d=mp&s=160"},"body":"Hi Tay Ray,\n\nThat patch works fine for me ;) I only hope it would get accepted into\nbugfix and next dev release (I guess it might annoy some of apache\nadmins a bit due to increase of their errors.log now even for well\nmaintained repositories, but well -- that is life ;-) )\n\nThanks!\nYarik\n\nOn Thu, 21 Jan 2010, Tay Ray Chuan wrote:\n> > afaik, putting a \"?var1=val1&var2=....\" still makes it a normal GET\n> > request, even if the url requested is just a plain file and not some\n> > cgi handler that uses those variables/values.\n\n> Yaroslav, sorry for making you run in circles - it really is git's\n> fault (sorta).\n\n> In recent versions of git, we were sending out the GET request for\n> info/refs with a query string (?serivce=). I'm not sure\n> why, but your server is not playing nice when the query string is\n> appended.\n\n> Could you try this patch and see if it solves the issue? I manage to\n> clone your repo successfully with it.\n-- \n .-.\n=------------------------------ /v\\ ----------------------------=\nKeep in touch // \\\\ (yoh@|www.)onerussian.com\nYaroslav Halchenko /( )\\ ICQ#: 60653192\n Linux User ^^-^^ [175555]\n"},{"id":"132300","messageId":"20100121224100.624c9c9d.rctay89@gmail.com","threadId":"22319","inReplyTo":"20100121140054.GH18213@onerussian.com","subject":"[PATCH] http/remote-curl: coddle picky servers","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-01-21T14:41:00Z","receivedAt":"2010-01-21T14:41:00Z","isPatch":true,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"When \"info/refs\" is a static file and not behind a CGI handler, some\nservers may not handle a GET request for it with a query string\nappended (eg. \"?foo=bar\") properly.\n\nIf such a request fails, retry it sans the query string. In addition,\nensure that the \"smart\" http protocol is not used (a service has to be\nspecified with \"?service=\" to be conformant).\n\nSigned-off-by: Tay Ray Chuan \nReported-and-tested-by: Yaroslav Halchenko \n---\n remote-curl.c | 18 ++++++++++++++++--\n 1 files changed, 16 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 1361006..a904164 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -102,7 +102,7 @@ static struct discovery* discover_refs(const char *service)\n \tstruct strbuf buffer = STRBUF_INIT;\n \tstruct discovery *last = last_discovery;\n \tchar *refs_url;\n-\tint http_ret, is_http = 0;\n+\tint http_ret, is_http = 0, proto_git_candidate = 1;\n\n \tif (last && !strcmp(service, last->service))\n \t\treturn last;\n@@ -121,6 +121,19 @@ static struct discovery* discover_refs(const char *service)\n\n \tinit_walker();\n \thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n+\n+\t/* try again with \"plain\" url (no ? or & appended) */\n+\tif (http_ret != HTTP_OK) {\n+\t\tfree(refs_url);\n+\t\tstrbuf_reset(&buffer);\n+\n+\t\tproto_git_candidate = 0;\n+\t\tstrbuf_addf(&buffer, \"%s/info/refs\", url);\n+\t\trefs_url = strbuf_detach(&buffer, NULL);\n+\n+\t\thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n+\t}\n+\n \tswitch (http_ret) {\n \tcase HTTP_OK:\n \t\tbreak;\n@@ -137,7 +150,8 @@ static struct discovery* discover_refs(const char *service)\n \tlast->buf_alloc = strbuf_detach(&buffer, &last->len);\n \tlast->buf = last->buf_alloc;\n\n-\tif (is_http && 5 <= last->len && last->buf[4] == '#') {\n+\tif (is_http && proto_git_candidate\n+\t\t&& 5 <= last->len && last->buf[4] == '#') {\n \t\t/* smart HTTP response; validate that the service\n \t\t * pkt-line matches our request.\n \t\t */\n--\n1.6.6.1.337.g96bc8\n"},{"id":"132301","messageId":"20100121155637.GA19078@spearce.org","threadId":"22319","inReplyTo":"20100121224100.624c9c9d.rctay89@gmail.com","subject":"Re: [PATCH] http/remote-curl: coddle picky servers","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-21T15:56:37Z","receivedAt":"2010-01-21T15:56:37Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Tay Ray Chuan wrote:\n> When \"info/refs\" is a static file and not behind a CGI handler, some\n> servers may not handle a GET request for it with a query string\n> appended (eg. \"?foo=bar\") properly.\n> \n> If such a request fails, retry it sans the query string. In addition,\n> ensure that the \"smart\" http protocol is not used (a service has to be\n> specified with \"?service=\" to be conformant).\n> \n> Signed-off-by: Tay Ray Chuan \n> Reported-and-tested-by: Yaroslav Halchenko \n\n*grumble* stupid Apache *grumble*\n\nAcked-by: Shawn O. Pearce \n\n-- \nShawn.\n"},{"id":"132303","messageId":"20100121160707.GA31276@glandium.org","threadId":"22319","inReplyTo":"20100121155637.GA19078@spearce.org","subject":"Re: [PATCH] http/remote-curl: coddle picky servers","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2010-01-21T16:07:07Z","receivedAt":"2010-01-21T16:07:07Z","isPatch":true,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Thu, Jan 21, 2010 at 07:56:37AM -0800, Shawn O. Pearce wrote:\n> Tay Ray Chuan wrote:\n> > When \"info/refs\" is a static file and not behind a CGI handler, some\n> > servers may not handle a GET request for it with a query string\n> > appended (eg. \"?foo=bar\") properly.\n> > \n> > If such a request fails, retry it sans the query string. In addition,\n> > ensure that the \"smart\" http protocol is not used (a service has to be\n> > specified with \"?service=\" to be conformant).\n> > \n> > Signed-off-by: Tay Ray Chuan \n> > Reported-and-tested-by: Yaroslav Halchenko \n> \n> *grumble* stupid Apache *grumble*\n\nstupid Apache... configuration.\n\nCheck the error message you get on\nhttp://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack:\n\nThe requested URL /gitweb.cgigit/pkg-exppsy/pymvpa.git/info/refs was not\nfound on this server.\n\nLook closely at the start of the requested URL: /gitweb.cgi...\nIt comes from this rule:\n\nRewriteCond %{QUERY_STRING} ^(.+)$\nRewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]\n\nwhich is global to the virtual host.\n\nAnyways, while git.debian.org can certainly be fixed for that, other\nservers may want to do some different things with urls with parameters.\n\nMike\n"},{"id":"132304","messageId":"20100121161016.GA16300@redhat.com","threadId":"22319","inReplyTo":"20100121160707.GA31276@glandium.org","subject":"git fetch -v not at all verbose?","fromName":"Michael S. Tsirkin","fromEmail":"mst@redhat.com","sentAt":"2010-01-21T16:10:17Z","receivedAt":"2010-01-21T16:10:17Z","isPatch":false,"sender":{"key":"mst@kernel.org","avatar":null},"body":"Hi!\nOn many of my trees (with linux kernel), git fetch is slower than git clone.\nEven more annoyingly, it would hang sometimes for tens of minutes without any\noutput, even if -v is supplied.\n\nstracing it shows a ton of lines like the following:\n16324 read(10, \"ACK 4bbdfe65d23014f539fec4227260\"..., 51) = 51\n16324 read(10, \"0037\", 4) = 4\n16324 read(10, \"ACK 322c06560fa314b04a6302ea03c0\"..., 51) = 51\n16324 read(10, \"0037\", 4) = 4\n16324 read(10, \"ACK 848ea2043b128b5947851866a114\"..., 51) = 51\n16324 read(10, \"0037\", 4) = 4\n\nIs there some way to make got fetch show progress at this stage,\nor even better, can it be made faster somehow?\n\nThanks!\n\n-- \nMST\n"},{"id":"132306","messageId":"20100121161858.GC19078@spearce.org","threadId":"22319","inReplyTo":"20100121161016.GA16300@redhat.com","subject":"Re: git fetch -v not at all verbose?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-21T16:18:58Z","receivedAt":"2010-01-21T16:18:58Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Michael S. Tsirkin\" wrote:\n> On many of my trees (with linux kernel), git fetch is slower than git clone.\n> Even more annoyingly, it would hang sometimes for tens of minutes without any\n> output, even if -v is supplied.\n\nOuch. I think -v -v boosts the output to be ever more verbose,\nand might actually show you something.\n \n> stracing it shows a ton of lines like the following:\n> 16324 read(10, \"ACK 4bbdfe65d23014f539fec4227260\"..., 51) = 51\n> 16324 read(10, \"0037\", 4) = 4\n> 16324 read(10, \"ACK 322c06560fa314b04a6302ea03c0\"..., 51) = 51\n> 16324 read(10, \"0037\", 4) = 4\n> 16324 read(10, \"ACK 848ea2043b128b5947851866a114\"..., 51) = 51\n> 16324 read(10, \"0037\", 4) = 4\n\nThat's the peers trying to determine a common base.\n \n> Is there some way to make got fetch show progress at this stage,\n> or even better, can it be made faster somehow?\n\nWe shouldn't need to show progress here, we should just be faster.\n\nGiven the symptom, it sounds to me like your local repository\nis some 1,000s of commits ahead of the remote repository you are\nfetching from. Is that true?\n\nAre you fetching from a configured remote that has tracking branches,\nor are you fetching through a one-shot URL pasted onto the command\nline?\n\n-- \nShawn.\n"},{"id":"132307","messageId":"be6fef0d1001210820u638f5262jaa062a20fdfbc18b@mail.gmail.com","threadId":"22319","inReplyTo":"20100121160707.GA31276@glandium.org","subject":"Re: [PATCH] http/remote-curl: coddle picky servers","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-01-21T16:20:26Z","receivedAt":"2010-01-21T16:20:26Z","isPatch":true,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Fri, Jan 22, 2010 at 12:07 AM, Mike Hommey wrote:\n> Look closely at the start of the requested URL: /gitweb.cgi...\n> It comes from this rule:\n>\n> RewriteCond %{QUERY_STRING} ^(.+)$\n> RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]\n>\n> which is global to the virtual host.\n>\n> Anyways, while git.debian.org can certainly be fixed for that, other\n> servers may want to do some different things with urls with parameters.\n\nheh, I was suspecting some URL rewriting was going on.\n\nIs this an issue that should be fixed in gitweb?\n\n(added John 'Warthog9' Hawley to the Cc list, perhaps he might know.)\n\n-- \nCheers,\nRay Chuan\n"},{"id":"132308","messageId":"20100121162402.GD19078@spearce.org","threadId":"22319","inReplyTo":"be6fef0d1001210820u638f5262jaa062a20fdfbc18b@mail.gmail.com","subject":"Re: [PATCH] http/remote-curl: coddle picky servers","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-21T16:24:02Z","receivedAt":"2010-01-21T16:24:02Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Tay Ray Chuan wrote:\n> On Fri, Jan 22, 2010 at 12:07 AM, Mike Hommey wrote:\n> > Look closely at the start of the requested URL: /gitweb.cgi...\n> > It comes from this rule:\n> >\n> > RewriteCond %{QUERY_STRING} ^(.+)$\n> > RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]\n> >\n> > which is global to the virtual host.\n> >\n> > Anyways, while git.debian.org can certainly be fixed for that, other\n> > servers may want to do some different things with urls with parameters.\n> \n> heh, I was suspecting some URL rewriting was going on.\n> \n> Is this an issue that should be fixed in gitweb?\n\nI don't see why it should be. gitweb isn't a service CGI. I find\nit odd that someone would configure their website to route anything\nwith a query string into gitweb. WTF?\n\n-- \nShawn.\n"},{"id":"132311","messageId":"20100121163419.GA31659@glandium.org","threadId":"22319","inReplyTo":"be6fef0d1001210820u638f5262jaa062a20fdfbc18b@mail.gmail.com","subject":"Re: [PATCH] http/remote-curl: coddle picky servers","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2010-01-21T16:34:19Z","receivedAt":"2010-01-21T16:34:19Z","isPatch":true,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, Jan 22, 2010 at 12:20:26AM +0800, Tay Ray Chuan wrote:\n> Hi,\n> \n> On Fri, Jan 22, 2010 at 12:07 AM, Mike Hommey wrote:\n> > Look closely at the start of the requested URL: /gitweb.cgi...\n> > It comes from this rule:\n> >\n> > RewriteCond %{QUERY_STRING} ^(.+)$\n> > RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]\n> >\n> > which is global to the virtual host.\n> >\n> > Anyways, while git.debian.org can certainly be fixed for that, other\n> > servers may want to do some different things with urls with parameters.\n> \n> heh, I was suspecting some URL rewriting was going on.\n> \n> Is this an issue that should be fixed in gitweb?\n\nNah, that's just an issue with the config at git.debian.org.\n\nIt's fixed already.\n\nMike\n"},{"id":"132310","messageId":"20100121163425.GB31659@glandium.org","threadId":"22319","inReplyTo":"20100121162402.GD19078@spearce.org","subject":"Re: [PATCH] http/remote-curl: coddle picky servers","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2010-01-21T16:34:25Z","receivedAt":"2010-01-21T16:34:25Z","isPatch":true,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Thu, Jan 21, 2010 at 08:24:02AM -0800, Shawn O. Pearce wrote:\n> Tay Ray Chuan wrote:\n> > On Fri, Jan 22, 2010 at 12:07 AM, Mike Hommey wrote:\n> > > Look closely at the start of the requested URL: /gitweb.cgi...\n> > > It comes from this rule:\n> > >\n> > > RewriteCond %{QUERY_STRING} ^(.+)$\n> > > RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]\n> > >\n> > > which is global to the virtual host.\n> > >\n> > > Anyways, while git.debian.org can certainly be fixed for that, other\n> > > servers may want to do some different things with urls with parameters.\n> > \n> > heh, I was suspecting some URL rewriting was going on.\n> > \n> > Is this an issue that should be fixed in gitweb?\n> \n> I don't see why it should be. gitweb isn't a service CGI. I find\n> it odd that someone would configure their website to route anything\n> with a query string into gitweb. WTF?\n\nThere was a good reason for it, but the implementation was too broad:\nThe main gitweb list (http://git.debian.org/) is made statically,\nbecause it is too long for gitweb to create it in a timely fashion.\n\nSo while the main page is made to be a static file, when the request has\na query string, which means it's not the main gitweb list, the cgi is\nused.\n\nExcept this rule was also used for unrelated urls.\n\nBut as I said in my reply to Tay Ray Chuan, it's fixed.\n\nMike\n"},{"id":"132313","messageId":"20100121163518.GA16466@redhat.com","threadId":"22319","inReplyTo":"20100121161858.GC19078@spearce.org","subject":"Re: git fetch -v not at all verbose?","fromName":"Michael S. Tsirkin","fromEmail":"mst@redhat.com","sentAt":"2010-01-21T16:35:18Z","receivedAt":"2010-01-21T16:35:18Z","isPatch":false,"sender":{"key":"mst@kernel.org","avatar":null},"body":"On Thu, Jan 21, 2010 at 08:18:58AM -0800, Shawn O. Pearce wrote:\n> \"Michael S. Tsirkin\" wrote:\n> > On many of my trees (with linux kernel), git fetch is slower than git clone.\n> > Even more annoyingly, it would hang sometimes for tens of minutes without any\n> > output, even if -v is supplied.\n> \n> Ouch. I think -v -v boosts the output to be ever more verbose,\n> and might actually show you something.\n> \n> > stracing it shows a ton of lines like the following:\n> > 16324 read(10, \"ACK 4bbdfe65d23014f539fec4227260\"..., 51) = 51\n> > 16324 read(10, \"0037\", 4) = 4\n> > 16324 read(10, \"ACK 322c06560fa314b04a6302ea03c0\"..., 51) = 51\n> > 16324 read(10, \"0037\", 4) = 4\n> > 16324 read(10, \"ACK 848ea2043b128b5947851866a114\"..., 51) = 51\n> > 16324 read(10, \"0037\", 4) = 4\n> \n> That's the peers trying to determine a common base.\n> \n> > Is there some way to make got fetch show progress at this stage,\n> > or even better, can it be made faster somehow?\n> \n> We shouldn't need to show progress here, we should just be faster.\n> \n> Given the symptom, it sounds to me like your local repository\n> is some 1,000s of commits ahead of the remote repository you are\n> fetching from. Is that true?\n\nHmm, no, but what is true is that I fetched several remotes\nthat diverged significantly into the same local repository.\nWould that have same effect?\n\n> Are you fetching from a configured remote that has tracking branches,\n> or are you fetching through a one-shot URL pasted onto the command\n> line?\n\nConfigured remote.\n\n> -- \n> Shawn.\n"},{"id":"132314","messageId":"20100121165737.GG19078@spearce.org","threadId":"22319","inReplyTo":"20100121163518.GA16466@redhat.com","subject":"Re: git fetch -v not at all verbose?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-21T16:57:37Z","receivedAt":"2010-01-21T16:57:37Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Michael S. Tsirkin\" wrote:\n> On Thu, Jan 21, 2010 at 08:18:58AM -0800, Shawn O. Pearce wrote:\n> > \"Michael S. Tsirkin\" wrote:\n> > > On many of my trees (with linux kernel), git fetch is slower than git clone.\n> > > Even more annoyingly, it would hang sometimes for tens of minutes without any\n> > > output, even if -v is supplied.\n...\n> > Given the symptom, it sounds to me like your local repository\n> > is some 1,000s of commits ahead of the remote repository you are\n> > fetching from. Is that true?\n> \n> Hmm, no, but what is true is that I fetched several remotes\n> that diverged significantly into the same local repository.\n> Would that have same effect?\n\nYes.\n\n> > Are you fetching from a configured remote that has tracking branches,\n> > or are you fetching through a one-shot URL pasted onto the command\n> > line?\n> \n> Configured remote.\n\nHmm. I wonder if we should try to shortcut the commit walking in\na case like this and just feed the tracking branches we already have.\n\n-- \nShawn.\n"},{"id":"132317","messageId":"20100121173010.GB16707@redhat.com","threadId":"22319","inReplyTo":"20100121165737.GG19078@spearce.org","subject":"Re: git fetch -v not at all verbose?","fromName":"Michael S. Tsirkin","fromEmail":"mst@redhat.com","sentAt":"2010-01-21T17:30:10Z","receivedAt":"2010-01-21T17:30:10Z","isPatch":false,"sender":{"key":"mst@kernel.org","avatar":null},"body":"On Thu, Jan 21, 2010 at 08:57:37AM -0800, Shawn O. Pearce wrote:\n> \"Michael S. Tsirkin\" wrote:\n> > On Thu, Jan 21, 2010 at 08:18:58AM -0800, Shawn O. Pearce wrote:\n> > > \"Michael S. Tsirkin\" wrote:\n> > > > On many of my trees (with linux kernel), git fetch is slower than git clone.\n> > > > Even more annoyingly, it would hang sometimes for tens of minutes without any\n> > > > output, even if -v is supplied.\n> ...\n> > > Given the symptom, it sounds to me like your local repository\n> > > is some 1,000s of commits ahead of the remote repository you are\n> > > fetching from. Is that true?\n> > \n> > Hmm, no, but what is true is that I fetched several remotes\n> > that diverged significantly into the same local repository.\n> > Would that have same effect?\n> \n> Yes.\n> \n> > > Are you fetching from a configured remote that has tracking branches,\n> > > or are you fetching through a one-shot URL pasted onto the command\n> > > line?\n> > \n> > Configured remote.\n> \n> Hmm. I wonder if we should try to shortcut the commit walking in\n> a case like this and just feed the tracking branches we already have.\n\nOr for the case of 1,000s of commits ahead, git could try to implement a\nheuristic to reduce the number of commits sent. Currently all commits\nare sent in order, correct? How about binary search like what git\nbisect does?\n\n> -- \n> Shawn.\n"},{"id":"132318","messageId":"7v8wbrtkvn.fsf@alter.siamese.dyndns.org","threadId":"22319","inReplyTo":"20100121165737.GG19078@spearce.org","subject":"Re: git fetch -v not at all verbose?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-21T17:42:36Z","receivedAt":"2010-01-21T17:42:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n>> > Are you fetching from a configured remote that has tracking branches,\n>> > or are you fetching through a one-shot URL pasted onto the command\n>> > line?\n>> \n>> Configured remote.\n>\n> Hmm. I wonder if we should try to shortcut the commit walking in\n> a case like this and just feed the tracking branches we already have.\n\nYou mean that the main culprit is the presense of thousdands of commits\nthat fetcher has obtained through the other remotes (and his own) that the\nuploader makes fetcher walk all the way, in the false hope that there\nmight be a commit among them that is closer to the commits being fetched\nthan the ones at the tip of tracking branch the fetcher has for this\nuploader currently?\n\nAnd the solution might be to tell only about the tips of remote tracking\nbranches fetcher has obtained from this particular uploader, not about\nother remote tracking bracnesh it got from others or his own local\nbranches (which may have merged from other remotes)?\n\nIt is a clever idea but I suspect it may not work well in practice. For\nexample, suppose a project is two-tier, say, with top-level and subsystem\nrepositories, the former of which regularly merge from the latter, and you\nare a participant primarily working on the subsystem. You fetch daily\nfrom the subsystem repository, but weekly from the top-level.\n\nNow, when you fetch from the top-level, the remote tracking refs you have\nfor it are much more stale than your other refs. The top-level would have\nacquired a lot more commits from the same subsystem repository since you\nfetched from there the last time, and you already have many of them\nthrough your daily fetch from the subsystem repository. To minimize the\ntransfer in such a case, the fetcher does want to tell the uploader that\nit has those commits from the same subsystem repository, so that the\ncommit walker can stop at a recent merge into the top-level from the\nsubsystem repository.\n\nThere was a discussion about updating the commit walk exchange to bisect\nthe history (skip and try a much older one to see if it is reachable, but\nto avoid overshooting, step back and see if a newer one is still common).\nIt would be a lot more work and needs to be implemented as a new protocol\ncapability, but I think it is the right way to go in the longer term.\n"},{"id":"132319","messageId":"201001211847.40003.trast@student.ethz.ch","threadId":"22319","inReplyTo":"20100121173010.GB16707@redhat.com","subject":"Re: git fetch -v not at all verbose?","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-01-21T17:47:39Z","receivedAt":"2010-01-21T17:47:39Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"On Thursday 21 January 2010 18:30:10 Michael S. Tsirkin wrote:\n> On Thu, Jan 21, 2010 at 08:57:37AM -0800, Shawn O. Pearce wrote:\n> > \"Michael S. Tsirkin\" wrote:\n> > > Hmm, no, but what is true is that I fetched several remotes\n> > > that diverged significantly into the same local repository.\n> > > Would that have same effect?\n[...]\n> > Hmm. I wonder if we should try to shortcut the commit walking in\n> > a case like this and just feed the tracking branches we already have.\n> \n> Or for the case of 1,000s of commits ahead, git could try to implement a\n> heuristic to reduce the number of commits sent. Currently all commits\n> are sent in order, correct? How about binary search like what git\n> bisect does?\n\nI had a patch for this ages ago (that combines exponential-stride\nbackwards search and later bisection), but it was shot down on grounds\nof not working at times and code convolution and I forgot about it...\n\nI can give this another shot, but it seems most of the code has moved\ndue to the transport handlers changes, so I'll first have to read into\nit again.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"155040","messageId":"20101103095249.GA9144@redhat.com","threadId":"22319","inReplyTo":"7v8wbrtkvn.fsf@alter.siamese.dyndns.org","subject":"Re: git fetch -v not at all verbose?","fromName":"Michael S. Tsirkin","fromEmail":"mst@redhat.com","sentAt":"2010-11-03T09:52:49Z","receivedAt":"2010-11-03T09:52:49Z","isPatch":false,"sender":{"key":"mst@kernel.org","avatar":null},"body":"On Thu, Jan 21, 2010 at 09:42:36AM -0800, Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> >> > Are you fetching from a configured remote that has tracking branches,\n> >> > or are you fetching through a one-shot URL pasted onto the command\n> >> > line?\n> >> \n> >> Configured remote.\n> >\n> > Hmm. I wonder if we should try to shortcut the commit walking in\n> > a case like this and just feed the tracking branches we already have.\n> \n> You mean that the main culprit is the presense of thousdands of commits\n> that fetcher has obtained through the other remotes (and his own) that the\n> uploader makes fetcher walk all the way, in the false hope that there\n> might be a commit among them that is closer to the commits being fetched\n> than the ones at the tip of tracking branch the fetcher has for this\n> uploader currently?\n> \n> And the solution might be to tell only about the tips of remote tracking\n> branches fetcher has obtained from this particular uploader, not about\n> other remote tracking bracnesh it got from others or his own local\n> branches (which may have merged from other remotes)?\n> \n> It is a clever idea but I suspect it may not work well in practice. For\n> example, suppose a project is two-tier, say, with top-level and subsystem\n> repositories, the former of which regularly merge from the latter, and you\n> are a participant primarily working on the subsystem. You fetch daily\n> from the subsystem repository, but weekly from the top-level.\n> \n> Now, when you fetch from the top-level, the remote tracking refs you have\n> for it are much more stale than your other refs. The top-level would have\n> acquired a lot more commits from the same subsystem repository since you\n> fetched from there the last time, and you already have many of them\n> through your daily fetch from the subsystem repository. To minimize the\n> transfer in such a case, the fetcher does want to tell the uploader that\n> it has those commits from the same subsystem repository, so that the\n> commit walker can stop at a recent merge into the top-level from the\n> subsystem repository.\n> \n> There was a discussion about updating the commit walk exchange to bisect\n> the history (skip and try a much older one to see if it is reachable, but\n> to avoid overshooting, step back and see if a newer one is still common).\n> It would be a lot more work and needs to be implemented as a new protocol\n> capability, but I think it is the right way to go in the longer term.\n\nI thought about this some more: it seems that nothing in\npack-protocol.txt dictates that client has to send have\nlines in order. The whole logic would be on client side.\n\nSo a new capability will be there just in case we find a use for a\nserver-side optimization later on, we don't need the client to behave\ndifferently in any way when this capability is enabled/disabled.\nRight?\n\n-- \nMST\n"},{"id":"155052","messageId":"7vwrouxu42.fsf@alter.siamese.dyndns.org","threadId":"22319","inReplyTo":"20101103095249.GA9144@redhat.com","subject":"Re: git fetch -v not at all verbose?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-11-03T16:14:05Z","receivedAt":"2010-11-03T16:14:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Michael S. Tsirkin\" writes:\n\n> On Thu, Jan 21, 2010 at 09:42:36AM -0800, Junio C Hamano wrote:\n> ...\n>> There was a discussion about updating the commit walk exchange to bisect\n>> the history (skip and try a much older one to see if it is reachable, but\n>> to avoid overshooting, step back and see if a newer one is still common).\n>> It would be a lot more work and needs to be implemented as a new protocol\n>> capability, but I think it is the right way to go in the longer term.\n>\n> I thought about this some more: it seems that nothing in\n> pack-protocol.txt dictates that client has to send have\n> lines in order. The whole logic would be on client side.\n\nThe current protocol may not require any order for it to function\ncorrectly in the sense that the sent pack will contain everything that is\nnecessary, but it does require that commits on a lineage to be sent from\nnear tip to near root if you want to have a _good_ common ancestor to be\nfound.\n\nIf the downloader sends older commits first without sending some new ones,\nthe uploader can say \"Ok, I know about that old one you told me you have,\nso we could use that as a common commit\" [*1*]. But there is no way for\nit to continue the sentence with \"... but I cannot tell if other ones you\ntold me you have that I know nothing about are all directly connected to\nthat common one we just found (in which case that common one is the best\nwe can do), or you have newer ones than the common commit that I also have\nbut you omitted from the listing (in other words, if you didn't omit them,\nwe could have found a better common commit). Could you please back up a\nbit and let us see if we can do better with newer ones?\" with the current\nprotocol exchange.\n\nThe downloader _could_, upon seeing an ACK to a commit that is an ancestor\nof commits that it skipped, try sending these skipped commits, without\ntelling the uploader that it what it is doing. But the uploader will\nunilaterally decide when it thinks it has heard enough, after giving an\nACK back in the original protocol, or after finding enough common\nancestors to cover all the tips requested with WANTs, so I suspect that\nyou may not have a chance to play such a game without an explicit protocol\nextension.\n\n\n[Footnote]\n\n*1* That is what an ACK means. In an multi-ack exchange, it also tells\nthe downloader there is no point to give any ancestors of that commit, but\nallows the downloader to continue sending commits from other lineage.\n"}]}