threads / discuss / 2370

Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

Subject: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

## tl;dr

12 messages between Nov 6, 2005 and Nov 9, 2005.

replies: 11people: 5as markdown or json

Martin Langhoff· Nov 6, 2005, 21:54 UTC · lore
Strange!
I'm getting errors when cloning over http

git-clone http://locke.catalyst.net.nz/git/moodle.git mdlfoo (...) error: (curl_result = 3601440, http_code = 200, sha1 = f04241b142edfbf28fff2babb426cbab5b44e26b) Getting pack list error: Getting alternates list error: Unable to find f04241b142edfbf28fff2babb426cbab5b44e26b under http://locke.catalyst.net.nz/git/moodle.git/

Cannot obtain needed commit f04241b142edfbf28fff2babb426cbab5b44e26b while processing commit 0965f28d4d75f324b86c8f7490830fea471c65c5.

This commit object is easily reachable at http://mirrors.catalyst.net.nz/git/moodle.git/objects/f0/4241b142edfbf28fff2babb426cbab5b44e26b

If I use cg-clone, I get a similar error

cg-clone http://locke.catalyst.net.nz/git/moodle.git#mdl-artena-tairawhiti mdlfooo (...) Cannot obtain needed object 214e6374d49e6d014f0ba6f159d585a3fe468909 while processing commit 0000000000000000000000000000000000000000. cg-fetch: objects fetch failed cg-clone: fetch failed

This commit object seems to be in a pack: http://mirrors.catalyst.net.nz/git/moodle.git/objects/pack/pack-094560c0177ad659a6e172739c4be53da749e5f0.pack

git-cat-file on the server works correctly, and cloning/working over git+ssh works too.

cheers,
martin
Martin Langhoff· Nov 7, 2005, 02:52 UTC · re: Martin Langhoff · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

On 11/7/05, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> Strange!
>
> I'm getting errors when cloning over http
More info on this.

git-fetch-pack (invoked by cg-fetch) bails out because it thinks it got a 404 fetching one of the packs:

Getting pack 9cbe4a5eb777d4ee535f08feb471e812208ed3a5
 which contains 7004cdf821ab5ddcded7819dea34015b0e84cd9a
error: Unable to get pack file
http://locke.catalyst.net.nz/git/moodle.git//objects/pack/pack-9cbe4a5eb777d4ee535f08feb471e812208ed3a5.pack
The requested URL returned error: 404

However, the url is reachable via http (tested with curl and wget) and Apache records the transaction as a 200 OK -- there's no 404 there! (There are 404s, of course, for objects that are in the pack but not for any pack). Trying to read http-fetch.c and http-pull.c to figure out where we could get the return status wrong, but my C is just nonexistant. Hints appreciated.

cheers,
martin
Nick Hengeveld· Nov 7, 2005, 04:37 UTC · re: Martin Langhoff · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

On Mon, Nov 07, 2005 at 03:52:41PM +1300, Martin Langhoff wrote:
Show 6 quoted lines
> > I'm getting errors when cloning over http
> 
> More info on this.
> 
> git-fetch-pack (invoked by cg-fetch) bails out because it thinks it
> got a 404 fetching one of the packs:
Does it always error out on the same pack file?
Which build of git are you using, and which version of curl?
-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.
Martin Langhoff· Nov 7, 2005, 04:50 UTC · re: Nick Hengeveld · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

On 11/7/05, Nick Hengeveld <nickh@reactrix.com> wrote:
Show 10 quoted lines
> On Mon, Nov 07, 2005 at 03:52:41PM +1300, Martin Langhoff wrote:
>
> > > I'm getting errors when cloning over http
> >
> > More info on this.
> >
> > git-fetch-pack (invoked by cg-fetch) bails out because it thinks it
> > got a 404 fetching one of the packs:
>
> Does it always error out on the same pack file?
Yes.
> Which build of git are you using, and which version of curl?

A mix of 0.99.9, 0.99.9{b,c,d} and today's. Tested with Johannes' patch of today for curl message handling, thinking it was related. No dice.

On the Debian x86 box, it's
$ curl -V
curl 7.13.2 (i386-pc-linux-gnu) libcurl/7.14.0 OpenSSL/0.9.7g
zlib/1.2.2 libidn/0.5.18
Protocols: ftp gopher telnet dict ldap http file https ftps
Features: IDN IPv6 Largefile NTLM SSL libz

On the MacOSX box, it's $ curl -V curl 7.10.2 (powerpc-apple-darwin7.0) libcurl/7.10.2 OpenSSL/0.9.7g zlib/1.1.4

cheers,
martin
Nick Hengeveld· Nov 7, 2005, 17:14 UTC · re: Martin Langhoff · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

On Mon, Nov 07, 2005 at 05:50:02PM +1300, Martin Langhoff wrote:
> > Does it always error out on the same pack file?
> 
> Yes.

You might try this to see exactly what request/response headers curl thinks are passing back and forth.

diff --git a/http-fetch.c b/http-fetch.c
index ea8af1b..11d4dca 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -325,6 +325,7 @@ static struct active_request_slot *get_a
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_range_header);
 	curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);
+	curl_easy_setopt(slot->curl, CURLOPT_VERBOSE, 1);
 
 	return slot;
 }

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.
Martin Langhoff· Nov 8, 2005, 02:37 UTC · re: Nick Hengeveld · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

On 11/8/05, Nick Hengeveld <nickh@reactrix.com> wrote:
> You might try this to see exactly what request/response headers
> curl thinks are passing back and forth.

It definitely looks like it's doing a few requests in parallel and getting them mixed up. BTW, this repo is public and sitting on a box that doubles up as kernel mirror -- feel free to hit it ;-)

> GET /git/moodle.git/objects/f0/6a06d24eee0b7819e2aaf48ad0e255301394e0 HTTP/1.1
Host: locke.catalyst.net.nz
Accept: */*
* Connected to locke.catalyst.net.nz (202.78.240.39) port 80
* Couldn't find host locke.catalyst.net.nz in the .netrc file, using defaults
* About to connect() to locke.catalyst.net.nz port 80
*   Trying 202.78.240.39... > GET
/git/moodle.git/objects/d9/6d5ee03a225ab4e750fb864dbea35d42c51b8b
HTTP/1.1
Host: locke.catalyst.net.nz
Accept: */*
* Connected to locke.catalyst.net.nz (202.78.240.39) port 80
* The requested URL returned error: 404
* Closing connection #0
* The requested URL returned error: 404
* Closing connection #0
> GET /git/moodle.git/objects/5e/0e0d41f781d53344fa67b5e5a0138b586e2946 HTTP/1.1
Host: locke.catalyst.net.nz
Accept: */*
* The requested URL returned error: 404
* Closing connection #0
* The requested URL returned error: 404
* Closing connection #0
error: Unable to get pack file
http://locke.catalyst.net.nz/git/moodle.git//objects/pack/pack-9cbe4a5eb777d4ee535f08feb471e812208ed3a5.pack
The requested URL returned error: 404
error: Unable to find 7004cdf821ab5ddcded7819dea34015b0e84cd9a under
http://locke.catalyst.net.nz/git/moodle.git/

Cannot obtain needed blob 7004cdf821ab5ddcded7819dea34015b0e84cd9a while processing commit b065a5cb7f757dd6e271249cb49e19e8c34b26ce. cg-fetch: objects fetch failed cg-clone: fetch failed

cheers,
martin
Nick Hengeveld· Nov 8, 2005, 06:31 UTC · re: Martin Langhoff · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

On Tue, Nov 08, 2005 at 03:37:29PM +1300, Martin Langhoff wrote:
> It definitely looks like it's doing a few requests in parallel and
> getting them mixed up. BTW, this repo is public and sitting on a box
> that doubles up as kernel mirror -- feel free to hit it ;-)

Can you run git-http-fetch after the failure to attempt to pick up where it left off? It should fail right away, and hopefully contain less confusing output from parallel requests.

Was there a request header for pack-9cbe...d3a5.pack earlier in the output, and were there response headers for successful or failed requests?

FWIW, I've tried cloning that repository a few times and haven't seen the problem yet.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.
Johannes Schindelin· Nov 8, 2005, 11:54 UTC · re: Martin Langhoff · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

Hi,
On Tue, 8 Nov 2005, Martin Langhoff wrote:
Show 8 quoted lines
> On 11/8/05, Nick Hengeveld <nickh@reactrix.com> wrote:
> 
> > You might try this to see exactly what request/response headers
> > curl thinks are passing back and forth.
> 
> It definitely looks like it's doing a few requests in parallel and
> getting them mixed up. BTW, this repo is public and sitting on a box
> that doubles up as kernel mirror -- feel free to hit it ;-)

It could be a similar problem to what I experienced in http-push: Uninintialized memory due to the code path. Can you run

	valgrind --leak-check=yes --show-reachable=yes git-http-fetch...
and check for jump depending on uninitialized memory?

Hth, Dscho

Ben Clifford· Nov 8, 2005, 11:19 UTC · re: Martin Langhoff · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

I got similar today; www.hawaga.org.uk is some apache server on linux, and piva.hawaga.org.uk, the client Mac OS X.

Feel free to hit the URL yourselves -- its a public webserver.
According to system profiler on piva:
   System Version:    Mac OS X 10.4.2 (8D37)
   Kernel Version:    Darwin 8.2.1
and
!582 [1] benc@piva:~/tmp$ curl --version
curl 7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7g  
zlib/1.2.3
Protocols: ftp gopher telnet dict ldap http file https ftps
Features: IPv6 Largefile NTLM SSL libz
!583 [1] benc@piva:~/tmp$

This is on an unpacked repository, so I would guess that none of the GETs for objects should be failing...

Note that it *doesn't* always error out on the same object:
!549 [0] benc@piva:~/tmp$ cg clone http://www.hawaga.org.uk/ 
gitcompletion.gitdefaulting to local storage area
20:56:33 URL:http://www.hawaga.org.uk/gitcompletion.git/HEAD [41/41] - 
 > "refs/heads/.origin-fetching" [1]
progress: 11 objects, 2875 bytes
error:  (curl_result = 3181280, http_code = 200, sha1 =  
c99aa418704f576aad8249d042cd6afecf38afc4)
Getting pack list
error:
Getting alternates list
error: Unable to find c99aa418704f576aad8249d042cd6afecf38afc4 under  
http://www.hawaga.org.uk/gitcompletion.git/

Cannot obtain needed blob c99aa418704f576aad8249d042cd6afecf38afc4 while processing commit 912eb2baf57b70178c2e1f10df12eae911e1d748. cg-fetch: objects fetch failed cg-clone: fetch failed !550 [0] benc@piva:~/tmp$

!556 [0] benc@piva:~/tmp$ cg clone http://www.hawaga.org.uk/ 
gitcompletion.git
defaulting to local storage area
20:58:48 URL:http://www.hawaga.org.uk/gitcompletion.git/HEAD [41/41] - 
 > "refs/heads/.origin-fetching" [1]
progress: 12 objects, 4581 bytes
error:  (curl_result = 3179088, http_code = 200, sha1 =  
682cfbd4d75e204fada2cabe9e0f040e522ce61d)
Getting pack list
Getting alternates list
error: Unable to find 682cfbd4d75e204fada2cabe9e0f040e522ce61d under  
http://www.hawaga.org.uk/gitcompletion.git/

Cannot obtain needed commit 682cfbd4d75e204fada2cabe9e0f040e522ce61d while processing commit 912eb2baf57b70178c2e1f10df12eae911e1d748. Waiting for http://www.hawaga.org.uk/gitcompletion.git/objects/cb/ c8285a736142b83894ec412fe413d9ee7363ab cg-fetch: objects fetch failed cg-clone: fetch failed !557 [0] benc@piva:~/tmp$

When I run it lots of times in succession, it gets between 10 and 15 objects in before failing on something.

When I try with plain git:

!580 [1] benc@piva:~/tmp$ git clone http://www.hawaga.org.uk/ gitcompletion.git here defaulting to local storage area got a108bdc110dad770ec5c092759a8bc511790d21f walk a108bdc110dad770ec5c092759a8bc511790d21f got 912eb2baf57b70178c2e1f10df12eae911e1d748 got 261466f44dce8fb3cac4e0d0f1bf5a46fa84e07e walk 912eb2baf57b70178c2e1f10df12eae911e1d748 got f766b276eff3ec52ac3c0425b13c936d87c607ee got e05988e4a193f9eac2862ce6e67b19e3785bd8f1 got c99aa418704f576aad8249d042cd6afecf38afc4 got d743b178b82e9a8f10a1bf267d32d070e36c8046 got 96a0d6b6b182d978c04290f7c87049a5d3cc41b7 got 676bd846066cf3794796cadb258a23a76744819c got 2e04450bdc81b1380a8f75223d8ce8019400dd70 error: (curl_result = 3179232, http_code = 200, sha1 = 682cfbd4d75e204fada2cabe9e0f040e522ce61d) Getting pack list error: Getting alternates list error: Unable to find 682cfbd4d75e204fada2cabe9e0f040e522ce61d under http://www.hawaga.org.uk/gitcompletion.git/

Cannot obtain needed commit 682cfbd4d75e204fada2cabe9e0f040e522ce61d while processing commit 912eb2baf57b70178c2e1f10df12eae911e1d748. !581 [1] benc@piva:~/tmp$

but a curl by hand for one of the object URLs works:
!581 [1] benc@piva:~/tmp$ curl http://www.hawaga.org.uk/ 
gitcompletion.git/objects/68/2cfbd4d75e204fada2cabe9e0f040e522ce61d  
 > /dev/null
   % Total    % Received % Xferd  Average Speed   Time    Time      
Time  Current
                                  Dload  Upload   Total   Spent     
Left  Speed
100   250  100   250    0     0    183      0  0:00:01  0:00:01  
--:--:--     0
!582 [1] benc@piva:~/tmp$
and works each time when I run it repeatedly.
On 7 Nov 2005, at 08:54, Martin Langhoff wrote:
Show 50 quoted lines
> Strange!
>
> I'm getting errors when cloning over http
>
> git-clone http://locke.catalyst.net.nz/git/moodle.git mdlfoo
> (...)
> error:  (curl_result = 3601440, http_code = 200, sha1 =
> f04241b142edfbf28fff2babb426cbab5b44e26b)
> Getting pack list
> error:
> Getting alternates list
> error: Unable to find f04241b142edfbf28fff2babb426cbab5b44e26b under
> http://locke.catalyst.net.nz/git/moodle.git/
>
> Cannot obtain needed commit f04241b142edfbf28fff2babb426cbab5b44e26b
> while processing commit 0965f28d4d75f324b86c8f7490830fea471c65c5.
>
> This commit object is easily reachable at
> http://mirrors.catalyst.net.nz/git/moodle.git/objects/ 
> f0/4241b142edfbf28fff2babb426cbab5b44e26b
>
> If I use cg-clone, I get a similar error
>
> cg-clone http://locke.catalyst.net.nz/git/moodle.git#mdl-artena- 
> tairawhiti
> mdlfooo
> (...)
> Cannot obtain needed object 214e6374d49e6d014f0ba6f159d585a3fe468909
> while processing commit 0000000000000000000000000000000000000000.
> cg-fetch: objects fetch failed
> cg-clone: fetch failed
>
> This commit object seems to be in a pack:
> http://mirrors.catalyst.net.nz/git/moodle.git/objects/pack/ 
> pack-094560c0177ad659a6e172739c4be53da749e5f0.pack
>
> git-cat-file on the server works correctly, and cloning/working over
> git+ssh works too.
>
> cheers,
>
>
> martin
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
-- 
Ben ベン Бэн
http://www.hawaga.org.uk/ben/
Nick Hengeveld· Nov 9, 2005, 01:09 UTC · re: Ben Clifford · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

On Tue, Nov 08, 2005 at 10:19:53PM +1100, Ben Clifford wrote:
Show 5 quoted lines
> I got similar today; www.hawaga.org.uk is some apache server on  
> linux, and piva.hawaga.org.uk, the client Mac OS X.
> ...
> error:  (curl_result = 3181280, http_code = 200, sha1 =  
> c99aa418704f576aad8249d042cd6afecf38afc4)

Those curl result codes all look wrong, and sounds like a memory issue that Johannes Schindelin recently fixed in commit 90279074ca5cc336a8bfffd47d19d089b291b432. Does your git build have that patch?

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.
Ben Clifford· Nov 9, 2005, 09:49 UTC · re: Nick Hengeveld · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

On 9 Nov 2005, at 12:09, Nick Hengeveld wrote:
Show 6 quoted lines
>
> Those curl result codes all look wrong, and sounds like a memory issue
> that Johannes Schindelin recently fixed in commit
> 90279074ca5cc336a8bfffd47d19d089b291b432.  Does your git build have  
> that
> patch?

I did not - I've pulled and rebuilt from master, and have that commit now. It works better (not perfectly (there's a tags 403) but I suspect that may be permissions config at my server end or absence of tags in my repo or something mumble). I can do this a bunch of times in a row with the same results and the resulting cloned repository looks sane.

Ben
!535 [0] benc@piva:~/tmp/xa4$ cg clone http://www.hawaga.org.uk/ 
gitcompletion.git
defaulting to local storage area
20:46:16 URL:http://www.hawaga.org.uk/gitcompletion.git/HEAD [41/41] - 
 > "refs/heads/.origin-fetching" [1]
progress: 28 objects, 7901 bytes
http://www.hawaga.org.uk/gitcompletion.git/refs/tags/:
20:46:36 ERROR 403: Forbidden.
FINISHED --20:46:36--
Downloaded: 0 bytes in 0 files
New branch: a108bdc110dad770ec5c092759a8bc511790d21f
Cloned to gitcompletion/ (origin http://www.hawaga.org.uk/ 
gitcompletion.git available as branch "origin")
-- 
Ben ベン Бэн
http://www.hawaga.org.uk/ben/
Petr Baudis· Nov 9, 2005, 11:20 UTC · re: Ben Clifford · lore

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

Dear diary, on Wed, Nov 09, 2005 at 10:49:30AM CET, I got a letter where Ben Clifford <benc@hawaga.org.uk> said that...

Show 15 quoted lines
> 
> On 9 Nov 2005, at 12:09, Nick Hengeveld wrote:
> >
> >Those curl result codes all look wrong, and sounds like a memory issue
> >that Johannes Schindelin recently fixed in commit
> >90279074ca5cc336a8bfffd47d19d089b291b432.  Does your git build have  
> >that
> >patch?
> 
> I did not - I've pulled and rebuilt from master, and have that commit  
> now.
> It works better (not perfectly (there's a tags 403) but I suspect  
> that may be permissions config at my server end or absence of tags in  
> my repo or something mumble). I can do this a bunch of times in a row  
> with the same results and the resulting cloned repository looks sane.

This likely means that you don't permit directory listing of the tags/ subdir. Since Cogito is just about to cease using it, though, it's up to you whether you'll go through the trouble of enabling it or waiting for new Cogito version. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

← back to recent threads