threads / discuss / 2334

Problem cloning the Linux history tree

Subject: Problem cloning the Linux history tree

## tl;dr

11 messages between Nov 3, 2005 and Nov 7, 2005.

replies: 10people: 5as markdown or json

Marcel Holtmann· Nov 3, 2005, 08:56 UTC · lore
Hi guys,

I have some problems cloning the Linux history tree from Thomas Gleixner over the HTTP transport. The rsync transport seems to work.

# cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git defaulting to local storage area 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1] Getting alternates list Getting pack list error: The requested URL returned error: 404 error: Unable to find e7e173af42dbf37b1d946f9ee00219cb3b2bea6a under http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/

Cannot obtain needed object e7e173af42dbf37b1d946f9ee00219cb3b2bea6a while processing commit 0000000000000000000000000000000000000000. cg-fetch: objects fetch failed cg-clone: fetch failed

Regards
Marcel
Junio C Hamano· Nov 4, 2005, 03:02 UTC · re: Marcel Holtmann · lore

Re: Problem cloning the Linux history tree

Marcel Holtmann <marcel@holtmann.org> writes:
Show 6 quoted lines
> # cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> defaulting to local storage area
> 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
> Getting alternates list
> Getting pack list
> error: The requested URL returned error: 404

That repository is packed but lacks objects/info/packs file. Perhaps you need to ask tglx to run git-update-server-info in the repository.

Cloning over git native transport should work, though:
    $ git clone git://git.kernel.org/pub/scm/lin.../history.git history
Wolfgang Denk· Nov 4, 2005, 09:21 UTC · re: Junio C Hamano · lore

Re: Problem cloning the Linux history tree

In message <7vmzkl3yl0.fsf@assigned-by-dhcp.cox.net> you wrote:
> 
> > # cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> > defaulting to local storage area
> > 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
...
> That repository is packed but lacks objects/info/packs file.
> Perhaps you need to ask tglx to run git-update-server-info in
> the repository.
I have the same (or a very similar) problem with our Linux repo:
-> cg-clone http://www.denx.de/git/linux-2.6-denx.git foo
...
defaulting to local storage area
09:54:53 URL:http://www.denx.de/git/linux-2.6-denx.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
Getting alternates list
Getting pack list
Getting index for pack eceba3479417b338c2e156b1c6dfd69f3f0eaad1
Getting index for pack b3c6fbdfa36a326815de6358885c7a570a986b1b
Getting index for pack 6122f3ffa476a089122c0bffbbb7a7ce31304b56
Getting index for pack 06949712bb970fc1dd2e13f50accdcd2bf992536
Getting index for pack 459148e2ea863c15479d69ed4886a6f200c79a49
Getting pack 459148e2ea863c15479d69ed4886a6f200c79a49
which contains 010a4c0059a82b834b4d7f65d5cdd4a9717d8ea7
progress: 3 objects, 0 bytes
Getting pack 06949712bb970fc1dd2e13f50accdcd2bf992536
which contains 5014bfa48ac169e0748e1e9651897788feb306dc
progress: 12210 objects, 0 bytes
09:58:42 ERROR 404: Not Found.
09:58:42 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/DENX-2005-10-02-18:30 [41/41] -> "refs/tags/DENX-2005-10-02-18:30" [1]
...
09:58:43 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/v2.6.14-rc5 [41/41] -> "refs/tags/v2.6.14-rc5" [1]
FINISHED --09:58:43--
Downloaded: 19,240 bytes in 30 files
Missing object of tag v2.6.14-rc3... unable to retrieve
New branch: 010a4c0059a82b834b4d7f65d5cdd4a9717d8ea7
Cloned to foo/ (origin http://www.denx.de/git/linux-2.6-denx.git available as branch "origin")
And I did run git-update-server-info, even with --force.
In the server log I see messages like these:

[Fri Nov 04 09:54:52 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/01 [Fri Nov 04 09:54:52 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/info/http-alternates [Fri Nov 04 09:54:53 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/info/alternates [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/5b [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/ca [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/50 [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/98 [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/79 [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/4e [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/79 [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/c1 [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/f9 [Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/e5 [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/e1 [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/67 [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/0c [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/b6 [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/bc

These directories really don't exist:

$ cd /srv/git/linux-2.6-denx.git/objects . ls -l total 8 drwxrwxr-x 2 wd wd 4096 Nov 4 09:52 info drwxrwxr-x 2 wd wd 4096 Nov 3 15:15 pack

The question is : why are they requested?
> Cloning over git native transport should work, though:
> 
>     $ git clone git://git.kernel.org/pub/scm/lin.../history.git history

This does not work for me, either. It runs a long time without obvious problems, but finally fails with:

-> git clone http://www.denx.de/git/linux-2.6-denx.git bar ... walk c264852726dde251a0c09ec22f61a9be8b0db68b walk 69903d6500c73af8329a5fba7153b0d50748981c walk b59a9504cb93db7fae31e60760725d48652a1fc3 error: Could not interpret tags/DENX-2005-10-02-18:30 as something to pull

I have to admit that I have absolutely no clue (1) what might have caused this situation nor (2) how to fix it :-(

Best regards,
Wolfgang Denk
-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
 The software required `Windows 95 or better', so I installed Linux.
Petr Baudis· Nov 4, 2005, 09:26 UTC · re: Wolfgang Denk · lore

Re: Problem cloning the Linux history tree

Dear diary, on Fri, Nov 04, 2005 at 10:21:46AM CET, I got a letter where Wolfgang Denk <wd@denx.de> told me that...

Show 40 quoted lines
> In message <7vmzkl3yl0.fsf@assigned-by-dhcp.cox.net> you wrote:
> > 
> > > # cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> > > defaulting to local storage area
> > > 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
> ...
> > That repository is packed but lacks objects/info/packs file.
> > Perhaps you need to ask tglx to run git-update-server-info in
> > the repository.
> 
> I have the same (or a very similar) problem with our Linux repo:
> 
> -> cg-clone http://www.denx.de/git/linux-2.6-denx.git foo
> ...
> defaulting to local storage area
> 09:54:53 URL:http://www.denx.de/git/linux-2.6-denx.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
> Getting alternates list
> Getting pack list
> Getting index for pack eceba3479417b338c2e156b1c6dfd69f3f0eaad1
> Getting index for pack b3c6fbdfa36a326815de6358885c7a570a986b1b
> Getting index for pack 6122f3ffa476a089122c0bffbbb7a7ce31304b56
> Getting index for pack 06949712bb970fc1dd2e13f50accdcd2bf992536
> Getting index for pack 459148e2ea863c15479d69ed4886a6f200c79a49
> Getting pack 459148e2ea863c15479d69ed4886a6f200c79a49
> which contains 010a4c0059a82b834b4d7f65d5cdd4a9717d8ea7
> progress: 3 objects, 0 bytes
> Getting pack 06949712bb970fc1dd2e13f50accdcd2bf992536
> which contains 5014bfa48ac169e0748e1e9651897788feb306dc
> progress: 12210 objects, 0 bytes
> 09:58:42 ERROR 404: Not Found.
> 09:58:42 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/DENX-2005-10-02-18:30 [41/41] -> "refs/tags/DENX-2005-10-02-18:30" [1]
> ...
> 09:58:43 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/v2.6.14-rc5 [41/41] -> "refs/tags/v2.6.14-rc5" [1]
> FINISHED --09:58:43--
> Downloaded: 19,240 bytes in 30 files
> Missing object of tag v2.6.14-rc3... unable to retrieve
> New branch: 010a4c0059a82b834b4d7f65d5cdd4a9717d8ea7
> Cloned to foo/ (origin http://www.denx.de/git/linux-2.6-denx.git available as branch "origin")
> 
> And I did run git-update-server-info, even with --force.
And what exactly is the problem? It seemed to clone ok.
> In the server log I see messages like these:
> 
> [Fri Nov 04 09:54:52 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/01
..snip..
> [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/bc

The fetching code first tries to look into the unpacked object store (that are those errors) and if that fails, it grabs a pack.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Wolfgang Denk· Nov 4, 2005, 09:38 UTC · re: Petr Baudis · lore

Re: Problem cloning the Linux history tree

In message <20051104092626.GL1431@pasky.or.cz> you wrote:
Show 5 quoted lines
> 
> > 09:58:42 ERROR 404: Not Found.
> > 09:58:42 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/DENX-2005-10-02-18:30 [41/41] -> "refs/tags/DENX-2005-10-02-18:30" [1]
> > ...
> > Missing object of tag v2.6.14-rc3... unable to retrieve
> And what exactly is the problem? It seemed to clone ok.
There was a 404 error, and the "Missing object of tag" message.
Best regards,
Wolfgang Denk
-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
It usually takes more than three weeks to prepare  a  good  impromptu
speech.                                                  - Mark Twain
Junio C Hamano· Nov 4, 2005, 09:44 UTC · re: Petr Baudis · lore

Re: Problem cloning the Linux history tree

Petr Baudis <pasky@suse.cz> writes:
>> 09:58:42 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/DENX-2005-10-02-18:30 [41/41] -> "refs/tags/DENX-2005-10-02-18:30" [1]

I do not know if Cogito works it around or the above just happens to work, but (notice colon there) tag names that do not pass git-check-ref-format(1) are not supported.

>> And I did run git-update-server-info, even with --force.

You may want to double check that all your refs appear in denx.git/info/refs file, and similarly all your packs are described in denx.git/objects/info/packs file.

> And what exactly is the problem? It seemed to clone ok.
Oh, if that is the case I would stop worrying about it, then.
However, there is this bit that looks suspicious.
>> Missing object of tag v2.6.14-rc3... unable to retrieve

This does not look like an error message I generate, so I am assuming it is coming from cg-fetch.

Junio C Hamano· Nov 4, 2005, 09:59 UTC · re: Wolfgang Denk · lore

Re: Problem cloning the Linux history tree

Wolfgang Denk <wd@denx.de> writes:
> Missing object of tag v2.6.14-rc3... unable to retrieve

I do not have a clue on this one; I am assuming that it is coming from cg-fetch.

Show 9 quoted lines
> [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/bc
>
> These directories really don't exist:
>
> $ cd /srv/git/linux-2.6-denx.git/objects
> . ls -l
> total 8
> drwxrwxr-x  2 wd wd 4096 Nov  4 09:52 info
> drwxrwxr-x  2 wd wd 4096 Nov  3 15:15 pack

Quick question. Is the source repository fully packed (i.e. git repack followed by git prune)? If that is the case there is nothing to worry about. The latest git prunes empty $GIT_OBJECT_DIRECTORY/??/ directories and recreates them on demand.

> The question is : why are they requested?

Fetching over http is what we call a "commit walker" -- it learns the top commit object names, gets that object and reads it to learn what other objects it needs to slurp, and keeps asking the next object, one by one. When dealing with a packed repository, it first asks individual objects (expecting they would be found in $GIT_OBJECT_DIRECTORY/??/ directory), and when not found it fetchs packs and looks inside. Recent commit walker knows how to issue multiple requests while earlier request is still in transit, so you can expect some requests to individual objects. That is benign.

Show 5 quoted lines
>> Cloning over git native transport should work, though:
>> 
>>     $ git clone git://git.kernel.org/pub/scm/lin.../history.git history
>
> This does not work for me,  either.
The above uses git:// URL  but your failure below uses http://
Show 6 quoted lines
> -> git clone http://www.denx.de/git/linux-2.6-denx.git bar
> ...
> walk c264852726dde251a0c09ec22f61a9be8b0db68b
> walk 69903d6500c73af8329a5fba7153b0d50748981c
> walk b59a9504cb93db7fae31e60760725d48652a1fc3
> error: Could not interpret tags/DENX-2005-10-02-18:30 as something to pull

This one I can answer. It is the colon in your tag name. Please rename it to make git-check-ref-format(1) happy, run git-update-server-info(1) and try again.

Wolfgang Denk· Nov 4, 2005, 13:17 UTC · re: Junio C Hamano · lore

Re: Problem cloning the Linux history tree

In message <7vacgkzqau.fsf@assigned-by-dhcp.cox.net> you wrote:
Show 6 quoted lines
>
> > Missing object of tag v2.6.14-rc3... unable to retrieve
> 
> I do not have a clue on this one; I am assuming that it is
> coming from cg-fetch.
> 
> Quick question.  Is the source repository fully packed (i.e. git
> repack followed by git prune)?  If that is the case there is

Yes, it is. I decided to run repack + update-server-info hoping that would help to cleanup the state.

> nothing to worry about.  The latest git prunes empty
> $GIT_OBJECT_DIRECTORY/??/ directories and recreates them on
> demand.
I see. Thanks.
Show 5 quoted lines
> > error: Could not interpret tags/DENX-2005-10-02-18:30 as something to pull
> 
> This one I can answer.  It is the colon in your tag name.
> Please rename it to make git-check-ref-format(1) happy, run
> git-update-server-info(1) and try again.
Understood. Thanks.
Best regards,
Wolfgang Denk
-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
No one can guarantee the actions of another.
	-- Spock, "Day of the Dove", stardate unknown
Marcel Holtmann· Nov 4, 2005, 15:47 UTC · re: Junio C Hamano · lore

Re: Problem cloning the Linux history tree

Hi Thomas,
Show 10 quoted lines
> > # cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> > defaulting to local storage area
> > 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
> > Getting alternates list
> > Getting pack list
> > error: The requested URL returned error: 404
> 
> That repository is packed but lacks objects/info/packs file.
> Perhaps you need to ask tglx to run git-update-server-info in
> the repository.
can please try to fix it according to Junio's proposal.
Regards
Marcel

← back to recent threads