threads / patch / 1876

patch, 8 partscogito: Don't ever assume object type in cg-object-id

Subject: [PATCH 6/8] cogito: Don't ever assume object type in cg-object-id

## tl;dr

8 messages between Sep 20, 2005 and Sep 22, 2005. Diffs are folded; open one to read it.

replies: 7people: 2as markdown or json

Pavel Roskin· Sep 20, 2005, 02:25 UTC · lore
Don't ever assume object type in cg-object-id.

If type is unknown in normalize_id(), make it empty so it's checked later. Check parent ID that it's a commit.

Show changes to cg-object-id +11 −9
diff --git a/cg-object-id b/cg-object-id
--- a/cg-object-id
+++ b/cg-object-id
@@ -24,22 +24,22 @@
 USAGE="cg-object-id [-p] [-t] [OBJECT_ID]"
 
 . ${COGITO_LIB}cg-Xlib
 
 
-# Normalize argument.  The normalized SHA1 ID is put to $normid,
-# type is put to $type.
+# Normalize argument.  The normalized SHA1 ID is put to $normid.
+# If type is known, it's put to $type.
 normalize_id()
 {
 	local id="$1"
+	local type=
 
 	if [ "${id:(-1):1}" = "^" ]; then
 		# find first parent
 		normalize_id "${id%^}"
 		normid=$(git-cat-file commit "$normid" | \
 			 awk '/^parent/{print $2; exit};/^$/{exit}') || exit 1
-		type="commit"
 		return
 	fi
 
 	if [ ! "$id" ] || [ "$id" = "this" ] || [ "$id" = "HEAD" ]; then
 		read id < "$_git/HEAD"
@@ -118,31 +118,33 @@ if [ "$show_parent" -a "$show_tree" ]; t
 	exit 1
 fi
 
 id="${ARGS[0]}"
 normalize_id "$id"
+[ "$type" ] || type=$(git-cat-file -t "$normid")
+
+if [ -z "$show_tree" ]; then
+	if [ "$type" != "commit" ]; then
+		echo "Invalid commit id: $normid" >&2
+		exit 1
+	fi
+fi
 
 if [ "$show_parent" ]; then
 	git-cat-file commit "$normid" | awk '/^parent/{print $2};/^$/{exit}'
 	exit 0
 fi
 
-[ "$type" ] || type=$(git-cat-file -t "$normid")
 if [ "$show_tree" ]; then
 	if [ "$type" = "commit" ]; then
 		normid=$(git-cat-file commit "$normid" | sed -e 's/tree //;q')
 		type=$(git-cat-file -t "$normid")
 	fi
 
 	if [ "$type" != "tree" ]; then
 		echo "Invalid tree id: $normid" >&2
 		exit 1
 	fi
-else
-	if [ "$type" != "commit" ]; then
-		echo "Invalid commit id: $normid" >&2
-		exit 1
-	fi
 fi
 
 echo $normid
 
-- 
Regards,
Pavel Roskin
Petr Baudis· Sep 21, 2005, 10:00 UTC · re: Pavel Roskin · lore

Re: [PATCH 6/8] cogito: Don't ever assume object type in cg-object-id

Dear diary, on Tue, Sep 20, 2005 at 04:25:20AM CEST, I got a letter where Pavel Roskin <proski@gnu.org> told me that...

> Don't ever assume object type in cg-object-id.
> 
> If type is unknown in normalize_id(), make it empty so it's checked
> later.  Check parent ID that it's a commit.

Why? In most of the cases, you have one pointless fork() more, and we "assume" only if it was in a parent line, and if that's not a commit, you've got a *serious* problem with your database. fsck should yell you to death, and I don't think you could even normally commit such a commit, so this should never happen. There's no point in wasting time trying to handle seriously messed up database.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Pavel Roskin· Sep 22, 2005, 03:27 UTC · re: Petr Baudis · lore

Re: [PATCH 6/8] cogito: Don't ever assume object type in cg-object-id

Hello, Petr!
On Wed, 2005-09-21 at 12:00 +0200, Petr Baudis wrote:
Show 13 quoted lines
> Dear diary, on Tue, Sep 20, 2005 at 04:25:20AM CEST, I got a letter
> where Pavel Roskin <proski@gnu.org> told me that...
> > Don't ever assume object type in cg-object-id.
> > 
> > If type is unknown in normalize_id(), make it empty so it's checked
> > later.  Check parent ID that it's a commit.
> 
> Why? In most of the cases, you have one pointless fork() more, and we
> "assume" only if it was in a parent line, and if that's not a commit,
> you've got a *serious* problem with your database. fsck should yell you
> to death, and I don't think you could even normally commit such a
> commit, so this should never happen. There's no point in wasting time
> trying to handle seriously messed up database.

I agree in this case. I actually was unsure, but I decided to play safe after reading this comment in cg-tag:

# This is most usually the ID of the commit to tag. Tagging # other objects than commits is possible, but rather "unusual".

As for fsck "yelling to death", when was the last time you tried that on the cogito repository fetched by cogito? That's what I get:

[proski@dv cogito]$ git-fsck-objects
missing tree 0842d4cb3cd36675c518c241b16cf25fad0c5384
broken link from  commit 2c70421be7d88fbee49986d7a5584d1f010a25de
              to    tree 8c365c79293c9a8a86dc6802f6230e389e876acb
broken link from  commit 2c70421be7d88fbee49986d7a5584d1f010a25de
              to  commit 659bd6b6489ae254a07b1bc578ff44778ed7b8b4
broken link from  commit 463d05c7c4fe7f24da29749f4c7f25893fc20b8c
              to    tree 0842d4cb3cd36675c518c241b16cf25fad0c5384
broken link from  commit 463d05c7c4fe7f24da29749f4c7f25893fc20b8c
              to  commit ed34298a39b05e05bd333d9dac658df19e5b2dab
missing commit 659bd6b6489ae254a07b1bc578ff44778ed7b8b4
missing tree 8c365c79293c9a8a86dc6802f6230e389e876acb
missing commit c4cd9bc72a6e0ed355041d331bb4034d99738f82
broken link from  commit d14925c87cdb6ca6345bcb3c8e34a2d659c79451
              to    tree e5ea614d1884a02314a3be8e09cb3f8c786fc0e9
broken link from  commit d14925c87cdb6ca6345bcb3c8e34a2d659c79451
              to  commit c4cd9bc72a6e0ed355041d331bb4034d99738f82
missing tree e5ea614d1884a02314a3be8e09cb3f8c786fc0e9
missing commit ed34298a39b05e05bd333d9dac658df19e5b2dab

It's not like broken trees are exceedingly rare. Sure, this must be some "better" kind of breakage, but anyway.

-- 
Regards,
Pavel Roskin
Petr Baudis· Sep 22, 2005, 09:46 UTC · re: Pavel Roskin · lore

Re: [PATCH 6/8] cogito: Don't ever assume object type in cg-object-id

Dear diary, on Thu, Sep 22, 2005 at 05:27:05AM CEST, I got a letter where Pavel Roskin <proski@gnu.org> told me that...

Show 22 quoted lines
> As for fsck "yelling to death", when was the last time you tried that on
> the cogito repository fetched by cogito?  That's what I get:
> 
> [proski@dv cogito]$ git-fsck-objects
> missing tree 0842d4cb3cd36675c518c241b16cf25fad0c5384
> broken link from  commit 2c70421be7d88fbee49986d7a5584d1f010a25de
>               to    tree 8c365c79293c9a8a86dc6802f6230e389e876acb
> broken link from  commit 2c70421be7d88fbee49986d7a5584d1f010a25de
>               to  commit 659bd6b6489ae254a07b1bc578ff44778ed7b8b4
> broken link from  commit 463d05c7c4fe7f24da29749f4c7f25893fc20b8c
>               to    tree 0842d4cb3cd36675c518c241b16cf25fad0c5384
> broken link from  commit 463d05c7c4fe7f24da29749f4c7f25893fc20b8c
>               to  commit ed34298a39b05e05bd333d9dac658df19e5b2dab
> missing commit 659bd6b6489ae254a07b1bc578ff44778ed7b8b4
> missing tree 8c365c79293c9a8a86dc6802f6230e389e876acb
> missing commit c4cd9bc72a6e0ed355041d331bb4034d99738f82
> broken link from  commit d14925c87cdb6ca6345bcb3c8e34a2d659c79451
>               to    tree e5ea614d1884a02314a3be8e09cb3f8c786fc0e9
> broken link from  commit d14925c87cdb6ca6345bcb3c8e34a2d659c79451
>               to  commit c4cd9bc72a6e0ed355041d331bb4034d99738f82
> missing tree e5ea614d1884a02314a3be8e09cb3f8c786fc0e9
> missing commit ed34298a39b05e05bd333d9dac658df19e5b2dab
Heh, your database seems fairly broken.
> It's not like broken trees are exceedingly rare.  Sure, this must be
> some "better" kind of breakage, but anyway.

$ git-fsck-objects error: b3e9704ecdf48869f635f0aa99ddfb513f885aff: object not found

and some dangling stuff, that's all I get. This seems to be caused by the mere existence of that (empty) file, so I just removed it from the objects database and all is fine now.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Petr Baudis· Sep 22, 2005, 09:52 UTC · re: Petr Baudis · lore

Re: [PATCH 6/8] cogito: Don't ever assume object type in cg-object-id

Dear diary, on Thu, Sep 22, 2005 at 11:46:43AM CEST, I got a letter where Petr Baudis <pasky@suse.cz> told me that...

Show 27 quoted lines
> Dear diary, on Thu, Sep 22, 2005 at 05:27:05AM CEST, I got a letter
> where Pavel Roskin <proski@gnu.org> told me that...
> > As for fsck "yelling to death", when was the last time you tried that on
> > the cogito repository fetched by cogito?  That's what I get:
> > 
> > [proski@dv cogito]$ git-fsck-objects
> > missing tree 0842d4cb3cd36675c518c241b16cf25fad0c5384
> > broken link from  commit 2c70421be7d88fbee49986d7a5584d1f010a25de
> >               to    tree 8c365c79293c9a8a86dc6802f6230e389e876acb
> > broken link from  commit 2c70421be7d88fbee49986d7a5584d1f010a25de
> >               to  commit 659bd6b6489ae254a07b1bc578ff44778ed7b8b4
> > broken link from  commit 463d05c7c4fe7f24da29749f4c7f25893fc20b8c
> >               to    tree 0842d4cb3cd36675c518c241b16cf25fad0c5384
> > broken link from  commit 463d05c7c4fe7f24da29749f4c7f25893fc20b8c
> >               to  commit ed34298a39b05e05bd333d9dac658df19e5b2dab
> > missing commit 659bd6b6489ae254a07b1bc578ff44778ed7b8b4
> > missing tree 8c365c79293c9a8a86dc6802f6230e389e876acb
> > missing commit c4cd9bc72a6e0ed355041d331bb4034d99738f82
> > broken link from  commit d14925c87cdb6ca6345bcb3c8e34a2d659c79451
> >               to    tree e5ea614d1884a02314a3be8e09cb3f8c786fc0e9
> > broken link from  commit d14925c87cdb6ca6345bcb3c8e34a2d659c79451
> >               to  commit c4cd9bc72a6e0ed355041d331bb4034d99738f82
> > missing tree e5ea614d1884a02314a3be8e09cb3f8c786fc0e9
> > missing commit ed34298a39b05e05bd333d9dac658df19e5b2dab
> 
> 
> Heh, your database seems fairly broken.

Ok, this seems like broken database on kernel.org, sorry about that. It actually just contained some superfluous objects. I pruned it and it seems that fsck is happy now.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Pavel Roskin· Sep 22, 2005, 15:37 UTC · re: Petr Baudis · lore

Re: [PATCH 6/8] cogito: Don't ever assume object type in cg-object-id

Hi, Petr!
On Thu, 2005-09-22 at 11:46 +0200, Petr Baudis wrote:
Show 6 quoted lines
> $ git-fsck-objects
> error: b3e9704ecdf48869f635f0aa99ddfb513f885aff: object not found
> 
> and some dangling stuff, that's all I get. This seems to be caused by
> the mere existence of that (empty) file, so I just removed it from the
> objects database and all is fine now.

Maybe kernel.org site is not in sync yet, but anyway, here's what I found.

I ran this script:

[proski@dv cogito]$ for i in .git/refs/tags/*; do git-tar-tree `cat $i` >/dev/null || echo $i; done fatal: not a reference to a tag, commit or tree object: 463d05c7c4fe7f24da29749f4c7f25893fc20b8c .git/refs/tags/git-pasky-0.1 fatal: not a reference to a tag, commit or tree object: 2c70421be7d88fbee49986d7a5584d1f010a25de .git/refs/tags/git-pasky-0.2 fatal: not a reference to a tag, commit or tree object: d14925c87cdb6ca6345bcb3c8e34a2d659c79451 .git/refs/tags/git-pasky-0.3

If I delete the bad tags and prune the tree, git-fsck-objects has nothing to complain about:

[proski@dv cogito]$ rm .git/refs/tags/git-pasky-0.1 [proski@dv cogito]$ rm .git/refs/tags/git-pasky-0.2 [proski@dv cogito]$ rm .git/refs/tags/git-pasky-0.3 [proski@dv cogito]$ git-prune origin `cat .git/refs/tags/*` [proski@dv cogito]$ git-fsck-objects [proski@dv cogito]$

That's probably what you should do on the master tree if you haven't done it.

I can run "git-prune origin", but then cg-fetch would refetch objects for more tags:

Missing object of tag cogito-0.12... retrieved Missing object of tag cogito-0.12.1... retrieved Missing object of tag cogito-0.13... retrieved Missing object of tag cogito-0.14... retrieved Missing object of tag cogito-0.14.1... retrieved Missing object of tag cogito-0.14.2... retrieved Missing object of tag cogito-0.15... retrieved Missing object of tag git-pasky-0.1... retrieved Missing object of tag git-pasky-0.2... retrieved Missing object of tag git-pasky-0.3... retrieved

How come that cogito-0.12 - cogito-0.15 are not on the origin tree, but older cogito tags are?

-- 
Regards,
Pavel Roskin
Petr Baudis· Sep 22, 2005, 15:50 UTC · re: Pavel Roskin · lore

Re: [PATCH 6/8] cogito: Don't ever assume object type in cg-object-id

Dear diary, on Thu, Sep 22, 2005 at 05:37:59PM CEST, I got a letter where Pavel Roskin <proski@gnu.org> told me that...

> Hi, Petr!
Hi,
Show 10 quoted lines
> On Thu, 2005-09-22 at 11:46 +0200, Petr Baudis wrote:
> > $ git-fsck-objects
> > error: b3e9704ecdf48869f635f0aa99ddfb513f885aff: object not found
> > 
> > and some dangling stuff, that's all I get. This seems to be caused by
> > the mere existence of that (empty) file, so I just removed it from the
> > objects database and all is fine now.
> 
> Maybe kernel.org site is not in sync yet, but anyway, here's what I
> found.
root      4036  2.7  2.8 106756 104080 ?     SN   00:04  14:32 rsync
:/
Show 9 quoted lines
> I ran this script:
> 
> [proski@dv cogito]$ for i in .git/refs/tags/*; do git-tar-tree `cat $i` >/dev/null || echo $i; done
> fatal: not a reference to a tag, commit or tree object: 463d05c7c4fe7f24da29749f4c7f25893fc20b8c
> .git/refs/tags/git-pasky-0.1
> fatal: not a reference to a tag, commit or tree object: 2c70421be7d88fbee49986d7a5584d1f010a25de
> .git/refs/tags/git-pasky-0.2
> fatal: not a reference to a tag, commit or tree object: d14925c87cdb6ca6345bcb3c8e34a2d659c79451
> .git/refs/tags/git-pasky-0.3
Huh. Huh?

xpasky@machine[1:0]~/git-devel$ git-cat-file -t $(cg-object-id git-pasky-0.1) commit xpasky@machine[1:0]~/git-devel$ git-cat-file -t $(cg-object-id git-pasky-0.2) commit xpasky@machine[1:0]~/git-devel$ git-cat-file -t $(cg-object-id git-pasky-0.3) commit

Same on master.kernel.org.
Show 16 quoted lines
> I can run "git-prune origin", but then cg-fetch would refetch objects
> for more tags:
> 
> Missing object of tag cogito-0.12... retrieved
> Missing object of tag cogito-0.12.1... retrieved
> Missing object of tag cogito-0.13... retrieved
> Missing object of tag cogito-0.14... retrieved
> Missing object of tag cogito-0.14.1... retrieved
> Missing object of tag cogito-0.14.2... retrieved
> Missing object of tag cogito-0.15... retrieved
> Missing object of tag git-pasky-0.1... retrieved
> Missing object of tag git-pasky-0.2... retrieved
> Missing object of tag git-pasky-0.3... retrieved
> 
> How come that cogito-0.12 - cogito-0.15 are not on the origin tree, but
> older cogito tags are?

That's because the older tags tag commits directly, not tag objects. git-prune removed the tag objects because they were unreferenced by its perspective. Anyway, git-prune without any arguments should get it right.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Pavel Roskin· Sep 22, 2005, 16:55 UTC · re: Petr Baudis · lore

Re: [PATCH 6/8] cogito: Don't ever assume object type in cg-object-id

On Thu, 2005-09-22 at 17:50 +0200, Petr Baudis wrote:
Show 14 quoted lines
> Dear diary, on Thu, Sep 22, 2005 at 05:37:59PM CEST, I got a letter
> where Pavel Roskin <proski@gnu.org> told me that...
> > [proski@dv cogito]$ for i in .git/refs/tags/*; do git-tar-tree `cat $i` >/dev/null || echo $i; done
> > fatal: not a reference to a tag, commit or tree object: 463d05c7c4fe7f24da29749f4c7f25893fc20b8c
> > .git/refs/tags/git-pasky-0.1
> > fatal: not a reference to a tag, commit or tree object: 2c70421be7d88fbee49986d7a5584d1f010a25de
> > .git/refs/tags/git-pasky-0.2
> > fatal: not a reference to a tag, commit or tree object: d14925c87cdb6ca6345bcb3c8e34a2d659c79451
> > .git/refs/tags/git-pasky-0.3
> 
> Huh. Huh?
> 
> xpasky@machine[1:0]~/git-devel$ git-cat-file -t $(cg-object-id git-pasky-0.1)
> commit

The error message is misleading. In fact, it's the tree referred to by the commit that was missing. Anyway, that object exists on the server, so it was my local problem.

It would be nice to have a script fetch an object by SHA1, possibly with the dependencies. git-fetch doesn't do it yet.

I copied the whole objects tree for cogito using rsync, and I see lots or error messages from git-fsck-objects. I'll wait for your changes to propagate. As it stands now, b3e9704ecdf48869f635f0aa99ddfb513f885aff (a file you said you have removed) and 11ed64c1b141c9ba397a1ca76aef2cd250976007 are still present and empty.

Are you sure rsync that copies to kernel.org uses the "delete" option? If not, we should treat kernel.org as "dumb transport" where directory listings should never be used.

Show 7 quoted lines
> > How come that cogito-0.12 - cogito-0.15 are not on the origin tree, but
> > older cogito tags are?
> 
> That's because the older tags tag commits directly, not tag objects.
> git-prune removed the tag objects because they were unreferenced by its
> perspective. Anyway, git-prune without any arguments should get it
> right.
Thanks for the explanation.  I'll be more careful with git-prune.
-- 
Regards,
Pavel Roskin

← back to recent threads