threads / discuss / 20109

Recover broken git repository?

Subject: Recover broken git repository?

## tl;dr

9 messages between Jul 14, 2009 and Jul 14, 2009.

replies: 8people: 7as markdown or json

Florian Breitwieser· Jul 14, 2009, 13:20 UTC · lore

Hi, I have problems with my git repository, attached below are the steps I tried to resolve it. But now I am stuck. Is there any good way to recover?

Thank you Florian

-- 
Florian Breitwieser
PhD Student

Research Center of Molecular Medicine
of the Austrian Academy of Science

-------------------

$ git commit -m "Some message"
error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
fatal: Error building trees

$ git-fsck --full
dangling tree c2549a3cdd83098a523cb707f217f4656cde7eb5

$ git prune
$ git commit -m "Some message"
Created commit e32d5dd: Some message
 2 files changed, 167 insertions(+), 10 deletions(-)

$ git push
Counting objects: 15, done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 2.44 KiB, done.
Total 8 (delta 4), reused 0 (delta 0)
Unpacking objects: 100% (8/8), done.
fatal: unresolved deltas left after unpacking
error: unpack failed: unpacker exited with error code
To /mnt/extstore2/bioinformatic/git/ppi.git
 ! [remote rejected] master -> master (n/a (unpacker error))
error: failed to push some refs to
'/mnt/extstore2/bioinformatic/git/ppi.git'

-- 
View this message in context: http://n2.nabble.com/Recover-broken-git-repository--tp3255106p3255106.html
Sent from the git mailing list archive at Nabble.com.
Alex Riesen· Jul 14, 2009, 14:40 UTC · re: Florian Breitwieser · lore

Re: Recover broken git repository?

On Tue, Jul 14, 2009 at 15:20, Florian Breitwieser<florian.bw@gmail.com> wrote:
> I have problems with my git repository, attached below are the steps I tried
> to resolve it. But now I am stuck. Is there any good way to recover?

There are recovery instructions all over the web, and on this mailing list. Try googling for "git repository corruption recovery". There is even one in Git's repository:

  Documentation/howto/recover-corrupted-blob-object.txt
> $ git commit -m "Some message"
> error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
Looks like you have hit a genuine repository corruption.

You can try to look for the object in all copies of the project (or just in any projects you possibly have laying around) and copy it into you repo. To test if the object is present in a repo:

  git cat-file -t 1086b1c606a04bcb78b92d1d411a299d20d18034

You can just copy (or reference in .git/objects/info/alternates) the whole repo containing the object in the broken repo. You can cleanup unused objects afterwards.

> fatal: Error building trees

You can try "git log --pretty=raw --raw --no-abbrev HEAD | less" and look for the first commit which introduced the invalid object. The history up to this commit must be intact, BTW. Try to find out what object it was. If it was a file, than maybe you have the original source somewhere, or the latest version is present and good enough. Then you can checkout the good commit, introduce the file from the source, commit, and cherry-pick all the newer commits on top of it. If it was a tree, you can try to guess what files did it contain in the commit which references it and try to rebuild it by using git add and git write-tree (the latter prints the SHA1 of the generated tree on stdout, so you can retry indefinitely). If it happens to be a commit (you'll find the SHA1 in parents of the good commits, but it does not look like it is the case), you can try git fsck --lost-found to find all the commits which are not referenced, try connecting them (by using grafts, for example. Or checkout and cherry-pick described above).

> $ git-fsck --full
> dangling tree c2549a3cdd83098a523cb707f217f4656cde7eb5
>
> $ git prune

Now that was dangerous. When something was unreferenced (but useful) now it is lost.

Nicolas Sebrecht· Jul 14, 2009, 14:54 UTC · re: Florian Breitwieser · lore

Re: Recover broken git repository?

The 14/07/09, Florian Breitwieser wrote:
> I have problems with my git repository, attached below are the steps I tried
> to resolve it. But now I am stuck. Is there any good way to recover?

You could git-format-patch your topics, clone a new repo and git-am the patches.

-- 
Nicolas Sebrecht
Jakub Narebski· Jul 14, 2009, 15:20 UTC · re: Nicolas Sebrecht · lore

Re: Recover broken git repository?

Nicolas Sebrecht <nicolas.s.dev@gmx.fr> writes:
Show 7 quoted lines
> The 14/07/09, Florian Breitwieser wrote:
> 
> > I have problems with my git repository, attached below are the steps I tried
> > to resolve it. But now I am stuck. Is there any good way to recover?
> 
> You could git-format-patch your topics, clone a new repo and git-am the
> patches.

Better solution, if possible, would be to use git-fast-export / git-fast-import pair.

-- 
Jakub Narebski
Poland
ShadeHawk on #git
Linus Torvalds· Jul 14, 2009, 17:33 UTC · re: Florian Breitwieser · lore

Re: Recover broken git repository?

On Tue, 14 Jul 2009, Florian Breitwieser wrote:
>
> $ git commit -m "Some message"
> error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
> fatal: Error building trees

Hmm. That "invalid object" error comes from 'update_one()' when it cannot find the object it is looking for. That, in turn, tends to be an issue of the index containing a SHA1 that does not exist in the database.

Do you trust your filesystem? The symptoms really sound like you did a "git add .." earlier, and populated the index, but now the object that got populated is no longer found.

What OS, what filesystem?  Oh, and what version of git?
> $ git-fsck --full
> dangling tree c2549a3cdd83098a523cb707f217f4656cde7eb5

The fsck seems to imply that things are ok. The fsck _should_ have checked the index too.

The 'git commit' obviously disagrees.
Two things to check:
 - do you use grafts to hide old history, rather than connect additional 
   history?
   That can be really dangerous, and will make fsck (and other tools) not 
   look at the hidden state. 
 - do you have alternates (is multiple _different_ repositories tat point 
   to each other)? That can cause problems if you then do things like git 
   prune in them. It wouldn't explain this particular case, but it's 
   something I look out for when I hear about corruption.
> $ git prune

Not a good idea. When you suspect corruption, the _last_ thing you want to do is prune things. Who knows what happened?

> $ git commit -m "Some message"
> Created commit e32d5dd: Some message
>  2 files changed, 167 insertions(+), 10 deletions(-)

So now the same commit worked? It _really_ sounds like your filesystem has a hard time finding the files it just created. Some local caching issue.

Is it a network filesystem? If so, what's the filesystem server and version, if you can find it out?

Show 12 quoted lines
> $ git push
> Counting objects: 15, done.
> Compressing objects: 100% (6/6), done.
> Writing objects: 100% (8/8), 2.44 KiB, done.
> Total 8 (delta 4), reused 0 (delta 0)
> Unpacking objects: 100% (8/8), done.
> fatal: unresolved deltas left after unpacking
> error: unpack failed: unpacker exited with error code
> To /mnt/extstore2/bioinformatic/git/ppi.git
>  ! [remote rejected] master -> master (n/a (unpacker error))
> error: failed to push some refs to
> '/mnt/extstore2/bioinformatic/git/ppi.git'

Here the other end doesn't have some object that we expect it to have, and that we sent a delta to it against. Sounds like potentially the exact same problem (it created some new file, but then couldn't see it). Is the thign you are pushing to a similar machine with similar filesystems?

We've had issues with both CIFS and NFS. Using
	[core]
		fsyncobjectfiles = true

might work around some issues where the filesystem does crazy things. But I'd really like to hear what OS and filesystem versions you're running.

			Linus
Stefan-W. Hahn· Jul 14, 2009, 17:48 UTC · re: Linus Torvalds · lore

Re: Recover broken git repository?

Also sprach Linus Torvalds am Tue, 14 Jul 2009 at 10:33:58 -0700:
Show 85 quoted lines
> 
> 
> On Tue, 14 Jul 2009, Florian Breitwieser wrote:
> >
> > $ git commit -m "Some message"
> > error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
> > fatal: Error building trees
> 
> Hmm. That "invalid object" error comes from 'update_one()' when it cannot 
> find the object it is looking for. That, in turn, tends to be an issue of 
> the index containing a SHA1 that does not exist in the database.
> 
> Do you trust your filesystem? The symptoms really sound like you did a 
> "git add .." earlier, and populated the index, but now the object that got 
> populated is no longer found.
> 
> What OS, what filesystem?  Oh, and what version of git?
> 
> > $ git-fsck --full
> > dangling tree c2549a3cdd83098a523cb707f217f4656cde7eb5
> 
> The fsck seems to imply that things are ok. The fsck _should_ have checked 
> the index too.
> 
> The 'git commit' obviously disagrees.
> 
> Two things to check:
> 
>  - do you use grafts to hide old history, rather than connect additional 
>    history?
> 
>    That can be really dangerous, and will make fsck (and other tools) not 
>    look at the hidden state. 
> 
>  - do you have alternates (is multiple _different_ repositories tat point 
>    to each other)? That can cause problems if you then do things like git 
>    prune in them. It wouldn't explain this particular case, but it's 
>    something I look out for when I hear about corruption.
> 
> > $ git prune
> 
> Not a good idea. When you suspect corruption, the _last_ thing you want to 
> do is prune things. Who knows what happened?
> 
> > $ git commit -m "Some message"
> > Created commit e32d5dd: Some message
> >  2 files changed, 167 insertions(+), 10 deletions(-)
> 
> So now the same commit worked? It _really_ sounds like your filesystem has 
> a hard time finding the files it just created. Some local caching issue.
> 
> Is it a network filesystem? If so, what's the filesystem server and 
> version, if you can find it out?
> 
> > $ git push
> > Counting objects: 15, done.
> > Compressing objects: 100% (6/6), done.
> > Writing objects: 100% (8/8), 2.44 KiB, done.
> > Total 8 (delta 4), reused 0 (delta 0)
> > Unpacking objects: 100% (8/8), done.
> > fatal: unresolved deltas left after unpacking
> > error: unpack failed: unpacker exited with error code
> > To /mnt/extstore2/bioinformatic/git/ppi.git
> >  ! [remote rejected] master -> master (n/a (unpacker error))
> > error: failed to push some refs to
> > '/mnt/extstore2/bioinformatic/git/ppi.git'
> 
> Here the other end doesn't have some object that we expect it to have, and 
> that we sent a delta to it against. Sounds like potentially the exact same 
> problem (it created some new file, but then couldn't see it). Is the thign 
> you are pushing to a similar machine with similar filesystems?
> 
> We've had issues with both CIFS and NFS. Using
> 
> 	[core]
> 		fsyncobjectfiles = true
> 
> might work around some issues where the filesystem does crazy things. But 
> I'd really like to hear what OS and filesystem versions you're running.
> 
> 			Linus
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Stefan-W. Hahn                          It is easy to make things.
/ mailto:stefan.hahn@s-hahn.de /        It is hard to make things simple.			

Please note that according to the German law on data retention,
information on every electronic information exchange with me is
retained for a period of six months.
Bundesgesetzblatt:
http://www.bgblportal.de/BGBL/bgbl1f/bgbl107s3198.pdf
Stefan-W. Hahn· Jul 14, 2009, 17:52 UTC · re: Linus Torvalds · lore

Re: Recover broken git repository?

Also sprach Linus Torvalds am Tue, 14 Jul 2009 at 10:33:58 -0700:
Hi,
> > $ git commit -m "Some message"
> > error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
> > fatal: Error building trees

I've trapped in a similar problem few days ago on windows using cygwin.

A simple "git status" told me an invalid object. Just doing git status corrupted the index file and only this. The reason was a wrong configured (unseen reconfigured from another script) mount via cygwin using "textmode" not "binary mode".

After remounting with "binary mode" a simple "git read-tree HEAD" repaired it. Just the index file had been corrupted.

Stefan
-- 
Stefan-W. Hahn                          It is easy to make things.
/ mailto:stefan.hahn@s-hahn.de /        It is hard to make things simple.			

Please note that according to the German law on data retention,
information on every electronic information exchange with me is
retained for a period of six months.
Bundesgesetzblatt:
http://www.bgblportal.de/BGBL/bgbl1f/bgbl107s3198.pdf
Linus Torvalds· Jul 14, 2009, 18:25 UTC · re: Linus Torvalds · lore

[PATCH] Improve on the 'invalid object' error message at commit time

Not that anybody should ever get it, but somebody did (probably because of a flaky filesystem, but whatever). And each time I see an error message that I haven't seen before, I decide that next time it will look better.

So this makes us write more relevant information about exactly which file ended up having issues with a missing object. Which will tell whether it was a tree object, for example, or just a regular file in the index (and which one).

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

Not a big deal. But since somebody actually saw this message, let's just make it more informative.

A lot of the "these can't happen unless you're seriously screwed" messages aren't very good, because there is little upside when doing development. But let's try to improve on them in case they happen in the future.

Not that this would make debugging much easier, but one thing that I started wondering about was whether the problem Florian saw was about one of the files he had done "git add" on, or whether it was a tree entry that was the result of "find_subtree()".

 cache-tree.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/cache-tree.c b/cache-tree.c
index 16a65df..d917437 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -329,7 +329,8 @@ static int update_one(struct cache_tree *it,
 			entlen = pathlen - baselen;
 		}
 		if (mode != S_IFGITLINK && !missing_ok && !has_sha1_file(sha1))
-			return error("invalid object %s", sha1_to_hex(sha1));
+			return error("invalid object %06o %s for '%.*s'",
+				mode, sha1_to_hex(sha1), entlen+baselen, path);
 
 		if (ce->ce_flags & CE_REMOVE)
 			continue; /* entry being removed */
Junio C Hamano· Jul 14, 2009, 19:43 UTC · re: Linus Torvalds · lore

Re: [PATCH] Improve on the 'invalid object' error message at commit time

Linus Torvalds <torvalds@linux-foundation.org> writes:
> ... one thing that I 
> started wondering about was whether the problem Florian saw was about one 
> of the files he had done "git add" on, or whether it was a tree entry that 
> was the result of "find_subtree()".
Same here.

Historically, cache-tree had issues that same corner case codepaths forgot invalidating subtrees when they should that could have led to kinds of bugs that are hard to diagnose. I think the API got harder to make that kind of mistakes while we squashed those bugs, though.

← back to recent threads