Git clone error

Hi all,

I'm a new git user (actually a darcs convert) and am running into a weird problem on a small and simple repository. The error I see is:

    git[14] > git clone /Volumes/work/scripts/
    Initialized empty Git repository in /tmp/git/scripts/.git/
    remote: Generating pack...
    Done counting 80 objects.
    remote: error: unable to unpack b28b949a1a3c8eb37ca6eefd024508fa8b253429
header
    fatal: unable to get type of object b28b949a1a3c8eb37ca6error:
git-upload-pack: git-pack-objects died with error.
    fatal: git-upload-pack: aborting due to possible repository corruption
on the remote side.
    remote: aborting due to possible repository corruption on the remote
side.
    fatal: early EOF
    fatal: index-pack died with error code 128
    fetch-pack from '/Volumes/work/scripts/.git' failed.

I have many git repos on my system (one for configuration files, one for scripts, various school projects, etc.) and this is the only one that has been problematic thus far. As you can see I am trying to clone a local repo -- there should be no network problems here. As I said, the repo is small:

    scripts[19] > find . -type f | wc -l
    118

(which of course includes all of git's files.)

$ git --version git version 1.5.2.4

$ uname -a Darwin 8.10.1 Darwin Kernel Version 8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386 i386 i386

I have little experience with git, but if you need any more info, let me know. Any suggestion on what might be the problem, and how to fix it?

-Denis

"Denis Bueno" <denbuen@sandia.gov> writes:

First thing to check would be...

        $ cd /Volumes/work/scripts
        $ git fsck --full

On Tue, 31 Jul 2007, Denis Bueno wrote:

Well, it says so, but the most likely issue really is that there is "corruption on the remote side".

Please do

	cd /Volumes/work/scripts
	git fsck --full

which I would guess will almost certainly talk about some kind of problems with that object "b28b949a1a3c8eb37ca6eefd024508fa8b253429". And possibly more.

The "unable to unpack .. header" problem would at a guess be a totally corrupted loose object. You should have a file named

	.git/objects/b2/8b949a1a3c8eb37ca6eefd024508fa8b253429

and it sounds like that file is corrupt. So far, apart from a CRLF conversion bug (that you wouldn't have triggered on OS X anyway), I think every single time we've seen that, it's been a real disk or memory corruption.

Trying to restore corrupt objects can be quite hard, since git stores them in a compressed format, and so git single-bit errors are very detectable (that's kind of the point of having cryptographically secure hashes!), but they are very much not fixable, unless you can find the original data that *resulted* in that object some way (in another clone of the git repository, or in a backup).

There are certainly ways to figure out what that object _should_ be, though. For example, if that object is the only corrupted entry, and it's a blob (ie pure data object), what you can generally do is still use the repo (as long as you avoid that particular blob), and do things like

	git log --raw -r --no-abbrev

and you'll see the git history with all blobs named with their SHA1's. Then you can just search for that b28b949.. name, and you'll see exactly which file (in which version) it was, and if you can just find a backup of that file (or re-create it *exactly* from previous versions and your memory of it), you can re-generate the git object and thus save your repository.

Of course, a much easier option tends to be to just have a backup repository that has that object in it, and then you can literally just copy that b28b949 object over.

In short: git basically guarantees that it will *find* all corruption, but it doesn't do backups for you. Backups are easy to do (cloning), and git also makes incremental backups trivial (ie just do "git fetch" or "git push"), but git won't do backups for you unless you ask for them that way.

			Linus

On 07/31/2007 20:38, "Linus Torvalds" <torvalds@linux-foundation.org> wrote:

Indeed:

    scripts[10] > git fsck --full
    error: b28b949a1a3c8eb37ca6eefd024508fa8b253429: object corrupt or
missing
    missing blob b28b949a1a3c8eb37ca6eefd024508fa8b253429

Fortunately, it's the only one.

    scripts[11] > git log --raw -r --no-abbrev | grep
b28b949a1a3c8eb37ca6eefd024508fa8b253429
    :100755 100755 b28b949a1a3c8eb37ca6eefd024508fa8b253429
9dbd5bb59198ab59e1850f838f2ed27c77364cde M      condor/condor-uninstall.sh
    :000000 100755 0000000000000000000000000000000000000000
b28b949a1a3c8eb37ca6eefd024508fa8b253429 A      condor/condor-uninstall.sh

When you say "re-generate", do you mean `git add <file> ; git commit <file>`? Or something a bit more clever? I suspect you actually meant the latter, since you suggest recreating it *exactly*.

If I just recreate a version I'm happy with, can I add that to the repo and go from there?

No such luck, but I'll back up my repos in the future. Any hint on what might have caused this kind of corruption? That repo was created on my laptop and only edited there; it's not a clone from some other machine. It's possible that I have Ctrl-C'd some git operation in the past -- could that have caused it?

Is there anything (e.g. trapping signals if a blob is being written until it's done, or backing out of the current blob on SIG{INT,HUP}) that could make this kind of corruption less easy to encounter? I ask because I have used VC systems (i.e. CVS, SVN, darcs, and now git) for 4+ years (not long by any means, but long enough) and the only repository corruption I had ever encountered was after I went mucking about in a CVS repo's internal representation -- and I deserved that.

                      Denis
--
"Dealing with failure is easy: Work hard to improve. Success is also easy to
handle: You've solved the wrong problem. Work hard to improve." -- Perlis

On Wed, 1 Aug 2007, Denis Bueno wrote:

Ok. That's the "easy" case, although the fact that the file went missing in the first place is certainly worrisome.

I think you said you were using OS X - have you seen crashes or are you using something like the experimental ZFS support?

Memory and disk corruption is also a possibility, so running fsck (I have *no* idea how you do that under OS X, but I bet others on the list know) is a good idea.

Also, *if* you actually have the file

	.git/objects/b2/8b949a1a3c8eb37ca6eefd024508fa8b253429

and it's not just missing entirely, please move it out, ie do something like

	mv .git/objects/b2/8b949a1a3c8eb37ca6eefd024508fa8b253429 ~/corrupt-object

You want to move it away from the .git/objects directory in any case, because if it exists (but is corrupted), git will never overwrite it with the non-corrupted version. And you don't want to just delete it, because I'd love to see exactly *what* the corruption was.

Ok, great. So it's the very first version of "condor/condor-uninstall.sh".

If you can find that original version, great. If not, you can get the *second* version by doing

	git cat-file blob 9dbd5bb59198ab59e1850f838f2ed27c77364cde > second-version

and if you remember what your changes were, try to just edit that back to the exact thing it was in the first version, if you can (note that by "exact", I do mean *exact*. Whitespace and all).

Actually you can literally just do 

	git add my-regenerated-file

and it would do the right thing. But it also changes your index, so a better way would often be to do

	git hash-object -w <filename>

which will take any random file, and hash it into the object database.

So if you were able to re-generate the original, you should now have a *non*corrupted .git/objects/b2/8b949a1a3c8eb37ca6eefd024508fa8b253429 file, and git-fsck will be happy again!

And after that, I'd really like to see both the corrupt and the non-corrupt object, to try to figure out what the corruption is.

Well, it's not so much a version _you_ are happy with: you'd have to be able to re-create the exact old version (with the exact same SHA1), in order for git to be happy.

No, things like Ctrl-C are safe: we create new objects by writing them to a temporary file, and then using an atomic rename() operation on them (actually, a "link() + unlink()", but we fall back to "rename()" if that fails - some filesystems cannot link files between directories).

So the only way to get corruption is literally with a broken filesystem, or memory corruption. Of course, if the OS's "rename()" is broken, and isn't atomic (ie if it's "copy + delete" operation).

Of course, git bugs could do this, but quite frankly, this is (a) very simple code, and (b) we really haven't had bugs in this area.

Git does end up finding corruption that other VCS's don't. The strong hashes and the fact that fsck (and, in fact, even simple operations like clone) will check every single hash, means that single-bit corruption gets noticed (and is a fatal error) every time. Other SCM's? Not so much.

			Linus

On Wed, 1 Aug 2007, Linus Torvalds wrote:

Btw, if you really cannot re-generate it, you'd basically need to create a whole new git archive without that blob (or basically with that blob replaced by another version).

We don't have wonderfully good support for that, because, quite frankly, we've not had this happen before. I think every time before, people have had the blob in some other copy of their git archive.

But the thing to use is "git filter-branch", which can take a git history and munge it arbitrarily. It would be the "tree-filter" that you'd use to replace that one blob that you cannot regenerate with another (ie you might decide to just replace the original version of the file with that *second* version, and regenerate the tree that way).

I'm cc'ing Dscho explicitly to see if he can help you with the exact syntax, and maybe we could even make this into a user-manual entry about how to handle corruption. I don't think we have anything in the documentation about this - we only cover the trivial cases where the objects are all good, but you've lost the pointers into it because you removed a branch by mistake or something.

		Linus

Hi,

On Wed, 1 Aug 2007, Linus Torvalds wrote:

I doubt it would be that easy; the tree filter expects a _valid_ tree, which it checks out, and only then you can filter it.

But this is what I would do if I had the problem: I would try to create a state which is as close to the corrupt revision as possible, use a graft to replace the initial commit with that revision, and rewrite the branch. I'd probably start by doing something like this:

	$ git symbolic-ref HEAD refs/heads/recreate-first && rm .git/index
	$ git ls-tree -r --name-only <initial-commit> |
		grep -v "^condor/condor-uninstall.sh$" |
		xargs git checkout <initial-commit>
	$ git checkout <second-commit> condor/condor-uninstall.sh
	[possibly some minor hacking on the latter file to make it work]
	$ git commit -c <initial-commit>

This starts a new branch, which has no ancestor yet. It takes all files except the corrupt one from the first commit, and a replacement of the corrupt one from the second commit. Then it takes all the meta-data from the first commit to create a replacement for the first commit.

Now I'd graft the second commit onto the first:

	$ git rev-parse <second-commit> recreate-first >> .git/info/grafts

This explicitely tells git to disregard the parent information given in <second-commit>, and to pretend that the recreated commit is its parent.

	BEWARE! Grafts are a powerful weapon, and you can spend tons of 
	time looking for an error, when all you did was grafting parts of 
	the history onto other parts of the history.  For example, I was 
	searching several nights for the bug preventing me to push one of 
	my repos, and it always gave an "unpacking" error.  Once I moved 
	the grafts out of the way, it miraculously worked.

After that, just call git-filter-branch (not yet released, but will probably be part of 1.5.3):

	$ git filter-branch <branchname>

Many things can go wrong through that process, so you want to make sure that all steps actually worked as intended (especially since I tested none of this, but wrote it in the mail program)!

Hth, Dscho

On 08/01/2007 11:17, "Johannes Schindelin" <Johannes.Schindelin@gmx.de> wrote:

Wow. `commit' and `checkout' are the only two commands that I have ever heard of in that sequence.

How difficult would it be to create a new git repo which is exactly the same minus the initial condor-uninstall.sh commit? That is, just to pretend the initial import of condor-uninstall.sh never existed, and use the second commit of the old repo the first commit of the new, and preserve the rest of the history of the entire repo? Or, to remove both condor-uninstall.sh commits from the history -- deleting that history altogether -- and add it back as a completely new file?

If I get enough courage, I'll attempt to understand exactly what it is you're doing in the commands above and try it out on (a copy of) the repo. My willingness to part with the current history and just reinitialise the repo with its current contents increases with time, though.

The real problem is that I'm sure I have no idea exactly what the first version of the file looked like, and how it differed from the next one.

                      Denis
--
"If we wish to count lines of code, we should not regard them as lines
produced but as lines spent." -- Edsger Dijkstra

Hi,

On Wed, 1 Aug 2007, Denis Bueno wrote:

That would be even easier. Just graft "nothingness" as parent of the second commit:

	$ git rev-parse <second-commit> >> .git/info/grafts

But of course, you should use filter-branch nevertheless, since you would have to do the same hack in _every_ repository.

Ciao, Dscho

On 08/01/2007 15:12, "Johannes Schindelin" <Johannes.Schindelin@gmx.de> wrote:

I must be misunderstanding:

    scripts[] > git fsck --full
    error: b28b949a1a3c8eb37ca6eefd024508fa8b253429: object corrupt or
missing
    missing blob b28b949a1a3c8eb37ca6eefd024508fa8b253429

    # b2... should fill in your <second-commit>?

    scripts[30] > git rev-parse b28b949a1a3c8eb37ca6eefd024508fa8b253429 >>
.git/info/grafts

    scripts[31] > git fsck --full
    error: b28b949a1a3c8eb37ca6eefd024508fa8b253429: object corrupt or
missing
    missing blob b28b949a1a3c8eb37ca6eefd024508fa8b253429

If I try to clone the repo, I get the same error.

                      Denis
--
"Program testing can be used to show the presence of bugs, but never to show
their absence." -- Edsger Dijkstra

Hi,

On Thu, 2 Aug 2007, Denis Bueno wrote:

Ah no. Linus diagnosed that this blob is your condor-uninstall script in your initial commit. You should be able to get at it by calling "git log --root" and taking the last commit. Just to make sure, "git show <first-commit>:bla/condor-uninstall.sh" should fail miserably, mentioning a corrupt blob.

The second commit I was referring to is the second last commit in the "git log --root" output.

Hth, Dscho

On Thu, 2 Aug 2007, Johannes Schindelin wrote:

Well, I'm not sure the commit was the initial one, but it was the one that *introduced* that file. It *may* be the initial one, of course (and quite likely is, for an import), but..

		Linus

On Aug 1, 2007, at 6:19 PM, Linus Torvalds wrote:

AppleJack is the easiest way I know of:

http://applejack.sourceforge.net/

	Steffen