threads / discuss / 8033

git pull failure, truncated object

Subject: git pull failure, truncated object

## tl;dr

6 messages between May 8, 2007 and May 8, 2007.

replies: 5people: 3as markdown or json

Bill Lear· May 8, 2007, 14:28 UTC · lore

Our QA dude had a disk fill up a while back. Forgot about it. Went to pull during a nightly build. Got this:

% git pull remote: Generating pack... remote: Done counting 104 objects. remote: Result has 72 objects. remote: Deltifying 72 objects. remote: 100% (72/72) done Unpacking 72 objects remote: Total 72 (delta 46), reused 52 (delta 26) error: failed to read delta-pack base object 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d fatal: unpack-objects died with error code 1

He did a git-gc, twice, and retried.  Still failed.

So, he called me in and we tried to see if the server was acting up --- perhaps an NFS problem, as we've had those before, but got very different error messages. Watched the log file from git-daemon, and saw nothing. Finally we took a look at the local repos .git/objects/4b, and 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d was of length 0.

So, I looked in the man page of git-gc and thought to try --prune, as this was not an active repository. This worked, and then the pull did as well.

I'm wondering why git-gc did not at least warn us of this problem when we tried it. It appeared to us that git-gc gave our repo a clean bill of health, and so we turned our attention to the remote and investigated there, instead of continuing in the local repo.

Should we have tried git fsck instead in this case?

It might be nice to have a better error message in this case, one that locates the problem in the local repo.

If the failure is a short/bad (?) read of a local object, then it might be nice to say something like:

error: this repo's local object 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d
       in .git/objects/4b is corrupt.  You will need to run
       XXX to diagnose and YYY to repair this.

Or some such.

Bill

Shawn O. Pearce· May 8, 2007, 14:59 UTC · re: Bill Lear · lore

Re: git pull failure, truncated object

Bill Lear <rael@zopyra.com> wrote: ...

> error: failed to read delta-pack base object
> 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d
> fatal: unpack-objects died with error code 1

Danger Will Robinson! Danger! Your repository is not acting normally...

> So, I looked in the man page of git-gc and thought to try --prune,
> as this was not an active repository.  This worked, and then
> the pull did as well.

Don't run git-gc on a repository that is acting strangely, unless you have concluded that the correct course of action is to just repack the repository. (It rarely is, btw.) You could make things worse if a packfile contains a corrupt object and you have the same valid loose object; a gc would delete the valid object and keep the corrupt one.

Show 6 quoted lines

> I'm wondering why git-gc did not at least warn us of this problem when
> we tried it.  It appeared to us that git-gc gave our repo a clean bill
> of health, and so we turned our attention to the remote and
> investigated there, instead of continuing in the local repo.
> 
> Should we have tried git fsck instead in this case?

Yes, git-fsck is meant for checking the "health" of a repository. In this case it should have caught the 0 length object and told you that loose object was corrupt. A sure sign that you have a problem. If you have a loose object problem, `git prune` might fix it, as it did here, but only if that object isn't actually needed. A `git fsck` immediately after would tell you if the prune fixed the issue, or not.

-- 
Shawn.

Shawn O. Pearce· May 8, 2007, 15:05 UTC · re: Shawn O. Pearce · lore

Re: git pull failure, truncated object

"Shawn O. Pearce" <spearce@spearce.org> wrote:

Show 6 quoted lines

> Don't run git-gc on a repository that is acting strangely, unless
> you have concluded that the correct course of action is to just
> repack the repository.  (It rarely is, btw.)  You could make things
> worse if a packfile contains a corrupt object and you have the
> same valid loose object; a gc would delete the valid object and
> keep the corrupt one.

OK, my statement is a little blown-out-of-proportion. Its pretty hard these days to corrupt an object within a packfile such that we'll be able to reuse it during the repack that goes on in git-gc, but still actually have it be corrupt enough that the object is useless. The recent index version 2 work from Nico makes it even harder, as the index adds an additional checksum over the entire object header and body.

But still, even though the risk is pretty small, I think that running a destructive operation like git-gc in a repository that is not acting normally is a bad idea. You should try to diagnose and correct the issue before making further changes (or reorganizations) to that repository's contents.

-- 
Shawn.

Nicolas Pitre· May 8, 2007, 15:41 UTC · re: Bill Lear · lore

Re: git pull failure, truncated object

On Tue, 8 May 2007, Bill Lear wrote:

Show 17 quoted lines

> He did a git-gc, twice, and retried.  Still failed.
> 
> So, he called me in and we tried to see if the server was acting up
> --- perhaps an NFS problem, as we've had those before, but got very
> different error messages.  Watched the log file from git-daemon, and
> saw nothing.  Finally we took a look at the local repos
> .git/objects/4b, and 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d was of
> length 0.
> 
> So, I looked in the man page of git-gc and thought to try --prune,
> as this was not an active repository.  This worked, and then
> the pull did as well.
> 
> I'm wondering why git-gc did not at least warn us of this problem when
> we tried it.  It appeared to us that git-gc gave our repo a clean bill
> of health, and so we turned our attention to the remote and
> investigated there, instead of continuing in the local repo.

git-gc != git-fsck.

Nicolas

Bill Lear· May 8, 2007, 15:46 UTC · re: Nicolas Pitre · lore

Re: git pull failure, truncated object

On Tuesday, May 8, 2007 at 11:41:14 (-0400) Nicolas Pitre writes:

Show 21 quoted lines

>On Tue, 8 May 2007, Bill Lear wrote:
>
>> He did a git-gc, twice, and retried.  Still failed.
>> 
>> So, he called me in and we tried to see if the server was acting up
>> --- perhaps an NFS problem, as we've had those before, but got very
>> different error messages.  Watched the log file from git-daemon, and
>> saw nothing.  Finally we took a look at the local repos
>> .git/objects/4b, and 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d was of
>> length 0.
>> 
>> So, I looked in the man page of git-gc and thought to try --prune,
>> as this was not an active repository.  This worked, and then
>> the pull did as well.
>> 
>> I'm wondering why git-gc did not at least warn us of this problem when
>> we tried it.  It appeared to us that git-gc gave our repo a clean bill
>> of health, and so we turned our attention to the remote and
>> investigated there, instead of continuing in the local repo.
>
>git-gc != git-fsck.

Indeed, as is now clear to me. Would it be prudent to have git-gc run a quick git-fsck internally and warn if things are not in a kosher state?

Bill

Nicolas Pitre· May 8, 2007, 17:51 UTC · re: Bill Lear · lore

Re: git pull failure, truncated object

On Tue, 8 May 2007, Bill Lear wrote:

Show 26 quoted lines

> On Tuesday, May 8, 2007 at 11:41:14 (-0400) Nicolas Pitre writes:
> >On Tue, 8 May 2007, Bill Lear wrote:
> >
> >> He did a git-gc, twice, and retried.  Still failed.
> >> 
> >> So, he called me in and we tried to see if the server was acting up
> >> --- perhaps an NFS problem, as we've had those before, but got very
> >> different error messages.  Watched the log file from git-daemon, and
> >> saw nothing.  Finally we took a look at the local repos
> >> .git/objects/4b, and 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d was of
> >> length 0.
> >> 
> >> So, I looked in the man page of git-gc and thought to try --prune,
> >> as this was not an active repository.  This worked, and then
> >> the pull did as well.
> >> 
> >> I'm wondering why git-gc did not at least warn us of this problem when
> >> we tried it.  It appeared to us that git-gc gave our repo a clean bill
> >> of health, and so we turned our attention to the remote and
> >> investigated there, instead of continuing in the local repo.
> >
> >git-gc != git-fsck.
> 
> Indeed, as is now clear to me.  Would it be prudent to have git-gc
> run a quick git-fsck internally and warn if things are not in a kosher
> state?

No. git-fsck is a potentially expensive operation and it is up to you to remember that git-gc isn't about repository sanity. git-gc only repacks things and if it encounters an object that is corrupt it will abort and leave your object store as is. In your case the corrupted object wasn't one that needed to be repacked which explains why git-gc succeeded.

Nicolas