threads / discuss / 16025

Verifying the whole repository

Subject: Verifying the whole repository

## tl;dr

4 messages between Oct 23, 2008 and Oct 23, 2008.

replies: 3people: 3as markdown or json

Alex Bennee· Oct 23, 2008, 13:59 UTC · lore

Hi,

While I was debugging a crash in parsecvs while converting our CVS repository I discovered it was because one of the CVS files had become corrupted (truncated). This is a problem I've had before with RCS based files which are prone to silent corruption that you won't notice until you try and checkout an old revision of the file.

As git is fundamentally hash based it's a lot easier to determine the health of the repository but I wonder if it's possible for silent corruption to creep in which won't be noticed until you try and checkout a historical commit of the tree. I notice there is a git-verify-pack command that checks the pack files are OK. Do any of the other commands implicitly ensure all objects in the repo are correct and valid? git-gc?

Are there any other parts of the .git metadata that are crucial or is it enough to say if all objects and packs match their hashes you have all the information you may need to recover an arbitrary revision of the repo?

-- 
Alex, homepage: http://www.bennee.com/~alex/

David Symonds· Oct 23, 2008, 14:05 UTC · re: Alex Bennee · lore

Re: Verifying the whole repository

On Thu, Oct 23, 2008 at 6:59 AM, Alex Bennee <kernel-hacker@bennee.com> wrote:

Show 7 quoted lines

> As git is fundamentally hash based it's a lot easier to determine the
> health of the repository but I wonder if it's possible for silent
> corruption to creep in which won't be noticed until you try and
> checkout a historical commit of the tree. I notice there is a
> git-verify-pack command that checks the pack files are OK. Do any of
> the other commands implicitly ensure all objects in the repo are
> correct and valid? git-gc?

Try:  git fsck --full --strict

Dave.

Alex Bennee· Oct 23, 2008, 14:14 UTC · re: David Symonds · lore

Re: Verifying the whole repository

On Thu, Oct 23, 2008 at 3:05 PM, David Symonds <dsymonds@gmail.com> wrote:

Show 6 quoted lines

> On Thu, Oct 23, 2008 at 6:59 AM, Alex Bennee <kernel-hacker@bennee.com> wrote:
>> Do any of
>> the other commands implicitly ensure all objects in the repo are
>> correct and valid? git-gc?
>
> Try:  git fsck --full --strict

Ahh, I forgot that git was written by a filesystem guy ;-)

Thanks.

-- 
Alex, homepage: http://www.bennee.com/~alex/

Shawn O. Pearce· Oct 23, 2008, 14:28 UTC · re: Alex Bennee · lore

Re: Verifying the whole repository

Alex Bennee <kernel-hacker@bennee.com> wrote:

Show 7 quoted lines

> As git is fundamentally hash based it's a lot easier to determine the
> health of the repository but I wonder if it's possible for silent
> corruption to creep in which won't be noticed until you try and
> checkout a historical commit of the tree. I notice there is a
> git-verify-pack command that checks the pack files are OK. Do any of
> the other commands implicitly ensure all objects in the repo are
> correct and valid? git-gc?

As David pointed out, git fsck can be used to verify all of the hashes, but git-gc also does a quick sanity check using a CRC code when it copies data from one pack to another pack.

Unlike CVS Git has a write-once, read-many mentality, so with the exception of git gc (err, actually the git repack it calls) git never modifies an existing file. That really helps to reduce the risk of corruption.

If you never do a gc or fsck operation (but still use say commit or push into the repository) then yes, silent corruption can still sneak up on you in the form of disk block corruption.

> Are there any other parts of the .git metadata that are crucial or is
> it enough to say if all objects and packs match their hashes you have
> all the information you may need to recover an arbitrary revision of
> the repo?

Don't forget about the loose objects under .git/objects/?? but otherwise yes, you just need the object data. The refs under .git/refs are also useful, but the tips can be recovered if the refs space is lost by "git fsck --unreachable".

-- 
Shawn.