threads / discuss / 32260

does a successful 'git gc' imply 'git fsck'

Subject: does a successful 'git gc' imply 'git fsck'

## tl;dr

7 messages between Dec 2, 2012 and Dec 3, 2012.

replies: 6people: 4as markdown or json

Sitaram Chamarty· Dec 2, 2012, 02:31 UTC · lore
Hi,
Background: I have a situation where I have to fix up a few hundred
repos in terms of 'git gc' (the auto gc seems to have failed in many
cases; they have far more than 6700 loose objects).  I also found some
corrupted objects in some cases that prevent the gc from completing.

I am running "git gc" followed by "git fsck". The majority of the repos I have worked through so far appear to be fine, but in the larger repos (upwards of 2-3 GB) the git fsck is taking almost 5 times longer than the 'gc'.

If I could assume that a successful 'git gc' means an fsck is not needed, I'd save a lot of time. Hence my question.

-- 
Sitaram
Shawn Pearce· Dec 2, 2012, 04:28 UTC · re: Sitaram Chamarty · lore

Re: does a successful 'git gc' imply 'git fsck'

On Sat, Dec 1, 2012 at 6:31 PM, Sitaram Chamarty <sitaramc@gmail.com> wrote:
Show 12 quoted lines
> Background: I have a situation where I have to fix up a few hundred
> repos in terms of 'git gc' (the auto gc seems to have failed in many
> cases; they have far more than 6700 loose objects).  I also found some
> corrupted objects in some cases that prevent the gc from completing.
>
> I am running "git gc" followed by "git fsck".  The majority of the
> repos I have worked through so far appear to be fine, but in the
> larger repos (upwards of 2-3 GB) the git fsck is taking almost 5 times
> longer than the 'gc'.
>
> If I could assume that a successful 'git gc' means an fsck is not
> needed, I'd save a lot of time.  Hence my question.

Not really. For example fsck verifies that every blob when decompressed and fully inflated matches its SHA-1. gc only checks connectivity of the commit and tree graph by making sure every object was accounted for. But when creating the output pack it only verifies a CRC-32 was correct when copying the bits from the source to the destination, it does not verify that the data decompresses and matches the SHA-1 it should match.

So it depends on what level of check you need to feel safe.
Sitaram Chamarty· Dec 2, 2012, 08:46 UTC · re: Shawn Pearce · lore

Re: does a successful 'git gc' imply 'git fsck'

On Sun, Dec 2, 2012 at 9:58 AM, Shawn Pearce <spearce@spearce.org> wrote:
Show 16 quoted lines
> On Sat, Dec 1, 2012 at 6:31 PM, Sitaram Chamarty <sitaramc@gmail.com> wrote:
>> Background: I have a situation where I have to fix up a few hundred
>> repos in terms of 'git gc' (the auto gc seems to have failed in many
>> cases; they have far more than 6700 loose objects).  I also found some
>> corrupted objects in some cases that prevent the gc from completing.
>>
>> I am running "git gc" followed by "git fsck".  The majority of the
>> repos I have worked through so far appear to be fine, but in the
>> larger repos (upwards of 2-3 GB) the git fsck is taking almost 5 times
>> longer than the 'gc'.
>>
>> If I could assume that a successful 'git gc' means an fsck is not
>> needed, I'd save a lot of time.  Hence my question.
>
> Not really. For example fsck verifies that every blob when
> decompressed and fully inflated matches its SHA-1. gc only checks

OK that makes sense. After I posted I happened to check using strace and kinda guessed this from what I saw, but it's nice to have confirmation.

Show 7 quoted lines
> connectivity of the commit and tree graph by making sure every object
> was accounted for. But when creating the output pack it only verifies
> a CRC-32 was correct when copying the bits from the source to the
> destination, it does not verify that the data decompresses and matches
> the SHA-1 it should match.
>
> So it depends on what level of check you need to feel safe.
Yup; thanks.

All the repos my internal client manages are mirrored in multiple places, and they set (or were at least told to set, heh!) receive.fsckObjects so the lesser check is fine in most cases.

-- 
Sitaram
Junio C Hamano· Dec 2, 2012, 09:31 UTC · re: Sitaram Chamarty · lore

Re: does a successful 'git gc' imply 'git fsck'

Sitaram Chamarty <sitaramc@gmail.com> writes:
> If I could assume that a successful 'git gc' means an fsck is not
> needed, I'd save a lot of time.  Hence my question.

When it does "repack -a", it at least scans the whole history so you would be sure that all the commits and trees are readable for the purpose of enumerating the objects referred by them (and a bit flip in them will likely be noticed by zlib inflation).

But a "gc" does not necessarily run "repack -a" when it does not see too many pack files, so it can end up scanning only the surface of the history to collect the recently created loose objects into a pack, and stop its traversal without going into existing packfiles.

Sitaram Chamarty· Dec 3, 2012, 13:14 UTC · re: Junio C Hamano · lore

Re: does a successful 'git gc' imply 'git fsck'

On Sun, Dec 2, 2012 at 3:01 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 14 quoted lines
> Sitaram Chamarty <sitaramc@gmail.com> writes:
>
>> If I could assume that a successful 'git gc' means an fsck is not
>> needed, I'd save a lot of time.  Hence my question.
>
> When it does "repack -a", it at least scans the whole history so you
> would be sure that all the commits and trees are readable for the
> purpose of enumerating the objects referred by them (and a bit flip
> in them will likely be noticed by zlib inflation).
>
> But a "gc" does not necessarily run "repack -a" when it does not see
> too many pack files, so it can end up scanning only the surface of
> the history to collect the recently created loose objects into a
> pack, and stop its traversal without going into existing packfiles.
Thanks; I'd missed this nuance as well...
-- 
Sitaram
Matthieu Moy· Dec 3, 2012, 13:46 UTC · re: Junio C Hamano · lore

Re: does a successful 'git gc' imply 'git fsck'

Junio C Hamano <gitster@pobox.com> writes:
> But a "gc" does not necessarily run "repack -a" when it does not see
> too many pack files, so it can end up scanning only the surface of
> the history to collect the recently created loose objects into a
> pack, and stop its traversal without going into existing packfiles.
Isn't that the behavior of "git gc --auto", not plain "git gc" ?
-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
Junio C Hamano· Dec 3, 2012, 14:06 UTC · re: Matthieu Moy · lore

Re: does a successful 'git gc' imply 'git fsck'

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:
Show 8 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
>
>> But a "gc" does not necessarily run "repack -a" when it does not see
>> too many pack files, so it can end up scanning only the surface of
>> the history to collect the recently created loose objects into a
>> pack, and stop its traversal without going into existing packfiles.
>
> Isn't that the behavior of "git gc --auto", not plain "git gc" ?
True; I missed that Sitaram was running "gc" manually.

← back to recent threads