{"thread":{"id":"32260","subject":"does a successful 'git gc' imply 'git fsck'","startedAt":"2012-12-02T02:31:49Z","lastAt":"2012-12-03T14:06:21Z","messageCount":7,"participants":["Sitaram Chamarty","Shawn Pearce","Junio C Hamano","Matthieu Moy"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"204407","messageId":"CAMK1S_iBq1ReGkjuy2UBENSQXOWyKj2ZzSCcg7Z89FVtVL2wMw@mail.gmail.com","threadId":"32260","inReplyTo":null,"subject":"does a successful 'git gc' imply 'git fsck'","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2012-12-02T02:31:49Z","receivedAt":"2012-12-02T02:31:49Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"Hi,\n\nBackground: I have a situation where I have to fix up a few hundred\nrepos in terms of 'git gc' (the auto gc seems to have failed in many\ncases; they have far more than 6700 loose objects). I also found some\ncorrupted objects in some cases that prevent the gc from completing.\n\nI am running \"git gc\" followed by \"git fsck\". The majority of the\nrepos I have worked through so far appear to be fine, but in the\nlarger repos (upwards of 2-3 GB) the git fsck is taking almost 5 times\nlonger than the 'gc'.\n\nIf I could assume that a successful 'git gc' means an fsck is not\nneeded, I'd save a lot of time. Hence my question.\n\n-- \nSitaram\n"},{"id":"204413","messageId":"CAJo=hJvHeA4g1fJZXm9VxpdvMFoHeEJW_r5YtXTDACEp09Xm_Q@mail.gmail.com","threadId":"32260","inReplyTo":"CAMK1S_iBq1ReGkjuy2UBENSQXOWyKj2ZzSCcg7Z89FVtVL2wMw@mail.gmail.com","subject":"Re: does a successful 'git gc' imply 'git fsck'","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2012-12-02T04:28:34Z","receivedAt":"2012-12-02T04:28:34Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sat, Dec 1, 2012 at 6:31 PM, Sitaram Chamarty wrote:\n> Background: I have a situation where I have to fix up a few hundred\n> repos in terms of 'git gc' (the auto gc seems to have failed in many\n> cases; they have far more than 6700 loose objects). I also found some\n> corrupted objects in some cases that prevent the gc from completing.\n>\n> I am running \"git gc\" followed by \"git fsck\". The majority of the\n> repos I have worked through so far appear to be fine, but in the\n> larger repos (upwards of 2-3 GB) the git fsck is taking almost 5 times\n> longer than the 'gc'.\n>\n> If I could assume that a successful 'git gc' means an fsck is not\n> needed, I'd save a lot of time. Hence my question.\n\nNot really. For example fsck verifies that every blob when\ndecompressed and fully inflated matches its SHA-1. gc only checks\nconnectivity of the commit and tree graph by making sure every object\nwas accounted for. But when creating the output pack it only verifies\na CRC-32 was correct when copying the bits from the source to the\ndestination, it does not verify that the data decompresses and matches\nthe SHA-1 it should match.\n\nSo it depends on what level of check you need to feel safe.\n"},{"id":"204417","messageId":"CAMK1S_gUB6tpNQTLGTPPAvQ87+mX2HGiN1xpA7KoH33MX9tFBw@mail.gmail.com","threadId":"32260","inReplyTo":"CAJo=hJvHeA4g1fJZXm9VxpdvMFoHeEJW_r5YtXTDACEp09Xm_Q@mail.gmail.com","subject":"Re: does a successful 'git gc' imply 'git fsck'","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2012-12-02T08:46:33Z","receivedAt":"2012-12-02T08:46:33Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Sun, Dec 2, 2012 at 9:58 AM, Shawn Pearce wrote:\n> On Sat, Dec 1, 2012 at 6:31 PM, Sitaram Chamarty wrote:\n>> Background: I have a situation where I have to fix up a few hundred\n>> repos in terms of 'git gc' (the auto gc seems to have failed in many\n>> cases; they have far more than 6700 loose objects). I also found some\n>> corrupted objects in some cases that prevent the gc from completing.\n>>\n>> I am running \"git gc\" followed by \"git fsck\". The majority of the\n>> repos I have worked through so far appear to be fine, but in the\n>> larger repos (upwards of 2-3 GB) the git fsck is taking almost 5 times\n>> longer than the 'gc'.\n>>\n>> If I could assume that a successful 'git gc' means an fsck is not\n>> needed, I'd save a lot of time. Hence my question.\n>\n> Not really. For example fsck verifies that every blob when\n> decompressed and fully inflated matches its SHA-1. gc only checks\n\nOK that makes sense. After I posted I happened to check using strace\nand kinda guessed this from what I saw, but it's nice to have\nconfirmation.\n\n> connectivity of the commit and tree graph by making sure every object\n> was accounted for. But when creating the output pack it only verifies\n> a CRC-32 was correct when copying the bits from the source to the\n> destination, it does not verify that the data decompresses and matches\n> the SHA-1 it should match.\n>\n> So it depends on what level of check you need to feel safe.\n\nYup; thanks.\n\nAll the repos my internal client manages are mirrored in multiple\nplaces, and they set (or were at least told to set, heh!)\nreceive.fsckObjects so the lesser check is fine in most cases.\n\n-- \nSitaram\n"},{"id":"204419","messageId":"7vlidgls09.fsf@alter.siamese.dyndns.org","threadId":"32260","inReplyTo":"CAMK1S_iBq1ReGkjuy2UBENSQXOWyKj2ZzSCcg7Z89FVtVL2wMw@mail.gmail.com","subject":"Re: does a successful 'git gc' imply 'git fsck'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-12-02T09:31:34Z","receivedAt":"2012-12-02T09:31:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sitaram Chamarty writes:\n\n> If I could assume that a successful 'git gc' means an fsck is not\n> needed, I'd save a lot of time. Hence my question.\n\nWhen it does \"repack -a\", it at least scans the whole history so you\nwould be sure that all the commits and trees are readable for the\npurpose of enumerating the objects referred by them (and a bit flip\nin them will likely be noticed by zlib inflation).\n\nBut a \"gc\" does not necessarily run \"repack -a\" when it does not see\ntoo many pack files, so it can end up scanning only the surface of\nthe history to collect the recently created loose objects into a\npack, and stop its traversal without going into existing packfiles.\n"},{"id":"204445","messageId":"CAMK1S_gNgQORT8XRSMn0OoKRGqosoEB3_Jv84s0fQ9cpsHqMDQ@mail.gmail.com","threadId":"32260","inReplyTo":"7vlidgls09.fsf@alter.siamese.dyndns.org","subject":"Re: does a successful 'git gc' imply 'git fsck'","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2012-12-03T13:14:11Z","receivedAt":"2012-12-03T13:14:11Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Sun, Dec 2, 2012 at 3:01 PM, Junio C Hamano wrote:\n> Sitaram Chamarty writes:\n>\n>> If I could assume that a successful 'git gc' means an fsck is not\n>> needed, I'd save a lot of time. Hence my question.\n>\n> When it does \"repack -a\", it at least scans the whole history so you\n> would be sure that all the commits and trees are readable for the\n> purpose of enumerating the objects referred by them (and a bit flip\n> in them will likely be noticed by zlib inflation).\n>\n> But a \"gc\" does not necessarily run \"repack -a\" when it does not see\n> too many pack files, so it can end up scanning only the surface of\n> the history to collect the recently created loose objects into a\n> pack, and stop its traversal without going into existing packfiles.\n\nThanks; I'd missed this nuance as well...\n\n-- \nSitaram\n"},{"id":"204446","messageId":"vpq4nk3uu39.fsf@grenoble-inp.fr","threadId":"32260","inReplyTo":"7vlidgls09.fsf@alter.siamese.dyndns.org","subject":"Re: does a successful 'git gc' imply 'git fsck'","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-12-03T13:46:18Z","receivedAt":"2012-12-03T13:46:18Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> But a \"gc\" does not necessarily run \"repack -a\" when it does not see\n> too many pack files, so it can end up scanning only the surface of\n> the history to collect the recently created loose objects into a\n> pack, and stop its traversal without going into existing packfiles.\n\nIsn't that the behavior of \"git gc --auto\", not plain \"git gc\" ?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"204447","messageId":"7vpq2rtele.fsf@alter.siamese.dyndns.org","threadId":"32260","inReplyTo":"vpq4nk3uu39.fsf@grenoble-inp.fr","subject":"Re: does a successful 'git gc' imply 'git fsck'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-12-03T14:06:21Z","receivedAt":"2012-12-03T14:06:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> Junio C Hamano writes:\n>\n>> But a \"gc\" does not necessarily run \"repack -a\" when it does not see\n>> too many pack files, so it can end up scanning only the surface of\n>> the history to collect the recently created loose objects into a\n>> pack, and stop its traversal without going into existing packfiles.\n>\n> Isn't that the behavior of \"git gc --auto\", not plain \"git gc\" ?\n\nTrue; I missed that Sitaram was running \"gc\" manually.\n"}]}