{"thread":{"id":"37740","subject":"fsck option to remove corrupt objects - why/why not?","startedAt":"2014-10-16T00:13:34Z","lastAt":"2014-10-16T16:36:34Z","messageCount":6,"participants":["Ben Aveling","Johan Herland","Matthieu Moy","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"250739","messageId":"543F0DAE.2050205@optusnet.com.au","threadId":"37740","inReplyTo":"20141015234637.9B4FC781EFB@mail110.syd.optusnet.com.au","subject":"fsck option to remove corrupt objects - why/why not?","fromName":"Ben Aveling","fromEmail":"bena.001@optusnet.com.au","sentAt":"2014-10-16T00:13:34Z","receivedAt":"2014-10-16T00:13:34Z","isPatch":false,"sender":{"key":"bena.001@optusnet.com.au","avatar":null},"body":"On 14/10/2014 19:21, Jeff King wrote:\n> On Mon, Oct 13, 2014 at 09:37:27AM +1100, Ben Aveling wrote:\n>> A question about fsck - is there a reason it doesn't have an option to\n>> delete bad objects?\n> If the objects are reachable, then deleting them would create other big\n> problems (i.e., we would be breaking the object graph!).\n\nThe man page for fsck advises:\n\n    \"Any corrupt objects you will have to find in backups or other\n    archives (i.e., you can just remove them and do an /rsync/ with some\n    other site in the hopes that somebody else has the object you have\n    corrupted).\"\n\n\nAnd that seems sensible to me - the object is corrupt, it is unusable, \nthe object graph is already broken, we already have big problems, \nremoving the corrupt object(s) doesn't create any new problems, and it \nallows the possibility that the damaged objects can be restored.\n\nI ask because I have a corrupt repository, and every time I run fsck, it \nreports one corrupt object, then stops. I could write a script to \nrepeatedly call fsck and then remove the next corrupt object, but it \nraises the question for me; could it make sense to extend fsck with the \noption to do to the removes? Or even better, do the removes and then do \nthe necessary [r]sync, assuming the user has another repository that has \na good copy of the bad objects, which in this case I do.\n\nRegards, Ben\n"},{"id":"250746","messageId":"CALKQrgda8mVbqP5=Ag8juN9HMQp7iQ9eDJETfRJe1b0taAFGkg@mail.gmail.com","threadId":"37740","inReplyTo":"543F0DAE.2050205@optusnet.com.au","subject":"Re: fsck option to remove corrupt objects - why/why not?","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2014-10-16T09:04:04Z","receivedAt":"2014-10-16T09:04:04Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Thu, Oct 16, 2014 at 2:13 AM, Ben Aveling <bena.001@optusnet.com.au> wrote:\n> On 14/10/2014 19:21, Jeff King wrote:\n>> On Mon, Oct 13, 2014 at 09:37:27AM +1100, Ben Aveling wrote:\n>>> A question about fsck - is there a reason it doesn't have an option to\n>>> delete bad objects?\n>>\n>> If the objects are reachable, then deleting them would create other big\n>> problems (i.e., we would be breaking the object graph!).\n>\n>\n> The man page for fsck advises:\n>\n>    \"Any corrupt objects you will have to find in backups or other\n>    archives (i.e., you can just remove them and do an /rsync/ with some\n>    other site in the hopes that somebody else has the object you have\n>    corrupted).\"\n>\n>\n> And that seems sensible to me - the object is corrupt, it is unusable, the\n> object graph is already broken, we already have big problems, removing the\n> corrupt object(s) doesn't create any new problems, and it allows the\n> possibility that the damaged objects can be restored.\n>\n> I ask because I have a corrupt repository, and every time I run fsck, it\n> reports one corrupt object, then stops. I could write a script to repeatedly\n> call fsck and then remove the next corrupt object, but it raises the\n> question for me; could it make sense to extend fsck with the option to do to\n> the removes?\n\nI am positive to this idea. Yesterday a colleague of mine came to me\nwith a repo containing a single corrupt object (in a 1.2GB packfile).\nWe were lucky, since we had a copy of the repo with a good copy of the\nsame object. However, we were lucky in a couple of other respects, as\nwell:\n\nI simply copied the packfile containing the good copy into the\ncorrupted repo, and then ran a \"git gc\", which \"happened\" to use the\ngood copy of the corrupted object and complete successfully (instead\nof barfing on the bad copy). The GC then removed the old\n(now-obsolete) packfiles, and thus the corruption was gone.\n\nHowever, exactly _why_ git happened to prefer the good copy in my\ncopied packfile instead of the bad copy in the existing packfile, I do\nnot know. I suspect some amount of pure luck was involved. Indeed, I\nfeared I would have to explode the corrupt pack, then manually replace\nthe )(now-loose) bad copy with a good copy (from a similarly exploded\npristine pack), and then finally repack everything again. That said,\nI'm not at all sure that Git would be able to successfully explode a\npack containing corrupt objects...\n\nI think a better solution would be to tell fsck to remove the corrupt\nobject(s), as you suggest above, and then copy in the good pack. In\nthat case, there would be no question that the good copy would be used\nin the subsequent GC.\n\n> Or even better, do the removes and then do the necessary\n> [r]sync, assuming the user has another repository that has a good copy of\n> the bad objects, which in this case I do.\n\nHmm. I am not sure we want to automate the syncing step. First, git\ncannot know _which_ remote is likely to have a good copy of the bad\nobject. Second, we do not necessarily know what caused the corruption\nin the first place, and whether syncing with a remote (which will\ncreate certain amount of write activity on a possibly dying disk\ndrive) is a good idea at all. Finally, this syncing step will have to\nbypass Git's usual reachability analysis (which easily skips fetching\na corrupt blob from otherwise-reachable history), is more involved\nthan simply calling out to \"git fetch\"...\n\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"250749","messageId":"vpqmw8w6x9q.fsf@anie.imag.fr","threadId":"37740","inReplyTo":"543F0DAE.2050205@optusnet.com.au","subject":"Re: fsck option to remove corrupt objects - why/why not?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2014-10-16T11:59:13Z","receivedAt":"2014-10-16T11:59:13Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Ben Aveling <bena.001@optusnet.com.au> writes:\n\n> And that seems sensible to me - the object is corrupt, it is unusable,\n> the object graph is already broken, we already have big problems,\n> removing the corrupt object(s) doesn't create any new problems, and it\n> allows the possibility that the damaged objects can be restored.\n\nRemoving completely may remove a chance to restore the corrupt object\n(rather unlikely, but I can imagine fine binary file surgery to un-break\na broken object file).\n\nBut we could move them out of Git's object directory (a bit like\n.git/lost-found, we could have .git/corrupt). For unpacked objects, it's\ntrivial (just mv them in the directory). For packed objects, I don't\nknow what happens in case they are corrupt. That would solve essentially\nany problem that you can solve by removing the file, but it makes the\noperation reversible.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"250750","messageId":"20141016122533.GA8451@peff.net","threadId":"37740","inReplyTo":"CALKQrgda8mVbqP5=Ag8juN9HMQp7iQ9eDJETfRJe1b0taAFGkg@mail.gmail.com","subject":"Re: fsck option to remove corrupt objects - why/why not?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-10-16T12:25:33Z","receivedAt":"2014-10-16T12:25:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 16, 2014 at 11:04:04AM +0200, Johan Herland wrote:\n\n> I simply copied the packfile containing the good copy into the\n> corrupted repo, and then ran a \"git gc\", which \"happened\" to use the\n> good copy of the corrupted object and complete successfully (instead\n> of barfing on the bad copy). The GC then removed the old\n> (now-obsolete) packfiles, and thus the corruption was gone.\n> \n> However, exactly _why_ git happened to prefer the good copy in my\n> copied packfile instead of the bad copy in the existing packfile, I do\n> not know. I suspect some amount of pure luck was involved.\n\nI'm not sure that it is luck, but more like 8eca0b4 (implement some\nresilience against pack corruptions, 2008-06-23) working as intended[1].\nGenerally, git should be able to warn about corrupted objects and look\nin other packs for them (both for regular operations, and for\nrepacking).\n\n-Peff\n\n[1] That's just one of the many commits dealing with this. Try running\n    \"git log --author=Nicolas.Pitre --grep=corrupt\" for more. :)\n"},{"id":"250751","messageId":"CALKQrgdQt0gJ0Dy4M5i-aitiJNeYjHWC8UAY37O7=591MpsR5g@mail.gmail.com","threadId":"37740","inReplyTo":"20141016122533.GA8451@peff.net","subject":"Re: fsck option to remove corrupt objects - why/why not?","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2014-10-16T12:48:15Z","receivedAt":"2014-10-16T12:48:15Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Thu, Oct 16, 2014 at 2:25 PM, Jeff King <peff@peff.net> wrote:\n> On Thu, Oct 16, 2014 at 11:04:04AM +0200, Johan Herland wrote:\n>> I simply copied the packfile containing the good copy into the\n>> corrupted repo, and then ran a \"git gc\", which \"happened\" to use the\n>> good copy of the corrupted object and complete successfully (instead\n>> of barfing on the bad copy). The GC then removed the old\n>> (now-obsolete) packfiles, and thus the corruption was gone.\n>>\n>> However, exactly _why_ git happened to prefer the good copy in my\n>> copied packfile instead of the bad copy in the existing packfile, I do\n>> not know. I suspect some amount of pure luck was involved.\n>\n> I'm not sure that it is luck, but more like 8eca0b4 (implement some\n> resilience against pack corruptions, 2008-06-23) working as intended[1].\n> Generally, git should be able to warn about corrupted objects and look\n> in other packs for them (both for regular operations, and for\n> repacking).\n>\n> -Peff\n>\n> [1] That's just one of the many commits dealing with this. Try running\n>     \"git log --author=Nicolas.Pitre --grep=corrupt\" for more. :)\n\nIndeed, from reading the logs, it seems what I assumed was a lucky\nstrike, was actually carefully designed behavior. With that in mind,\nI'm no longer so sure that fsck actually needs an option to remove\ncorrupt objects. Instead, it's probably better to leave the corrupt\nobject in place until a good copy can be located and copied into the\nrepo, at which point Nicolas' brilliant work will make sure a simple\nrepack takes care of fixing the corruption.\n\nThat said, we should consider documenting this strategy for fixing corruptions:\n - Locate the a good copy of the affected objects in another repo\n - Copy relevant pack file or loose object into this repo\n - Run \"git gc\"\n - Profit!\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"250756","messageId":"xmqq8ukgug31.fsf@gitster.dls.corp.google.com","threadId":"37740","inReplyTo":"CALKQrgda8mVbqP5=Ag8juN9HMQp7iQ9eDJETfRJe1b0taAFGkg@mail.gmail.com","subject":"Re: fsck option to remove corrupt objects - why/why not?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-16T16:36:34Z","receivedAt":"2014-10-16T16:36:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland <johan@herland.net> writes:\n\n> I simply copied the packfile containing the good copy into the\n> corrupted repo, and then ran a \"git gc\", which \"happened\" to use the\n> good copy of the corrupted object and complete successfully (instead\n> of barfing on the bad copy). The GC then removed the old\n> (now-obsolete) packfiles, and thus the corruption was gone.\n>\n> However, exactly _why_ git happened to prefer the good copy in my\n> copied packfile instead of the bad copy in the existing packfile, I do\n> not know.\n\nBy design ;-)\n"}]}