{"thread":{"id":"28431","subject":"how to remove unreachable objects?","startedAt":"2011-09-19T09:08:31Z","lastAt":"2011-09-24T22:11:06Z","messageCount":9,"participants":["dieter@schoen.or.at","Jeff King","Andreas Schwab","Junio C Hamano","Dieter Schön"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"175780","messageId":"20110919110831.ewq03vnqos4w8cs8@webmail.edis.at","threadId":"28431","inReplyTo":null,"subject":"how to remove unreachable objects?","fromName":"","fromEmail":"dieter@schoen.or.at","sentAt":"2011-09-19T09:08:31Z","receivedAt":"2011-09-19T09:08:31Z","isPatch":false,"sender":{"key":"dieter@schoen.or.at","avatar":null},"body":"hi,\n\ni am relatively new to git, and am currently trying to get used to it.\n\nat the moment i am exploring how to remove unneeded objects, this\nshould be possible with prune, gc and/or fsck.\nmaybe i have not found the right combination or something in my\nunderstand is not correct.\n\nthis is my use case:\ni create a repository and produce several commits on master.\nthen i go back to a certain tag and create a new branch, where i also\ncommit.\nthen i switch back to master and delete (-D) the other branch.\nit should now be unreachable from within git (to prove its existence,\ni remember a commit SHA1 on the dead branch).\nthen i try to get rid of the unreachable objects with a series of\nprune, fsck and gc.\n\n-------------\nschoen.d@ax:~/projects/gitFeatures$ cat mk_dead_end.sh\n#!/bin/sh\n\nDEAD=dead_end\n\nrm -rf $DEAD\nmkdir $DEAD\ncd $DEAD\ngit init\necho \"first commit\" > A\ngit add A\ngit commit -m \"first commit\"\ngit tag first_commit\necho \"second commit\" >> A\ngit add A\ngit commit -m \"second commit\"\ngit checkout first_commit\necho \"commit in dead end\" >> A\ngit add A\ngit commit -m \"changed A in dead end\"\ngit checkout -b $DEAD\ndead_commit=`git log -1 --format=\"%H\"`\ngit checkout master\ngit branch -D $DEAD\ngit show $dead_commit\ngit fsck --unreachable --full --verbose\ngit fsck --unreachable HEAD \\\n                    $(git for-each-ref --format=\"%(objectname)\" refs/heads)\ngit fsck --lost-found\ngit prune -v $dead_commit\ngit prune $(git rev-parse --all)\ngit repack\ngit prune-packed\ngit gc --prune=now\ngit gc --aggressive\ngit show $dead_commit\n\n\n------\nif you look at the output of this script then you see that git knows  \nthat there\nare unreachable/dangling objects, but they remain.\n\nthankful for any pointer,\ndieter\n"},{"id":"175801","messageId":"20110919195335.GA31930@sigill.intra.peff.net","threadId":"28431","inReplyTo":"20110919110831.ewq03vnqos4w8cs8@webmail.edis.at","subject":"Re: how to remove unreachable objects?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-09-19T19:53:36Z","receivedAt":"2011-09-19T19:53:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 19, 2011 at 11:08:31AM +0200, dieter@schoen.or.at wrote:\n\n> this is my use case:\n> i create a repository and produce several commits on master.\n> then i go back to a certain tag and create a new branch, where i also\n> commit.\n> then i switch back to master and delete (-D) the other branch.\n> it should now be unreachable from within git (to prove its existence,\n> i remember a commit SHA1 on the dead branch).\n\nIt will still be referenced by the HEAD reflog, won't it?\n\n> git checkout master\n> git branch -D $DEAD\n> git show $dead_commit\n> git fsck --unreachable --full --verbose\n\nThis shows it reachable, because it is connected from the HEAD reflog.\n\n> git fsck --unreachable HEAD \\\n>                     $(git for-each-ref --format=\"%(objectname)\" refs/heads)\n\nAnd this shows it as unreachable, because you are asking git to only\nlook at the branch tips and HEAD (by default, it looks at all refs and\nreflogs).\n\nI suspect you copied this straight from the git-fsck manpage. That\nadvice is a bit outdated, I think. It blames (in some form) all the way\nback to the original documentation added in c64b9b8 (2005-05-05, only a\nfew weeks after git was born). A few weeks later, fsck learned to\ndefault to looking at all refs (1024932, 2005-05-18). And then other\nsane defaults like reflogs got tacked on later (reflogs came around the\n1.4.x era, in 2006).\n\n> git fsck --lost-found\n> git prune -v $dead_commit\n> git prune $(git rev-parse --all)\n> git repack\n> git prune-packed\n> git gc --prune=now\n> git gc --aggressive\n> git show $dead_commit\n\nIf you really want to make it unreachable, you should expire the\nreflogs, too:\n\n  git reflog expire --expire=now --all\n  # will now report unreachable\n  git fsck --unreachable\n  # will now actually delete objects\n  git prune -v\n  # gives \"bad object ...\"\n  git show $dead_commit\n\ngit-gc will do this for you, but of course the default expiration time\nis much longer (I think something like 90 days).\n\n-Peff\n"},{"id":"175806","messageId":"20110919201804.GB31930@sigill.intra.peff.net","threadId":"28431","inReplyTo":"20110919195335.GA31930@sigill.intra.peff.net","subject":"Re: how to remove unreachable objects?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-09-19T20:18:04Z","receivedAt":"2011-09-19T20:18:04Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 19, 2011 at 03:53:36PM -0400, Jeff King wrote:\n\n> > git fsck --unreachable HEAD \\\n> >                     $(git for-each-ref --format=\"%(objectname)\" refs/heads)\n> \n> And this shows it as unreachable, because you are asking git to only\n> look at the branch tips and HEAD (by default, it looks at all refs and\n> reflogs).\n> \n> I suspect you copied this straight from the git-fsck manpage. That\n> advice is a bit outdated, I think. It blames (in some form) all the way\n> back to the original documentation added in c64b9b8 (2005-05-05, only a\n> few weeks after git was born). A few weeks later, fsck learned to\n> default to looking at all refs (1024932, 2005-05-18). And then other\n> sane defaults like reflogs got tacked on later (reflogs came around the\n> 1.4.x era, in 2006).\n\nSo we should probably do something like this:\n\n-- >8 --\nSubject: [PATCH] docs: brush up obsolete bits of git-fsck manpage\n\nAfter the description and options, the fsck manpage contains\nsome discussion about what it does. Over time, this\ndiscussion has become somewhat obsolete, both in content and\nformatting. In particular:\n\n  1. There are many options now, so starting the discussion\n     with \"It tests...\" makes it unclear whether we are\n     talking about the last option, or about the tool in\n     general. Let's start a new \"discussion\" section and\n     make our antecedent more clear.\n\n  2. It gave an example for --unreachable using for-each-ref\n     to mention all of the heads, saying that it will do \"a\n     _lot_ of verification\". This is hopelessly out-of-date,\n     as giving no arguments will check much more (reflogs,\n     the index, non-head refs).\n\n  3. It goes on to mention tests \"to be added\" (like tree\n     object sorting). We now have these tests.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nI was tempted to just drop this section entirely. It's mostly redundant\nwith the DESCRIPTION section, and any extra details could be folded in\nthere. The most useful bit is the \"what do you do when there is\ncorruption\". But that should perhaps get its own section, if somebody\nfeels like writing something more detailed (I thought we had a guide\nsomewhere, but I couldn't find it).\n\n Documentation/git-fsck.txt |   26 ++++++++------------------\n 1 files changed, 8 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\nindex a2a508d..55b33d7 100644\n--- a/Documentation/git-fsck.txt\n+++ b/Documentation/git-fsck.txt\n@@ -72,30 +72,20 @@ index file, all SHA1 references in .git/refs/*, and all reflogs (unless\n \ta blob, the contents are written into the file, rather than\n \tits object name.\n \n-It tests SHA1 and general object sanity, and it does full tracking of\n-the resulting reachability and everything else. It prints out any\n-corruption it finds (missing or bad objects), and if you use the\n-'--unreachable' flag it will also print out objects that exist but\n-that aren't reachable from any of the specified head nodes.\n-\n-So for example\n-\n-\tgit fsck --unreachable HEAD \\\n-\t\t$(git for-each-ref --format=\"%(objectname)\" refs/heads)\n+DISCUSSION\n+----------\n \n-will do quite a _lot_ of verification on the tree. There are a few\n-extra validity tests to be added (make sure that tree objects are\n-sorted properly etc), but on the whole if 'git fsck' is happy, you\n-do have a valid tree.\n+git-fsck tests SHA1 and general object sanity, and it does full tracking\n+of the resulting reachability and everything else. It prints out any\n+corruption it finds (missing or bad objects), and if you use the\n+'--unreachable' flag it will also print out objects that exist but that\n+aren't reachable from any of the specified head nodes (or the default\n+set, as mentioned above).\n \n Any corrupt objects you will have to find in backups or other archives\n (i.e., you can just remove them and do an 'rsync' with some other site in\n the hopes that somebody else has the object you have corrupted).\n \n-Of course, \"valid tree\" doesn't mean that it wasn't generated by some\n-evil person, and the end result might be crap. git is a revision\n-tracking system, not a quality assurance system ;)\n-\n Extracted Diagnostics\n ---------------------\n \n-- \n1.7.7.rc1.3.gb95be\n"},{"id":"175807","messageId":"m239fsuv4w.fsf@igel.home","threadId":"28431","inReplyTo":"20110919110831.ewq03vnqos4w8cs8@webmail.edis.at","subject":"Re: how to remove unreachable objects?","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2011-09-19T20:36:47Z","receivedAt":"2011-09-19T20:36:47Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"dieter@schoen.or.at writes:\n\n> if you look at the output of this script then you see that git knows  \n> that there\n> are unreachable/dangling objects, but they remain.\n\nYou also need to prune the reflog of HEAD.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"175815","messageId":"7vsjns8b6m.fsf@alter.siamese.dyndns.org","threadId":"28431","inReplyTo":"20110919201804.GB31930@sigill.intra.peff.net","subject":"Re: how to remove unreachable objects?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-09-19T21:38:41Z","receivedAt":"2011-09-19T21:38:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> I was tempted to just drop this section entirely. It's mostly redundant\n> with the DESCRIPTION section, and any extra details could be folded in\n> there. The most useful bit is the \"what do you do when there is\n> corruption\". But that should perhaps get its own section, if somebody\n> feels like writing something more detailed (I thought we had a guide\n> somewhere, but I couldn't find it).\n\nYeah, I've been thinking about making it an error to give refs to fsck, as\nI do not think the use cases for feature justifies the possible confusion\nit may cause.\n\nOne possible use case might be when your repository is corrupt, and does\nnot pass \"git fsck\" (without any argument).  In such a case, if you are\nlucky and your disk corrupted objects only reachable from a recent topic\nbranch, you might find that this command:\n\n\t$ git fsck master next ...list other topics here...\n\nstill succeeds, so that you can figure out which topic makes such a\nlimited fsck fail when it is listed on the command line, judge its\nimportance and resurrect what you can from there, before nuking it to\nbring the repository back in health so that you can recreate the topic.\n"},{"id":"175823","messageId":"20110919225219.GD4056@sigill.intra.peff.net","threadId":"28431","inReplyTo":"7vsjns8b6m.fsf@alter.siamese.dyndns.org","subject":"Re: how to remove unreachable objects?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-09-19T22:52:19Z","receivedAt":"2011-09-19T22:52:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 19, 2011 at 02:38:41PM -0700, Junio C Hamano wrote:\n\n> Yeah, I've been thinking about making it an error to give refs to fsck, as\n> I do not think the use cases for feature justifies the possible confusion\n> it may cause.\n>\n> One possible use case might be when your repository is corrupt, and does\n> not pass \"git fsck\" (without any argument).  In such a case, if you are\n> lucky and your disk corrupted objects only reachable from a recent topic\n> branch, you might find that this command:\n> \n> \t$ git fsck master next ...list other topics here...\n> \n> still succeeds, so that you can figure out which topic makes such a\n> limited fsck fail when it is listed on the command line, judge its\n> importance and resurrect what you can from there, before nuking it to\n> bring the repository back in health so that you can recreate the topic.\n\nDoes that work? I had the impression from the documentation that the\narguments are purely about the reachability analysis, and that the\nactual corruption/correctness checks actually look through the object db\ndirectly, making sure each object is well-formed. Skimming cmd_fsck\nseems to confirm that.\n\n-Peff\n"},{"id":"175828","messageId":"7vfwjs82sc.fsf@alter.siamese.dyndns.org","threadId":"28431","inReplyTo":"20110919225219.GD4056@sigill.intra.peff.net","subject":"Re: how to remove unreachable objects?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-09-20T00:40:03Z","receivedAt":"2011-09-20T00:40:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>> One possible use case might be when your repository is corrupt, and does\n>> not pass \"git fsck\" (without any argument).  In such a case, if you are\n>> lucky and your disk corrupted objects only reachable from a recent topic\n>> branch, you might find that this command:\n>> \n>> \t$ git fsck master next ...list other topics here...\n>> \n>> still succeeds, so that you can figure out which topic makes such a\n>> limited fsck fail when it is listed on the command line, judge its\n>> importance and resurrect what you can from there, before nuking it to\n>> bring the repository back in health so that you can recreate the topic.\n>\n> Does that work? I had the impression from the documentation that the\n> arguments are purely about the reachability analysis, and that the\n> actual corruption/correctness checks actually look through the object db\n> directly, making sure each object is well-formed. Skimming cmd_fsck\n> seems to confirm that.\n\nYou are right that you may see \"corrupt object\" for unreachable from the\ntips in the object store, but I was talking more about verifying\neverything that is needed for reachability analysis from the given tips\ncan be read, iow, \"missing object\" errors, lack of which would mean you\ncan salvage everything reachable from the refs involved in the traversal.\n"},{"id":"175830","messageId":"20110920005159.GA1674@coredump.intra.peff.net","threadId":"28431","inReplyTo":"7vfwjs82sc.fsf@alter.siamese.dyndns.org","subject":"Re: how to remove unreachable objects?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-09-20T00:51:59Z","receivedAt":"2011-09-20T00:51:59Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 19, 2011 at 05:40:03PM -0700, Junio C Hamano wrote:\n\n> > Does that work? I had the impression from the documentation that the\n> > arguments are purely about the reachability analysis, and that the\n> > actual corruption/correctness checks actually look through the object db\n> > directly, making sure each object is well-formed. Skimming cmd_fsck\n> > seems to confirm that.\n> \n> You are right that you may see \"corrupt object\" for unreachable from the\n> tips in the object store, but I was talking more about verifying\n> everything that is needed for reachability analysis from the given tips\n> can be read, iow, \"missing object\" errors, lack of which would mean you\n> can salvage everything reachable from the refs involved in the traversal.\n\nTrue. Though one could also do that with \"git log\", and it would be much\ncheaper (since each trial you run with git-fsck is going to actually\nfsck the object db, which is expensive).\n\nI can't help but think the right solution there is something like:\n\n  1. If the corrupted or missing object is a blob or tree, figure out\n     which commits reference it with something like:\n\n       a. Create a set B of bad objects (blobs or trees).\n\n       b. For each tree in the object db, open and see if it contains\n          any elements of B. If so, add the tree to another set, B'.\n\n       c. If B' is empty, done. Otherwise, add elements from B' to B and\n          goto step (b).\n\n       d. For each commit in the object db, open and check the tree\n          pointer. If it points to an element of B, then the commit is\n          bad.\n\n  2. If the object is a commit, or if you arrived at a set of bad\n     commits through step (1), then use \"branch --contains\" on the\n     bad commits.\n\nwhich is algorithmically efficient (though probably slow if you had to\ncat-file each tree). It might be a handy special command, though (I have\nseen people ask for \"which part of history references this blob\" on\noccasion). I've never bothered writing it because I've never had a\ncorrupt object. :)\n\nAnyway, that is perhaps not relevant to your point. But I do think that\nfsck with arguments is more likely to confuse someone than to actually\nbe part of a productive use-case. I have no problem with deprecating or\nremoving it.\n\n-Peff\n"},{"id":"176145","messageId":"326CBCA0-9960-41B9-B5CD-F231A6880DFE@schoen.or.at","threadId":"28431","inReplyTo":"20110919195335.GA31930@sigill.intra.peff.net","subject":"Re: how to remove unreachable objects?","fromName":"Dieter Schön","fromEmail":"dieter@schoen.or.at","sentAt":"2011-09-24T22:11:06Z","receivedAt":"2011-09-24T22:11:06Z","isPatch":false,"sender":{"key":"dieter@schoen.or.at","avatar":null},"body":"\nAm 19.09.2011 um 21:53 schrieb Jeff King:\n\n> On Mon, Sep 19, 2011 at 11:08:31AM +0200, dieter@schoen.or.at wrote:\n> \n>> this is my use case:\n>> i create a repository and produce several commits on master.\n>> then i go back to a certain tag and create a new branch, where i also\n>> commit.\n>> then i switch back to master and delete (-D) the other branch.\n>> it should now be unreachable from within git (to prove its existence,\n>> i remember a commit SHA1 on the dead branch).\n> \n> It will still be referenced by the HEAD reflog, won't it?\n\nthanks to all that answered!\nit was very helpful and i gained a bit more insight.\n\nkind regards,\ndieter\n"}]}