{"thread":{"id":"38145","subject":"Blobs not referenced by file (anymore) are not removed by GC","startedAt":"2014-12-08T16:22:23Z","lastAt":"2014-12-10T23:41:02Z","messageCount":9,"participants":["Martin Scherer","Roberto Tyley","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"253368","messageId":"5485D03F.3060008@fu-berlin.de","threadId":"38145","inReplyTo":null,"subject":"Blobs not referenced by file (anymore) are not removed by GC","fromName":"Martin Scherer","fromEmail":"m.scherer@fu-berlin.de","sentAt":"2014-12-08T16:22:23Z","receivedAt":"2014-12-08T16:22:23Z","isPatch":false,"sender":{"key":"m.scherer@fu-berlin.de","avatar":null},"body":"Hi,\n\nafter using BFG on a repo given certain directory globs, all of those\nfiles(names) are gone from history, but can not be collected by garbage\ncollection anymore. So the blobs of the underlying files are not deleted\nand only the file names are not associated with the blob anymore. I\nwonder, if I discovered a bug (at least in bfg). But I expect git to\ndiscover that this blobs are not used in any way (so they have to\nassociated to something right?)\n\n# invoke bfg --delete-folders something multiple times with different\npattern.\n\n# try to cleanup\n\ngit gc --aggressive --prune=now # big blobs still in history\ngit fsck # no results\ngit fsck --full  --unreachable --dangling # no results\n\nto verify if the blobs are still there, see the output of\n\ngit gc && git verify-pack -v .git/objects/pack/pack-*.idx | egrep \"^\\w+\nblob\\W+[0-9]+ [0-9]+ [0-9]+$\" | sort -k 3 -n -r > bigobjects\n.txt\n\nhead bigobjects.txt # outputs 9451427d7335395779b91864418630d2f0af780a\nblob   7895212 1869047 7657491\n\n\nAlso if bfg is being told to remove the biggest blob (bfg -B 1) with\nno-blob-protection, it does not succeed in removing it.\n\n--- output of bfg -B 1\n\nFound 1 blob ids for large blobs - biggest=7895212 smallest=7895212\n....\n\nBFG aborting: No refs to update - no dirty commits found??\n---\n\nThe repo can be found here.\n\nhttps://github.com/marscher/stallone_stale_objects\n\nI will restart all over to cleanup the history, but I guess this might\nbe interesting for git developers.\n\n\nBest,\nMartin\n"},{"id":"253371","messageId":"CAFY1edarWL1B72ZO0m=RALJ+p3KtnBBX9qQ85iueU6EVD0akaA@mail.gmail.com","threadId":"38145","inReplyTo":"CAFY1edaEq1zYV0vgSfiPAXU6bqVBzaA-apVnSn8DBMbzcAa2tQ@mail.gmail.com","subject":"Re: Blobs not referenced by file (anymore) are not removed by GC","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2014-12-08T16:47:49Z","receivedAt":"2014-12-08T16:47:49Z","isPatch":false,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"Hi Martin, I'm the developer of the BFG - I'd guess that there\nprobably isn't a bug for Git developers here, so you might want to\nopen one or more issues at\nhttps://github.com/rtyley/bfg-repo-cleaner/issues, where I'd be happy\nto take a look.\n\nbest regards,\nRoberto\n\n> On 8 Dec 2014 16:35, \"Martin Scherer\" <m.scherer@fu-berlin.de> wrote:\n>>\n>> Hi,\n>>\n>> after using BFG on a repo given certain directory globs, all of those\n>> files(names) are gone from history, but can not be collected by garbage\n>> collection anymore. So the blobs of the underlying files are not deleted\n>> and only the file names are not associated with the blob anymore. I\n>> wonder, if I discovered a bug (at least in bfg). But I expect git to\n>> discover that this blobs are not used in any way (so they have to\n>> associated to something right?)\n>>\n>> # invoke bfg --delete-folders something multiple times with different\n>> pattern.\n>>\n>> # try to cleanup\n>>\n>> git gc --aggressive --prune=now # big blobs still in history\n>> git fsck # no results\n>> git fsck --full  --unreachable --dangling # no results\n>>\n>> to verify if the blobs are still there, see the output of\n>>\n>> git gc && git verify-pack -v .git/objects/pack/pack-*.idx | egrep \"^\\w+\n>> blob\\W+[0-9]+ [0-9]+ [0-9]+$\" | sort -k 3 -n -r > bigobjects\n>> .txt\n>>\n>> head bigobjects.txt # outputs 9451427d7335395779b91864418630d2f0af780a\n>> blob   7895212 1869047 7657491\n>>\n>>\n>> Also if bfg is being told to remove the biggest blob (bfg -B 1) with\n>> no-blob-protection, it does not succeed in removing it.\n>>\n>> --- output of bfg -B 1\n>>\n>> Found 1 blob ids for large blobs - biggest=7895212 smallest=7895212\n>> ....\n>>\n>> BFG aborting: No refs to update - no dirty commits found??\n>> ---\n>>\n>> The repo can be found here.\n>>\n>> https://github.com/marscher/stallone_stale_objects\n>>\n>> I will restart all over to cleanup the history, but I guess this might\n>> be interesting for git developers.\n>>\n>>\n>> Best,\n>> Martin\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"253416","messageId":"20141209141457.GA18544@peff.net","threadId":"38145","inReplyTo":"5485D03F.3060008@fu-berlin.de","subject":"Re: Blobs not referenced by file (anymore) are not removed by GC","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-12-09T14:14:57Z","receivedAt":"2014-12-09T14:14:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Dec 08, 2014 at 05:22:23PM +0100, Martin Scherer wrote:\n\n> # invoke bfg --delete-folders something multiple times with different\n> pattern.\n> \n> # try to cleanup\n> \n> git gc --aggressive --prune=now # big blobs still in history\n> git fsck # no results\n> git fsck --full  --unreachable --dangling # no results\n\nMight you still have reflogs pointing to the objects? Try:\n\n  git reflog expire --expire-unreachable=now --all\n\nI also don't know if BFG keeps backup refs around (filter-branch, for\nexample, writes a copy of the original refs into refs/original; you\nwould want to delete that if you're trying to slim down the repo).\n\nIn general, you can see the on-disk size of the objects required for a\nparticular ref with something like:\n\n  size() {\n    git rev-list --objects \"$@\" |\n    cut -d' ' -f1 |\n    git cat-file --batch-check='%(objectsize:disk)' |\n    perl -lne '$t += $_; END { print $t }'\n  }\n\n  # size of master branch\n  size master\n\n  # size of each ref on top of what is in the master branch\n  git for-each-ref --format='%(refname)' |\n  while read ref; do\n    echo \"$(size master..$ref) $ref\"\n  done | sort -rn\n\n\nNote that these sizes are somewhat approximate. We may store object X\nneeded by one ref as a delta against Y used by another ref. The\naccounting shows X as tiny compared to Y. And then a repack may find the\ndelta in the opposite direction. But if you're talking about rewriting\nhistory to drop a bunch of gigantic objects, the output of the final\nloop is a good way to see which refs are still referring to the old\nhistory.\n\n-Peff\n"},{"id":"253417","messageId":"CAFY1edaEG040jnfTJA4G9a0bAkFJHc3N5sHjtwOOdXmndsu9YQ@mail.gmail.com","threadId":"38145","inReplyTo":"20141209141457.GA18544@peff.net","subject":"Re: Blobs not referenced by file (anymore) are not removed by GC","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2014-12-09T16:01:50Z","receivedAt":"2014-12-09T16:01:50Z","isPatch":false,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"On 9 December 2014 at 14:14, Jeff King <peff@peff.net> wrote:\n> On Mon, Dec 08, 2014 at 05:22:23PM +0100, Martin Scherer wrote:\n>\n>> # invoke bfg --delete-folders something multiple times with different\n>> pattern.\n>>\n>> # try to cleanup\n>>\n>> git gc --aggressive --prune=now # big blobs still in history\n>> git fsck # no results\n>> git fsck --full  --unreachable --dangling # no results\n>\n> Might you still have reflogs pointing to the objects? Try:\n>\n>   git reflog expire --expire-unreachable=now --all\n\nYeah, we figured that's what it was!\n\nhttps://github.com/rtyley/bfg-repo-cleaner/issues/62#issuecomment-66152559\n\n> I also don't know if BFG keeps backup refs around (filter-branch, for\n> example, writes a copy of the original refs into refs/original; you\n> would want to delete that if you're trying to slim down the repo).\n\nThe BFG reports the ref changes to the command line (and outputs a\nfull list of changed object-ids in\nrepo-name.git.bfg-report/[datetime]/object-id-map.old-new.txt) but\ndoesn't keep refs (like refs/original) around because that would get\nin the way of the BFG's explicit intended use-case of removing\nunwanted data.\n\nThanks for the object-size checking scripts, very useful.\n\nRoberto\n"},{"id":"253418","messageId":"20141209161133.GA17756@peff.net","threadId":"38145","inReplyTo":"CAFY1edaEG040jnfTJA4G9a0bAkFJHc3N5sHjtwOOdXmndsu9YQ@mail.gmail.com","subject":"Re: Blobs not referenced by file (anymore) are not removed by GC","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-12-09T16:11:33Z","receivedAt":"2014-12-09T16:11:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 09, 2014 at 04:01:50PM +0000, Roberto Tyley wrote:\n\n> > I also don't know if BFG keeps backup refs around (filter-branch, for\n> > example, writes a copy of the original refs into refs/original; you\n> > would want to delete that if you're trying to slim down the repo).\n> \n> The BFG reports the ref changes to the command line (and outputs a\n> full list of changed object-ids in\n> repo-name.git.bfg-report/[datetime]/object-id-map.old-new.txt) but\n> doesn't keep refs (like refs/original) around because that would get\n> in the way of the BFG's explicit intended use-case of removing\n> unwanted data.\n\nThanks for explaining; that information may come in handy.\n\nI actually think filter-branch's \"refs/original\" is a bit outdated at\nthis point. The information is there in the reflogs already, and\ndealing with refs/original often causes confusion in my experience. It\ncould probably use a \"git filter-branch --restore\" or something to\nswitch each $ref to $ref@{1} (after making sure that the reflog entry\nwas from filter-branch, of course).\n\nNot that I expect you to want to work on filter-branch. :) But maybe\nfood for thought for a BFG feature.\n\n-Peff\n"},{"id":"253457","messageId":"CAFY1edY=Ren9krK1-yFoxt92AAushUyqnbt=69hdXMeiBFAK=w@mail.gmail.com","threadId":"38145","inReplyTo":"20141209161133.GA17756@peff.net","subject":"Re: Blobs not referenced by file (anymore) are not removed by GC","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2014-12-09T22:15:31Z","receivedAt":"2014-12-09T22:15:31Z","isPatch":false,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"On Tuesday, 9 December 2014, Jeff King <peff@peff.net> wrote:\n> I actually think filter-branch's \"refs/original\" is a bit outdated at\n> this point. The information is there in the reflogs already, and\n> dealing with refs/original often causes confusion in my experience. It\n> could probably use a \"git filter-branch --restore\" or something to\n> switch each $ref to $ref@{1} (after making sure that the reflog entry\n> was from filter-branch, of course).\n\nYeah, I'd agree that refs/original can cause confusion.\n\n\n> Not that I expect you to want to work on filter-branch. :) But maybe\n> food for thought for a BFG feature.\n\nI haven't heard much demand for a recover/restore feature on the BFG\n(I think by the time people get to the BFG, they're pretty sure they\nwant to go ahead with the procedure!) but I'll bear it in mind. Mind\nyou, to make the post-rewrite clean-up easier, I'd be happy to\ncontribute a patch that gives 'gc' a flag to do the equivalent of:\n\ngit reflog expire --expire=now --all && git gc --prune=now --aggressive\n\nMaybe:\n\ngit gc --purge\n\n??\n"},{"id":"253478","messageId":"20141210071119.GA18104@peff.net","threadId":"38145","inReplyTo":"CAFY1edY=Ren9krK1-yFoxt92AAushUyqnbt=69hdXMeiBFAK=w@mail.gmail.com","subject":"Re: Blobs not referenced by file (anymore) are not removed by GC","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-12-10T07:11:19Z","receivedAt":"2014-12-10T07:11:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 09, 2014 at 10:15:31PM +0000, Roberto Tyley wrote:\n\n> > Not that I expect you to want to work on filter-branch. :) But maybe\n> > food for thought for a BFG feature.\n> \n> I haven't heard much demand for a recover/restore feature on the BFG\n> (I think by the time people get to the BFG, they're pretty sure they\n> want to go ahead with the procedure!) but I'll bear it in mind. Mind\n> you, to make the post-rewrite clean-up easier, I'd be happy to\n> contribute a patch that gives 'gc' a flag to do the equivalent of:\n> \n> git reflog expire --expire=now --all && git gc --prune=now --aggressive\n> \n> Maybe:\n> \n> git gc --purge\n\nYeah, that is common enough that it might be worthwhile (you probably\nwant --expire-unreachable in the reflog invocation, though).\n\n-Peff\n"},{"id":"253509","messageId":"xmqqbnnbjxoy.fsf@gitster.dls.corp.google.com","threadId":"38145","inReplyTo":"20141210071119.GA18104@peff.net","subject":"Re: Blobs not referenced by file (anymore) are not removed by GC","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-12-10T16:07:09Z","receivedAt":"2014-12-10T16:07:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>> ... I'd be happy to\n>> contribute a patch that gives 'gc' a flag to do the equivalent of:\n>> \n>> git reflog expire --expire=now --all && git gc --prune=now --aggressive\n>> \n>> Maybe:\n>> \n>> git gc --purge\n>\n> Yeah, that is common enough that it might be worthwhile (you probably\n> want --expire-unreachable in the reflog invocation, though).\n\nAlso you would not want an unconditional --aggressive.\n"},{"id":"253546","messageId":"CAFY1eda6LyZ3z-Uu2AQpqeeCpc9mn_JXwB1+TRLkyGsG+GQzPQ@mail.gmail.com","threadId":"38145","inReplyTo":"xmqqbnnbjxoy.fsf@gitster.dls.corp.google.com","subject":"Re: Blobs not referenced by file (anymore) are not removed by GC","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2014-12-10T23:41:02Z","receivedAt":"2014-12-10T23:41:02Z","isPatch":false,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"On 10 December 2014 at 16:07, Junio C Hamano <gitster@pobox.com> wrote:\n> Jeff King <peff@peff.net> writes:\n>>> git reflog expire --expire=now --all && git gc --prune=now --aggressive\n>>>\n>>> Maybe:\n>>>\n>>> git gc --purge\n>>\n>> Yeah, that is common enough that it might be worthwhile (you probably\n>> want --expire-unreachable in the reflog invocation, though).\n>\n> Also you would not want an unconditional --aggressive.\n\nAfter a big rewrite deleting files the re-optimisation of --aggressive\ncan make a big difference to packsize - for instance 1.2GB to 768MB in\na test I just ran - but of course it is *much* slower, so I suspect\nyou're right about not including it.\n\nI wasn't aware of the '--expire-unreachable=all' switch, though it\nseems like a 'milder' version of the '--expire=now' switch? - in that\nit would keep reflog entries if they haven't been changed, which is\nfair enough and compatible with the 'purge' goal.\n"}]}