{"thread":{"id":"9563","subject":"Rewriting history with git-filter-branch and leaking objects (?)","startedAt":"2007-08-17T17:18:51Z","lastAt":"2007-08-19T19:59:00Z","messageCount":7,"participants":["Mike Hommey","David Kastrup","Junio C Hamano","Sam Vilain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"50942","messageId":"20070817171851.GB13891@glandium.org","threadId":"9563","inReplyTo":null,"subject":"Rewriting history with git-filter-branch and leaking objects (?)","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-08-17T17:18:51Z","receivedAt":"2007-08-17T17:18:51Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nI've been playing with git-filter-branch, and was wondering how objects\nfrom the original branch are supposed to be removed.\n\nIt looks like removing the refs/original/* refs is not enough.\n\nAnd it also looks like when all references seem to be removed, git-prune\ndoesn't fully do its job...\n\nSee the following transcript:\n\n$ git init\nInitialized empty Git repository in .git/\n$ echo a > a ; echo b > b\n$ git add a b\n$ git commit -m \"add a b\"\nCreated initial commit b8875b1: add a b\n 2 files changed, 2 insertions(+), 0 deletions(-)\n create mode 100644 a\n create mode 100644 b\n$ echo a >> a\n$ git commit -a -m \"update a\"\nCreated commit fd97ed9: update a\n 1 files changed, 1 insertions(+), 0 deletions(-)\n$ git-filter-branch --index-filter 'git-update-index --remove b' HEAD\nRewrite fd97ed9a2fef62eca824361fb62269e3c1fc0fb8 (2/2)\nRef 'refs/heads/master' was rewritten\n\nThese refs were rewritten:\nfatal: Not a git repository: '/tmp/test/.git-rewrite/t/../../.git'\n\n(This is a message that happens when GIT_DIR is not set, I guess the\npatches that were sent a few days ago on the list fix this issue)\n\n$ git-cat-file commit b8875b1\ntree 3683f870be446c7cc05ffaef9fa06415276e1828\nauthor Mike Hommey 1187369087 +0200\ncommitter Mike Hommey 1187369087 +0200\n\nadd a b\n\n(not a surprise, since we still have the refs/original/refs/heads/master\nref)\n\n$ git-update-ref -d refs/original/refs/heads/master fd97ed9\n$ git-cat-file commit b8875b1\ntree 3683f870be446c7cc05ffaef9fa06415276e1828\nauthor Mike Hommey 1187369087 +0200\ncommitter Mike Hommey 1187369087 +0200\n\nadd a b\n$ git fsck\n\n(okay, so it is still here, and obviously still referenced ; it appears\nto be referenced in .git/logs/...)\n\n$ rm .git/logs/refs/heads/master \n$ rm .git/logs/HEAD\n$ git fsck\ndangling commit fd97ed9a2fef62eca824361fb62269e3c1fc0fb8\n\n(finally ! So here is a first question: is there a proper way to clean\nthis out ? rm of the logs sounds brutal...)\n\n$ git-prune -n\n3683f870be446c7cc05ffaef9fa06415276e1828 tree\nb8875b1095616c1e7e8f8ffce8ebc172059367ea commit\nfd97ed9a2fef62eca824361fb62269e3c1fc0fb8 commit\n$ git-cat-file commit fd97ed9a2fef62eca824361fb62269e3c1fc0fb8\ntree c1f89248e4b6e47a4529d50d37b0840a14d2efb0\nparent b8875b1095616c1e7e8f8ffce8ebc172059367ea\nauthor Mike Hommey 1187369110 +0200\ncommitter Mike Hommey 1187369110 +0200\n\nupdate a\n\n(Why doesn't prune -n tell me it would remove\nc1f89248e4b6e47a4529d50d37b0840a14d2efb0, which it should, AFAIK ?)\n\nMike\n"},{"id":"50945","messageId":"86k5ruxdxz.fsf@lola.quinscape.zz","threadId":"9563","inReplyTo":"20070817171851.GB13891@glandium.org","subject":"Re: Rewriting history with git-filter-branch and leaking objects (?)","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2007-08-17T17:34:32Z","receivedAt":"2007-08-17T17:34:32Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Mike Hommey writes:\n\n> I've been playing with git-filter-branch, and was wondering how objects\n> from the original branch are supposed to be removed.\n>\n> It looks like removing the refs/original/* refs is not enough.\n>\n> And it also looks like when all references seem to be removed, git-prune\n> doesn't fully do its job...\n\nIt is quite quite hard to get rid of objects. You need to get the\nreflogs for the commits and the files expired.\n\nThe last time I tried this, I ended up unpacking the packed objects,\ncalling git-fsck with appropriate options to tell me about\nunreferenced objects when ignoring reflogs, and removing the files\nmanually with xargs and rm.\n\nProbably I was not able to do something reasonably intelligent, but\nmaking git actually _lose_ data/commits/whatever is really, really\nhard. I have messed up my repo structure considerably several times,\nand everything is still there, with the reflog telling you how to get\nit.\n\nGiven how easy it is to shoot oneself in the foot with git, it is not\nthe worst thing. But you really have to work if you _mean_ it.\n\n-- \nDavid Kastrup\n"},{"id":"50947","messageId":"20070817174621.GA14503@glandium.org","threadId":"9563","inReplyTo":"86k5ruxdxz.fsf@lola.quinscape.zz","subject":"Re: Rewriting history with git-filter-branch and leaking objects (?)","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-08-17T17:46:21Z","receivedAt":"2007-08-17T17:46:21Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, Aug 17, 2007 at 07:34:32PM +0200, David Kastrup wrote:\n> Mike Hommey writes:\n> \n> > I've been playing with git-filter-branch, and was wondering how objects\n> > from the original branch are supposed to be removed.\n> >\n> > It looks like removing the refs/original/* refs is not enough.\n> >\n> > And it also looks like when all references seem to be removed, git-prune\n> > doesn't fully do its job...\n> \n> It is quite quite hard to get rid of objects. You need to get the\n> reflogs for the commits and the files expired.\n> \n> The last time I tried this, I ended up unpacking the packed objects,\n> calling git-fsck with appropriate options to tell me about\n> unreferenced objects when ignoring reflogs, and removing the files\n> manually with xargs and rm.\n> \n> Probably I was not able to do something reasonably intelligent, but\n> making git actually _lose_ data/commits/whatever is really, really\n> hard. I have messed up my repo structure considerably several times,\n> and everything is still there, with the reflog telling you how to get\n> it.\n> \n> Given how easy it is to shoot oneself in the foot with git, it is not\n> the worst thing. But you really have to work if you _mean_ it.\n\nWell, with the introduction of git-filter-branch, once you have\nrewritten your history and validated that everything is okay,\nyou might mean to remove the original branch...\n\nMike\n"},{"id":"50951","messageId":"20070817183115.GA15653@glandium.org","threadId":"9563","inReplyTo":"20070817171851.GB13891@glandium.org","subject":"Re: Rewriting history with git-filter-branch and leaking objects (?)","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-08-17T18:31:15Z","receivedAt":"2007-08-17T18:31:15Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, Aug 17, 2007 at 07:18:51PM +0200, Mike Hommey wrote:\n> $ rm .git/logs/refs/heads/master \n> $ rm .git/logs/HEAD\n\ngit-reflog expire --expire-unreachable=$(date +%s) --all\n\nis cleaner, but git prune -n still misses a tree.\n\nMike\n"},{"id":"50955","messageId":"7vzm0pg92c.fsf@gitster.siamese.dyndns.org","threadId":"9563","inReplyTo":"86k5ruxdxz.fsf@lola.quinscape.zz","subject":"Re: Rewriting history with git-filter-branch and leaking objects (?)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-08-17T21:11:55Z","receivedAt":"2007-08-17T21:11:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Kastrup writes:\n\n> Mike Hommey writes:\n>\n>> I've been playing with git-filter-branch, and was wondering how objects\n>> from the original branch are supposed to be removed.\n>>\n>> It looks like removing the refs/original/* refs is not enough.\n>>\n>> And it also looks like when all references seem to be removed, git-prune\n>> doesn't fully do its job...\n>\n> It is quite quite hard to get rid of objects. You need to get the\n> reflogs for the commits and the files expired.\n\nAn easier way is probably to make a new clone in the\nneighbouring directory locally.\n"},{"id":"50993","messageId":"46C7AC3E.5010806@vilain.net","threadId":"9563","inReplyTo":"20070817174621.GA14503@glandium.org","subject":"Re: Rewriting history with git-filter-branch and leaking objects (?)","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-08-19T02:34:38Z","receivedAt":"2007-08-19T02:34:38Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Mike Hommey wrote:\n> Well, with the introduction of git-filter-branch, once you have\n> rewritten your history and validated that everything is okay,\n> you might mean to remove the original branch...\n> \n\nRight, but it's probably better to leave a trail. I tend to use\nrefs/Attic/* for branches that I've re-written (if I published them). \nThat way, nothing will send or receive them by default. Sometimes I'll\nhave a \"clean\" repository which doesn't have any of them for faster\ninitial cloning.\n\nSam.\n"},{"id":"51015","messageId":"20070819195900.GA1313@glandium.org","threadId":"9563","inReplyTo":"20070817183115.GA15653@glandium.org","subject":"Re: Rewriting history with git-filter-branch and leaking objects (?)","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-08-19T19:59:00Z","receivedAt":"2007-08-19T19:59:00Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, Aug 17, 2007 at 08:31:15PM +0200, Mike Hommey wrote:\n> On Fri, Aug 17, 2007 at 07:18:51PM +0200, Mike Hommey wrote:\n> > $ rm .git/logs/refs/heads/master \n> > $ rm .git/logs/HEAD\n> \n> git-reflog expire --expire-unreachable=$(date +%s) --all\n> \n> is cleaner, but git prune -n still misses a tree.\n\nFWIW, I found out what was happening, and why there was still a tree not\nbeing pruned: it was used by the index.\n\nSo all in all, no spurious object after cleanup.\n\nMike\n"}]}