{"thread":{"id":"38356","subject":"Big repo not shrinking on repack or gc?","startedAt":"2015-01-14T11:51:30Z","lastAt":"2015-01-15T08:56:12Z","messageCount":11,"participants":["Andreas Krey","Jeff King","Junio C Hamano","Bryan Turner"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"254644","messageId":"20150114115130.GA5677@inner.h.apk.li","threadId":"38356","inReplyTo":null,"subject":"Big repo not shrinking on repack or gc?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2015-01-14T11:51:30Z","receivedAt":"2015-01-14T11:51:30Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"Hi everybody,\n\nI have a repo here that is 130G, but when I clone --mirror it, the result\nis only 25G big.  Because of the --mirror I don't think that I missed\nany refs that keep objects only in the source repo.\n\nI already tried 'git repack -fad' and 'git gc' to shrink the original repo,\nbut it only shaved off 3G, and there are a lot of loose objects and old\npack files that I simply don't expect to be there after a repack.\n\nShouldn't 'git gc' (even without --aggressive) or a 'repack -fad' remove\nthose redundant objects and packs?\n\nHow to clean this up? (Additional problem: I don't have enough space\nto run a repack anymore.)\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds <torvalds@*.org>\nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"254650","messageId":"20150114124936.GC30383@peff.net","threadId":"38356","inReplyTo":"20150114115130.GA5677@inner.h.apk.li","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-01-14T12:49:36Z","receivedAt":"2015-01-14T12:49:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 14, 2015 at 12:51:30PM +0100, Andreas Krey wrote:\n\n> I have a repo here that is 130G, but when I clone --mirror it, the result\n> is only 25G big.  Because of the --mirror I don't think that I missed\n> any refs that keep objects only in the source repo.\n\nPerhaps some objects are mentioned by reflogs, but not by the refs? They\nwould not be transferred as part of a clone. Try:\n\n  git rev-list --objects --all | cut -d' ' -f1 | sort >reachable\n  git rev-list --objects --reflog | cut -d' ' -f1 | sort >reflogs\n  comm -13 reachable reflogs |\n  git cat-file --batch-check='%(objectsize:disk)' |\n  perl -lne '$total += $_; END { print $total }'\n\nThat should print the size, in bytes, that reflog-only objects are using\non disk.  You can use \"git reflog expire --expire-unreachable=now --all\"\nto get rid of them (and then repack).\n\n> I already tried 'git repack -fad' and 'git gc' to shrink the original repo,\n\nYou don't need the \"-f\" here. Just \"git repack -ad\" should be enough\n(and the \"-f\" probably makes it _way_ slower).\n\n-Peff\n"},{"id":"254651","messageId":"20150114130731.GD13247@inner.h.apk.li","threadId":"38356","inReplyTo":"20150114124936.GC30383@peff.net","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2015-01-14T13:07:31Z","receivedAt":"2015-01-14T13:07:31Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Wed, 14 Jan 2015 07:49:36 +0000, Jeff King wrote:\n> On Wed, Jan 14, 2015 at 12:51:30PM +0100, Andreas Krey wrote:\n> \n> > I have a repo here that is 130G, but when I clone --mirror it, the result\n> > is only 25G big.  Because of the --mirror I don't think that I missed\n> > any refs that keep objects only in the source repo.\n> \n> Perhaps some objects are mentioned by reflogs, but not by the refs? They\n> would not be transferred as part of a clone. Try:\n> \n>   git rev-list --objects --all | cut -d' ' -f1 | sort >reachable\n>   git rev-list --objects --reflog | cut -d' ' -f1 | sort >reflogs\n\nActually, the output of 'git rev-list --objects --reflog' is empty, and\nthere isn't even a reflog (or similar) directory. (This is a bare repo\ninside atlass. stash.)\n\n...\n> > I already tried 'git repack -fad' and 'git gc' to shrink the original repo,\n> \n> You don't need the \"-f\" here. Just \"git repack -ad\" should be enough\n> (and the \"-f\" probably makes it _way_ slower).\n\nRight, the -f is an old workaround for old jgits in another repo.\n\nApparently, part of the trick is --prune=all or similar on 'git gc',\nto get rid of the loose objects faster. That got a copy of the repo\ndown to around 70G - still way to go.\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds <torvalds@*.org>\nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"254654","messageId":"20150114143946.GE13247@inner.h.apk.li","threadId":"38356","inReplyTo":"20150114124936.GC30383@peff.net","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2015-01-14T14:39:46Z","receivedAt":"2015-01-14T14:39:46Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Wed, 14 Jan 2015 07:49:36 +0000, Jeff King wrote:\n...\n> You don't need the \"-f\" here. Just \"git repack -ad\" should be enough\n> (and the \"-f\" probably makes it _way_ slower).\n\nIndeed, factor four.\n\nHowever, my expectation is that a repack -ad will remove all the\nold pack files, as what is in there is either referenced and put\ninto the new pack, or dropped => there should be a single pack file\nafterwards.\n\nThis is not the case. :-( (Done only with 1.8.2 due to\nlack of compilers for this box.)\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds <torvalds@*.org>\nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"254656","messageId":"20150114160033.GG13247@inner.h.apk.li","threadId":"38356","inReplyTo":"20150114143946.GE13247@inner.h.apk.li","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2015-01-14T16:00:33Z","receivedAt":"2015-01-14T16:00:33Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Wed, 14 Jan 2015 15:39:46 +0000, Andreas Krey wrote:\n...\n> This is not the case. :-( (Done only with 1.8.2 due to\n> lack of compilers for this box.)\n\nNeither for current git (copied repo to other machine)\nThere is one new pack file of a plausible size (25G),\nand 65G worth of old packfiles.\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds <torvalds@*.org>\nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"254659","messageId":"xmqqmw5lb7ho.fsf@gitster.dls.corp.google.com","threadId":"38356","inReplyTo":"20150114143946.GE13247@inner.h.apk.li","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-01-14T17:24:35Z","receivedAt":"2015-01-14T17:24:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andreas Krey <a.krey@gmx.de> writes:\n\n> On Wed, 14 Jan 2015 07:49:36 +0000, Jeff King wrote:\n> ...\n>> You don't need the \"-f\" here. Just \"git repack -ad\" should be enough\n>> (and the \"-f\" probably makes it _way_ slower).\n>\n> Indeed, factor four.\n>\n> However, my expectation is that a repack -ad will remove all the\n> old pack files, as what is in there is either referenced and put\n> into the new pack, or dropped => there should be a single pack file\n> afterwards.\n>\n> This is not the case. :-( (Done only with 1.8.2 due to\n> lack of compilers for this box.)\n\nGuess in the dark: \"ls -l .git/objects/pack\"\nDo you see any .keep files?\n"},{"id":"254718","messageId":"CAGyf7-EL03E8oFcuDzGcmN4bKQhroHw-T4Azm4mb0QX9F40RFw@mail.gmail.com","threadId":"38356","inReplyTo":"xmqqmw5lb7ho.fsf@gitster.dls.corp.google.com","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2015-01-15T01:23:00Z","receivedAt":"2015-01-15T01:23:00Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Thu, Jan 15, 2015 at 4:24 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Andreas Krey <a.krey@gmx.de> writes:\n>\n>> On Wed, 14 Jan 2015 07:49:36 +0000, Jeff King wrote:\n>> ...\n>>> You don't need the \"-f\" here. Just \"git repack -ad\" should be enough\n>>> (and the \"-f\" probably makes it _way_ slower).\n>>\n>> Indeed, factor four.\n>>\n>> However, my expectation is that a repack -ad will remove all the\n>> old pack files, as what is in there is either referenced and put\n>> into the new pack, or dropped => there should be a single pack file\n>> afterwards.\n>>\n>> This is not the case. :-( (Done only with 1.8.2 due to\n>> lack of compilers for this box.)\n>\n> Guess in the dark: \"ls -l .git/objects/pack\"\n> Do you see any .keep files?\n\nI'm one of the Stash developers and just noticed this thread. If the\nrepository in question has been forked via Stash there likely _will_\nbe .keep files. Stash uses alternates for forks, so it's possible, by\ndeleting those kept packs and pruning objects (which you've already\ndone I see) that you will corrupt, or have already corrupted, some\nnumber of the forks. (At the moment Stash packs \"garbage\" into a \"dead\npack\" which it flags with a .keep, to ensure forks don't lose access\nto objects that once existed upstream that they still reference.)\n\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"254724","messageId":"20150115063806.GH13247@inner.h.apk.li","threadId":"38356","inReplyTo":"CAGyf7-EL03E8oFcuDzGcmN4bKQhroHw-T4Azm4mb0QX9F40RFw@mail.gmail.com","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2015-01-15T06:38:06Z","receivedAt":"2015-01-15T06:38:06Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Thu, 15 Jan 2015 12:23:00 +0000, Bryan Turner wrote:\n...\n> > Guess in the dark: \"ls -l .git/objects/pack\"\n> > Do you see any .keep files?\n\nLots of. :-(\n\n> I'm one of the Stash developers and just noticed this thread. If the\n> repository in question has been forked via Stash there likely _will_\n> be .keep files. Stash uses alternates for forks, so it's possible, by\n> deleting those kept packs and pruning objects (which you've already\n> done I see) that you will corrupt, or have already corrupted, some\n> number of the forks.\n\nThere are a few forks in this stash instance, but the repository in\nquestion is neither the source nor the destination of any.\n\nSo, git seems to be mostly out of the equation now (gc and repack\napparently doing what they are supposed to do), and the question\nmoves to 'how can stash let such a repo grow to that size'.\n\n\n> (At the moment Stash packs \"garbage\" into a \"dead\n> pack\" which it flags with a .keep, to ensure forks don't lose access\n> to objects that once existed upstream that they still reference.)\n\nDoes it do so in any case even if there is no actual fork? That would\nexplain a lot - we are daily (force-)pushing new commit in there (and\npotentially big ones) that become garbage the next day, and should\nbe cleaned up rather fast.\n\n(We're pulling them into another non-stash repo for longer-term keeping -\nthese are backups of dev repos in the form of git stash commits including\nuntracked files.)\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds <torvalds@*.org>\nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"254729","messageId":"CAGyf7-HMsBnim5yoOgyNJ2=HNpnW8ftvitarMc6UO-zTLfQkBA@mail.gmail.com","threadId":"38356","inReplyTo":"20150115063806.GH13247@inner.h.apk.li","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2015-01-15T07:05:46Z","receivedAt":"2015-01-15T07:05:46Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Thu, Jan 15, 2015 at 5:38 PM, Andreas Krey <a.krey@gmx.de> wrote:\n> On Thu, 15 Jan 2015 12:23:00 +0000, Bryan Turner wrote:\n> ...\n>> > Guess in the dark: \"ls -l .git/objects/pack\"\n>> > Do you see any .keep files?\n>\n> Lots of. :-(\n>\n>> I'm one of the Stash developers and just noticed this thread. If the\n>> repository in question has been forked via Stash there likely _will_\n>> be .keep files. Stash uses alternates for forks, so it's possible, by\n>> deleting those kept packs and pruning objects (which you've already\n>> done I see) that you will corrupt, or have already corrupted, some\n>> number of the forks.\n>\n> There are a few forks in this stash instance, but the repository in\n> question is neither the source nor the destination of any.\n>\n> So, git seems to be mostly out of the equation now (gc and repack\n> apparently doing what they are supposed to do), and the question\n> moves to 'how can stash let such a repo grow to that size'.\n>\n>\n>> (At the moment Stash packs \"garbage\" into a \"dead\n>> pack\" which it flags with a .keep, to ensure forks don't lose access\n>> to objects that once existed upstream that they still reference.)\n>\n> Does it do so in any case even if there is no actual fork? That would\n> explain a lot - we are daily (force-)pushing new commit in there (and\n> potentially big ones) that become garbage the next day, and should\n> be cleaned up rather fast.\n\nNo, Stash will only do that in a repository which has been forked. In\nany non-forked repository, Stash does not interact with garbage\ncollection in any way. Auto GC is left enabled, and all pruning\nsettings are left at their defaults. The default pruning interval is\ntwo weeks, so if your development approach is rebase-heavy you may\nneed to adjust them.\n\nWhat are the contents of some of those .keep files? If they're written\nby Stash they contain a message saying so. (\"GENERATED BY ATLASSIAN\nSTASH - DO NOT REMOVE\")\n\n>\n> (We're pulling them into another non-stash repo for longer-term keeping -\n> these are backups of dev repos in the form of git stash commits including\n> untracked files.)\n>\n> Andreas\n>\n> --\n> \"Totally trivial. Famous last words.\"\n> From: Linus Torvalds <torvalds@*.org>\n> Date: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"254732","messageId":"20150115074334.GJ13247@inner.h.apk.li","threadId":"38356","inReplyTo":"CAGyf7-HMsBnim5yoOgyNJ2=HNpnW8ftvitarMc6UO-zTLfQkBA@mail.gmail.com","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2015-01-15T07:43:34Z","receivedAt":"2015-01-15T07:43:34Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Thu, 15 Jan 2015 18:05:46 +0000, Bryan Turner wrote:\n...\n> No, Stash will only do that in a repository which has been forked. In\n> any non-forked repository, Stash does not interact with garbage\n> collection in any way. Auto GC is left enabled, and all pruning\n> settings are left at their defaults. The default pruning interval is\n> two weeks, so if your development approach is rebase-heavy you may\n> need to adjust them.\n> \n> What are the contents of some of those .keep files? If they're written\n> by Stash they contain a message saying so. (\"GENERATED BY ATLASSIAN\n> STASH - DO NOT REMOVE\")\n\nThey do. So it seems it was forked once upon a time, but...\n\n/opt/apps/atlassian/stash-data/shared/data/repositories $ grep '' */objects/info/alternates\n158/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/20/objects\n45/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/33/objects\n93/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/91/objects\n\n...there is no trace of a fork still existing (the repo in question is 143).\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds <torvalds@*.org>\nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"254734","messageId":"CAGyf7-Hi2YZQZ610uH6iAsQYRNdLndPCy+ArzZpi4Up7NhkpqQ@mail.gmail.com","threadId":"38356","inReplyTo":"20150115074334.GJ13247@inner.h.apk.li","subject":"Re: Big repo not shrinking on repack or gc?","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2015-01-15T08:56:12Z","receivedAt":"2015-01-15T08:56:12Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Thu, Jan 15, 2015 at 6:43 PM, Andreas Krey <a.krey@gmx.de> wrote:\n> On Thu, 15 Jan 2015 18:05:46 +0000, Bryan Turner wrote:\n> ...\n>\n> They do. So it seems it was forked once upon a time, but...\n>\n> /opt/apps/atlassian/stash-data/shared/data/repositories $ grep '' */objects/info/alternates\n> 158/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/20/objects\n> 45/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/33/objects\n> 93/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/91/objects\n>\n> ...there is no trace of a fork still existing (the repo in question is 143).\n\nYes, the system doesn't currently detect when a repository becomes\nun-forked because it's not a common use case.\n\nAt this point I think we should probably take this off-list. You can\neither e-mail me directly (bturner at atlassian dot com), or, better\nstill, raise a ticket on support.atlassian.com. Either way I'll work\nwith you directly to un-fork the repository on disk and allow it to\nclean itself up.\n\n>\n> Andreas\n>\n> --\n> \"Totally trivial. Famous last words.\"\n> From: Linus Torvalds <torvalds@*.org>\n> Date: Fri, 22 Jan 2010 07:29:21 -0800\n"}]}