Big repo not shrinking on repack or gc?

Hi everybody,

I have a repo here that is 130G, but when I clone --mirror it, the result is only 25G big. Because of the --mirror I don't think that I missed any refs that keep objects only in the source repo.

I already tried 'git repack -fad' and 'git gc' to shrink the original repo, but it only shaved off 3G, and there are a lot of loose objects and old pack files that I simply don't expect to be there after a repack.

Shouldn't 'git gc' (even without --aggressive) or a 'repack -fad' remove those redundant objects and packs?

How to clean this up? (Additional problem: I don't have enough space to run a repack anymore.)

Andreas

On Wed, Jan 14, 2015 at 12:51:30PM +0100, Andreas Krey wrote:

Perhaps some objects are mentioned by reflogs, but not by the refs? They would not be transferred as part of a clone. Try:

  git rev-list --objects --all | cut -d' ' -f1 | sort >reachable
  git rev-list --objects --reflog | cut -d' ' -f1 | sort >reflogs
  comm -13 reachable reflogs |
  git cat-file --batch-check='%(objectsize:disk)' |
  perl -lne '$total += $_; END { print $total }'

That should print the size, in bytes, that reflog-only objects are using on disk. You can use "git reflog expire --expire-unreachable=now --all" to get rid of them (and then repack).

You don't need the "-f" here. Just "git repack -ad" should be enough (and the "-f" probably makes it _way_ slower).

-Peff

On Wed, 14 Jan 2015 07:49:36 +0000, Jeff King wrote:

Actually, the output of 'git rev-list --objects --reflog' is empty, and there isn't even a reflog (or similar) directory. (This is a bare repo inside atlass. stash.)

...

Right, the -f is an old workaround for old jgits in another repo.

Apparently, part of the trick is --prune=all or similar on 'git gc', to get rid of the loose objects faster. That got a copy of the repo down to around 70G - still way to go.

Andreas

On Wed, 14 Jan 2015 07:49:36 +0000, Jeff King wrote: ...

Indeed, factor four.

However, my expectation is that a repack -ad will remove all the old pack files, as what is in there is either referenced and put into the new pack, or dropped => there should be a single pack file afterwards.

This is not the case. :-( (Done only with 1.8.2 due to lack of compilers for this box.)

Andreas

On Wed, 14 Jan 2015 15:39:46 +0000, Andreas Krey wrote: ...

Neither for current git (copied repo to other machine) There is one new pack file of a plausible size (25G), and 65G worth of old packfiles.

Andreas

Andreas Krey <a.krey@gmx.de> writes:

Guess in the dark: "ls -l .git/objects/pack" Do you see any .keep files?

On Thu, Jan 15, 2015 at 4:24 AM, Junio C Hamano <gitster@pobox.com> wrote:

I'm one of the Stash developers and just noticed this thread. If the repository in question has been forked via Stash there likely _will_ be .keep files. Stash uses alternates for forks, so it's possible, by deleting those kept packs and pruning objects (which you've already done I see) that you will corrupt, or have already corrupted, some number of the forks. (At the moment Stash packs "garbage" into a "dead pack" which it flags with a .keep, to ensure forks don't lose access to objects that once existed upstream that they still reference.)

On Thu, 15 Jan 2015 12:23:00 +0000, Bryan Turner wrote: ...

Lots of. :-(

There are a few forks in this stash instance, but the repository in question is neither the source nor the destination of any.

So, git seems to be mostly out of the equation now (gc and repack apparently doing what they are supposed to do), and the question moves to 'how can stash let such a repo grow to that size'.

Does it do so in any case even if there is no actual fork? That would explain a lot - we are daily (force-)pushing new commit in there (and potentially big ones) that become garbage the next day, and should be cleaned up rather fast.

(We're pulling them into another non-stash repo for longer-term keeping - these are backups of dev repos in the form of git stash commits including untracked files.)

Andreas

On Thu, Jan 15, 2015 at 5:38 PM, Andreas Krey <a.krey@gmx.de> wrote:

No, Stash will only do that in a repository which has been forked. In any non-forked repository, Stash does not interact with garbage collection in any way. Auto GC is left enabled, and all pruning settings are left at their defaults. The default pruning interval is two weeks, so if your development approach is rebase-heavy you may need to adjust them.

What are the contents of some of those .keep files? If they're written by Stash they contain a message saying so. ("GENERATED BY ATLASSIAN STASH - DO NOT REMOVE")

On Thu, 15 Jan 2015 18:05:46 +0000, Bryan Turner wrote: ...

They do. So it seems it was forked once upon a time, but...

/opt/apps/atlassian/stash-data/shared/data/repositories $ grep '' */objects/info/alternates 158/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/20/objects 45/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/33/objects 93/objects/info/alternates:/data/opt_apps/atlassian/stash-data/shared/data/repositories/91/objects

...there is no trace of a fork still existing (the repo in question is 143).

Andreas

On Thu, Jan 15, 2015 at 6:43 PM, Andreas Krey <a.krey@gmx.de> wrote:

Yes, the system doesn't currently detect when a repository becomes un-forked because it's not a common use case.

At this point I think we should probably take this off-list. You can either e-mail me directly (bturner at atlassian dot com), or, better still, raise a ticket on support.atlassian.com. Either way I'll work with you directly to un-fork the repository on disk and allow it to clean itself up.