threads / discuss / 22534

git gc / git repack not removing unused objects?

Subject: git gc / git repack not removing unused objects?

## tl;dr

10 messages between Feb 5, 2010 and Feb 8, 2010.

replies: 9people: 2as markdown or json

Jon Nelson· Feb 5, 2010, 19:45 UTC · lore
[Using git 1.6.4.2]

In one repo I have (136G objects directory, fully packed) I'm having some trouble. I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen times and each time I do so it gets bigger, not smaller. Setting that aside for the moment, however, I've run into a stranger problem.

So I use "git verify-pack -v > gvp.out" and "sort -k3nr < gvp.out | head -n 20" to find the top 20 largest blobs. So I have a blob, b32c3d8e8e24d8d3035cf52f606c2873315fe2b8, and now I want to know what tree (or trees) it is in, so I try this:

for i in $( git branch -a | sed -e 's/\*//g' | grep -v branch ); do if git ls-tree -l -r -t $i | grep b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 > /dev/null; then echo $i; fi; done

The results: no branch or tree appears to contain that blob.
So I tried a different approach:

for i in $( grep tree gvp.out | awk '{ print $1 }' ); do if git ls-tree $i | grep b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 > /dev/null; then echo $i; fi ; done

This time, I find (at least) one tree (d813af1537358496ca34958bbff08b87590607bf) with the blob. But which branches might that tree appear in? None.

For each branch, I ran "git ls-tree -l -r -t" and saved the output in a file (one per branch). Then I grepped each file for the tree ( (d813af1537358496ca34958bbff08b87590607bf) - no luck. I grepped each file for the blob (b32...) - no luck.

The results seem to suggest that I have packed trees which reference blobs, but that the trees themselves are not referenced in any branch and therefore I would expect that they would be pruned.

-- 
Jon
Nicolas Pitre· Feb 5, 2010, 20:51 UTC · re: Jon Nelson · lore

Re: git gc / git repack not removing unused objects?

On Fri, 5 Feb 2010, Jon Nelson wrote:
Show 6 quoted lines
> [Using git 1.6.4.2]
> 
> In one repo I have (136G objects directory, fully packed) I'm having
> some trouble.
> I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
> times and each time I do so it gets bigger, not smaller.
Please tell us more.
Show 14 quoted lines
> Setting that aside for the moment, however, I've run into a stranger problem.
> 
> So I use "git verify-pack -v > gvp.out" and "sort -k3nr < gvp.out |
> head -n 20" to find the top 20 largest blobs.
> So I have a blob, b32c3d8e8e24d8d3035cf52f606c2873315fe2b8, and now I
> want to know what tree (or trees) it is in, so I try this:
> 
> 
> for i in $( git branch -a | sed -e 's/\*//g' | grep -v branch ); do if
> git ls-tree -l -r -t $i | grep
> b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 > /dev/null; then echo $i;
> fi; done
> 
> The results: no branch or tree appears to contain that blob.

What you did above is simply to list trees that are reachable from the _heads_ of your branches. If the blob belongs to a commit which isn't the latest revision of any of your branches then you won't see it like that.

Show 19 quoted lines
> So I tried a different approach:
> 
> for i in $( grep tree gvp.out  | awk '{ print $1 }' ); do if git
> ls-tree $i | grep b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 >
> /dev/null; then echo $i; fi ; done
> 
> This time, I find (at least) one tree
> (d813af1537358496ca34958bbff08b87590607bf) with the blob.
> But which branches might that tree appear in? None.
> 
> For each branch, I ran "git ls-tree -l -r -t" and saved the output in
> a file (one per branch).
> Then I grepped each file for the tree (
> (d813af1537358496ca34958bbff08b87590607bf) - no luck.
> I grepped each file for the blob (b32...) - no luck.
> 
> The results seem to suggest that I have packed trees which reference
> blobs, but that the trees themselves are not referenced in any branch
> and therefore I would expect that they would be pruned.

NO. If those trees and blobs are stil there then they do get referenced. But not from the latest commit on any of your branches. You need to dig further down in history to find a commit that actually references that blob/tree. One easy method is to do:

	git log --raw --all
and within the pager ('less' by default) simply search for "b32c3d8".
Nicolas
Jon Nelson· Feb 5, 2010, 21:04 UTC · re: Nicolas Pitre · lore

Re: git gc / git repack not removing unused objects?

On Fri, Feb 5, 2010 at 2:51 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
Show 10 quoted lines
> On Fri, 5 Feb 2010, Jon Nelson wrote:
>
>> [Using git 1.6.4.2]
>>
>> In one repo I have (136G objects directory, fully packed) I'm having
>> some trouble.
>> I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
>> times and each time I do so it gets bigger, not smaller.
>
> Please tell us more.
I'll tell you whatever I can -- as soon as I know what it is you want.
Show 48 quoted lines
>> Setting that aside for the moment, however, I've run into a stranger problem.
>>
>> So I use "git verify-pack -v > gvp.out" and "sort -k3nr < gvp.out |
>> head -n 20" to find the top 20 largest blobs.
>> So I have a blob, b32c3d8e8e24d8d3035cf52f606c2873315fe2b8, and now I
>> want to know what tree (or trees) it is in, so I try this:
>>
>>
>> for i in $( git branch -a | sed -e 's/\*//g' | grep -v branch ); do if
>> git ls-tree -l -r -t $i | grep
>> b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 > /dev/null; then echo $i;
>> fi; done
>>
>> The results: no branch or tree appears to contain that blob.
>
> What you did above is simply to list trees that are reachable from the
> _heads_ of your branches.  If the blob belongs to a commit which isn't
> the latest revision of any of your branches then you won't see it like
> that.
>
>> So I tried a different approach:
>>
>> for i in $( grep tree gvp.out  | awk '{ print $1 }' ); do if git
>> ls-tree $i | grep b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 >
>> /dev/null; then echo $i; fi ; done
>>
>> This time, I find (at least) one tree
>> (d813af1537358496ca34958bbff08b87590607bf) with the blob.
>> But which branches might that tree appear in? None.
>>
>> For each branch, I ran "git ls-tree -l -r -t" and saved the output in
>> a file (one per branch).
>> Then I grepped each file for the tree (
>> (d813af1537358496ca34958bbff08b87590607bf) - no luck.
>> I grepped each file for the blob (b32...) - no luck.
>>
>> The results seem to suggest that I have packed trees which reference
>> blobs, but that the trees themselves are not referenced in any branch
>> and therefore I would expect that they would be pruned.
>
> NO.  If those trees and blobs are stil there then they do get
> referenced.  But not from the latest commit on any of your branches.
> You need to dig further down in history to find a commit that actually
> references that blob/tree.  One easy method is to do:
>
>        git log --raw --all
>
> and within the pager ('less' by default) simply search for "b32c3d8".

OK. I'm piping "git log --raw --all" to a file this very moment. It'll take a while. However, one thing I did not mention is that there *should* be a 1:1 correlation between branches and commits. As in, every time I did a commit, the commit was on a new branch. I'll look into this, as I've fiddled with the repo a bunch of different ways lately. I suspect the answer will be found in the logs.

Thanks for the response!
-- 
Jon
Nicolas Pitre· Feb 5, 2010, 21:45 UTC · re: Jon Nelson · lore

Re: git gc / git repack not removing unused objects?

On Fri, 5 Feb 2010, Jon Nelson wrote:
Show 13 quoted lines
> On Fri, Feb 5, 2010 at 2:51 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Fri, 5 Feb 2010, Jon Nelson wrote:
> >
> >> [Using git 1.6.4.2]
> >>
> >> In one repo I have (136G objects directory, fully packed) I'm having
> >> some trouble.
> >> I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
> >> times and each time I do so it gets bigger, not smaller.
> >
> > Please tell us more.
> 
> I'll tell you whatever I can -- as soon as I know what it is you want.

I want to know what you did and what results you obtained to lead you to believe that 'git-gc --prune=now' makes your repository bigger.

Show 11 quoted lines
> > You need to dig further down in history to find a commit that actually
> > references that blob/tree.  One easy method is to do:
> >
> >        git log --raw --all
> >
> > and within the pager ('less' by default) simply search for "b32c3d8".
> 
> OK. I'm piping "git log --raw --all" to a file this very moment. It'll
> take a while. However, one thing I did not mention is that there
> *should* be a 1:1 correlation between branches and commits. As in,
> every time I did a commit, the commit was on a new branch. 
Ouch.  Why?  I'm curious.
> I'll look into this, as I've fiddled with the repo a bunch of 
> different ways lately. I suspect the answer will be found in the logs.

If the blob you're looking for is not in that log, then it must be referenced by one of the reflog entries. The reflog is a log of all the commits your branches, and the HEAD pointer, ever pointed at. Those entries are kept for a certain period (3 months by default). The HEAD reflog will keep a reference of all branches you might have visited, even deleted ones. So another log:

	git log --raw -g HEAD
should ultimately give you the blob reference.
Nicolas
Jon Nelson· Feb 6, 2010, 13:53 UTC · re: Nicolas Pitre · lore

Re: git gc / git repack not removing unused objects?

On Fri, Feb 5, 2010 at 3:45 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
Show 18 quoted lines
> On Fri, 5 Feb 2010, Jon Nelson wrote:
>
>> On Fri, Feb 5, 2010 at 2:51 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> > On Fri, 5 Feb 2010, Jon Nelson wrote:
>> >
>> >> [Using git 1.6.4.2]
>> >>
>> >> In one repo I have (136G objects directory, fully packed) I'm having
>> >> some trouble.
>> >> I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
>> >> times and each time I do so it gets bigger, not smaller.
>> >
>> > Please tell us more.
>>
>> I'll tell you whatever I can -- as soon as I know what it is you want.
>
> I want to know what you did and what results you obtained to lead you to
> believe that 'git-gc --prune=now' makes your repository bigger.

Last night, the repo size was 153G after removing some commits and objects by way of git filter-branch. I'm using "du -sh" in the .git directory to determine the disk usage.

Before: 136G
git repack -dAl
After: 153G
Then, just to make sure of some things, I changed nothing and simply
re-ran "git repack -dAl".
After: 167G
Config:
[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        logallrefupdates = true
        compression = 0
        loosecompression = 0
        packedGitLimit = 1500M
        #packedGitWindowSize = 32m
[pack]
        packsizelimit = 256m
        windowMemory = 128m
        deltaCacheSize = 1m
[gc]
    threads = 1
    packrefs = true
[repack]
        usedeltabaseoffset = true
Incorporating the system and global config (git config -l):

pack.packsizelimit=2M pack.threads=1 gc.auto=0 gc.autopacklimit=0 core.repositoryformatversion=0 core.filemode=true core.bare=true core.logallrefupdates=true core.compression=0 core.loosecompression=0 core.packedgitlimit=1500M pack.packsizelimit=256m pack.windowmemory=128m pack.deltacachesize=1m gc.threads=1 gc.packrefs=true repack.usedeltabaseoffset=true

-- 
Jon
Nicolas Pitre· Feb 7, 2010, 01:16 UTC · re: Jon Nelson · lore

Re: git gc / git repack not removing unused objects?

On Sat, 6 Feb 2010, Jon Nelson wrote:
Show 7 quoted lines
> Last night, the repo size was 153G after removing some commits and
> objects by way of git filter-branch.
> I'm using "du -sh" in the .git directory to determine the disk usage.
> 
> Before: 136G
> git repack -dAl
> After: 153G
Why are you using -A instead of -a ?
> Then, just to make sure of some things, I changed nothing and simply
> re-ran "git repack -dAl".
> After: 167G

Could you run 'git count-objects -v' before and after a repack in such cases as well?

> [pack]
>         packsizelimit = 256m
Why are you using this?
> pack.packsizelimit=2M
This is even worse.  150G / 2M per pack = approx 75000 packs.

What you should be aiming for is a _single_ pack for best performances. A couple packs is common and doesn't make much of a difference. A hundred packs is getting a bit large. But 75000 packs is totally insane.

Nicolas
Jon Nelson· Feb 7, 2010, 17:48 UTC · re: Nicolas Pitre · lore

Re: git gc / git repack not removing unused objects?

On Sat, Feb 6, 2010 at 7:16 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
Show 11 quoted lines
> On Sat, 6 Feb 2010, Jon Nelson wrote:
>
>> Last night, the repo size was 153G after removing some commits and
>> objects by way of git filter-branch.
>> I'm using "du -sh" in the .git directory to determine the disk usage.
>>
>> Before: 136G
>> git repack -dAl
>> After: 153G
>
> Why are you using -A instead of -a ?

As it turns out, I've been using both -a and -A. I suspect -A is a typo on my part.

Show 6 quoted lines
>> Then, just to make sure of some things, I changed nothing and simply
>> re-ran "git repack -dAl".
>> After: 167G
>
> Could you run 'git count-objects -v' before and after a repack in such
> cases as well?
Yes.
>> [pack]
>>         packsizelimit = 256m
>
> Why are you using this?
I didn't want my pack files to be too huge. I've bumped that up to 2G.
>> pack.packsizelimit=2M

My ~/.gitconfig normally uses 2M for easy rsyncing. In this repo I thought the value was overridden by the project's config (which was specifying 256m and now specifies 2048m).

Suboptimal or not, it still doesn't explain why the repo grows with each repack.
Now running:
 git count-objects -v ; git repack -ad ; git count-objects -v
-- 
Jon
Jon Nelson· Feb 7, 2010, 23:40 UTC · re: Jon Nelson · lore

Re: git gc / git repack not removing unused objects?

On Sun, Feb 7, 2010 at 11:48 AM, Jon Nelson <jnelson@jamponi.net> wrote:
Show 43 quoted lines
> On Sat, Feb 6, 2010 at 7:16 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> On Sat, 6 Feb 2010, Jon Nelson wrote:
>>
>>> Last night, the repo size was 153G after removing some commits and
>>> objects by way of git filter-branch.
>>> I'm using "du -sh" in the .git directory to determine the disk usage.
>>>
>>> Before: 136G
>>> git repack -dAl
>>> After: 153G
>>
>> Why are you using -A instead of -a ?
>
> As it turns out, I've been using both -a and -A. I suspect -A is a
> typo on my part.
>
>>> Then, just to make sure of some things, I changed nothing and simply
>>> re-ran "git repack -dAl".
>>> After: 167G
>>
>> Could you run 'git count-objects -v' before and after a repack in such
>> cases as well?
>
> Yes.
>
>>> [pack]
>>>         packsizelimit = 256m
>>
>> Why are you using this?
>
> I didn't want my pack files to be too huge. I've bumped that up to 2G.
>
>>> pack.packsizelimit=2M
>
> My ~/.gitconfig normally uses 2M for easy rsyncing. In this repo I
> thought the value was overridden by the project's config (which was
> specifying 256m and now specifies 2048m).
>
> Suboptimal or not, it still doesn't explain why the repo grows with each repack.
>
> Now running:
>
>  git count-objects -v ; git repack -ad ; git count-objects -v
The results (two lines of whitespace added for clarity):

turnip:/mnt/backups.git # git count-objects -v ; git repack -ad ; git count-objects -v count: 0 size: 0 in-pack: 2408195 packs: 675 size-pack: 174924053 prune-packable: 0 garbage: 0

Counting objects: 266309, done. Compressing objects: 100% (207820/207820), done. Writing objects: 100% (266309/266309), done. Total 266309 (delta 47751), reused 227730 (delta 45117)

count: 0 size: 0 in-pack: 2674504 packs: 686 size-pack: 196517094 prune-packable: 0 garbage: 0 turnip:/mnt/backups.git # du -sh . 188G . turnip:/mnt/backups.git # find objects -name '*.pack' | wc -l 686 turnip:/mnt/backups.git #

So we went from 136G to 153G to 167G and now 188G.

The oldest .pack file is Jan 29 at 7.0M The pack files vary considerably in size from 220B to 2.0G There are 433 packs that are between 200M and 300M in size. There are 78 packs that are between 500M and 600M in size. There are 211 packs that are less than 256M in size.

I ran "git prune -v --prune=now" and it claims to have removed nothing and the exit status was 0.

I hope this helps!
-- 
Jon
Nicolas Pitre· Feb 8, 2010, 02:11 UTC · re: Jon Nelson · lore

Re: git gc / git repack not removing unused objects?

On Sun, 7 Feb 2010, Jon Nelson wrote:
Show 13 quoted lines
> On Sat, Feb 6, 2010 at 7:16 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> >On Sat, 6 Feb 2010, Jon Nelson wrote:
> >>         packsizelimit = 256m
> >
> > Why are you using this?
> 
> I didn't want my pack files to be too huge. I've bumped that up to 2G.
> 
> >> pack.packsizelimit=2M
> 
> My ~/.gitconfig normally uses 2M for easy rsyncing. In this repo I
> thought the value was overridden by the project's config (which was
> specifying 256m and now specifies 2048m).

Why do you synchronize with rsync? Why not a simple 'git fetch' which is likely to be much faster in any case?

> Suboptimal or not, it still doesn't explain why the repo grows with each repack.
That's strange indeed.  Except that ...
Show 22 quoted lines
> turnip:/mnt/backups.git # git count-objects -v ; git repack -ad ; git
> count-objects -v
> count: 0
> size: 0
> in-pack: 2408195
> packs: 675
> size-pack: 174924053
> prune-packable: 0
> garbage: 0
> 
> Counting objects: 266309, done.
> Compressing objects: 100% (207820/207820), done.
> Writing objects: 100% (266309/266309), done.
> Total 266309 (delta 47751), reused 227730 (delta 45117)
> 
> count: 0
> size: 0
> in-pack: 2674504
> packs: 686
> size-pack: 196517094
> prune-packable: 0
> garbage: 0
You previously had 2408195 packed objects and after the repack this is 
2674504.  The difference is 266309, which incidentally is the number of 
objects 'git repack' counted.

If that 266309 object count is stable between repack attempts (and it should), then something is failing to clean up the old packs. If I remember right, you were playing with modifications to git-repack.sh lately? It is well possible that you broke it.

I tried to simulate your setup but I'm regretting it now. My test repo expanded to 10x its original size with 900 packs. Git is also painfully slow now while I'm trying to repack it properly back into a single pack. However I wasn't able to reproduce your repository growth. Please retest with a vanilla Git version.

I think you should really consider using an unlimited pack size. Not only you'll save quite a lot of disk space, but Git will be much faster too. And Git itself is far more efficient than rsync at keeping Git repositories in sync, regardless of pack sizes (unless you have too many packs of course).

Nicolas
Jon Nelson· Feb 8, 2010, 17:12 UTC · re: Nicolas Pitre · lore

Re: git gc / git repack not removing unused objects?

On Sun, Feb 7, 2010 at 8:11 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
Show 17 quoted lines
> On Sun, 7 Feb 2010, Jon Nelson wrote:
>> On Sat, Feb 6, 2010 at 7:16 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> >On Sat, 6 Feb 2010, Jon Nelson wrote:
>> >>         packsizelimit = 256m
>> >
>> > Why are you using this?
>>
>> I didn't want my pack files to be too huge. I've bumped that up to 2G.
>>
>> >> pack.packsizelimit=2M
>>
>> My ~/.gitconfig normally uses 2M for easy rsyncing. In this repo I
>> thought the value was overridden by the project's config (which was
>> specifying 256m and now specifies 2048m).
>
> Why do you synchronize with rsync?  Why not a simple 'git fetch' which
> is likely to be much faster in any case?

Often I need to sync the entire config, and am frequently rsyncing (although not in this case) rsyncing repos made with git-svn. I've had trouble using git clone on these repos, as I really want the duplicate repo to use the svn upstream and not the git repo it was cloned from.

..
Show 8 quoted lines
> You previously had 2408195 packed objects and after the repack this is
> 2674504.  The difference is 266309, which incidentally is the number of
> objects 'git repack' counted.
>
> If that 266309 object count is stable between repack attempts (and it
> should), then something is failing to clean up the old packs.  If I
> remember right, you were playing with modifications to git-repack.sh
> lately?  It is well possible that you broke it.
Indeedly do I did.
The (now expected) results:

turnip:/mnt/backups.git # du -sh ; echo ; git count-objects -v ; echo ; git repack -ad ; echo ; git count-objects -v ; echo ; du -sh 204G .

count: 0 size: 0 in-pack: 2939744 packs: 694 size-pack: 213284091 prune-packable: 0 garbage: 0

Counting objects: 266309, done. Compressing objects: 100% (205176/205176), done. Writing objects: 100% (266309/266309), done. Total 266309 (delta 47781), reused 230413 (delta 47770)

count: 0 size: 0 in-pack: 266309 packs: 11 size-pack: 21590451 prune-packable: 0 garbage: 0

21G . turnip:/mnt/backups.git #

and of course now everything is nice and (reasonably) fast. Thanks for the help, and pointing out that the problem was .. me.

-- 
Jon

← back to recent threads