threads / discuss / 54604

git gc ineffective

Subject: git gc ineffective

## tl;dr

8 messages between Nov 9, 2020 and Jan 7, 2021.

replies: 7people: 3as markdown or json

Michal Suchánek· Nov 9, 2020, 09:20 UTC · lore
Hello,
I am running git 2.29.1

I noticed I am running out of disk space, and one repository taking up about 38G. Did git gc --aggressive, and the used space *raised* to 42G, and git would report it does gc after every commit. Did gc without --aggressive, and the used space almost *doubled* to 75G (about 1G is checkout and some utracked files).

The expected size is about 20G. Is there some way to debug gc to figure out what went wrong?

Thanks
Michal
Andreas Schwab· Nov 9, 2020, 09:49 UTC · re: Michal Suchánek · lore

Re: git gc ineffective

On Nov 09 2020, Michal Suchánek wrote:
> I noticed I am running out of disk space, and one repository taking up
> about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> and git would report it does gc after every commit.
Do you have a lot of loose objects?
Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Michal Suchánek· Nov 9, 2020, 10:17 UTC · re: Andreas Schwab · lore

Re: git gc ineffective

On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
Show 7 quoted lines
> On Nov 09 2020, Michal Suchánek wrote:
> 
> > I noticed I am running out of disk space, and one repository taking up
> > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > and git would report it does gc after every commit.
> 
> Do you have a lot of loose objects?

{ for i in .git/objects/?? ; do ls $i ; done ; } | wc -l 53392

Show 7 quoted lines
> 
> Andreas.
> 
> -- 
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> "And now for something completely different."
Michal Suchánek· Nov 9, 2020, 10:43 UTC · re: Michal Suchánek · lore

Re: git gc ineffective

On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
Show 10 quoted lines
> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> > On Nov 09 2020, Michal Suchánek wrote:
> > 
> > > I noticed I am running out of disk space, and one repository taking up
> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > > and git would report it does gc after every commit.
> > 
> > Do you have a lot of loose objects?
> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> 53392
And in the double-size repository it's doubled, too:
 { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
 101167
Michal Suchánek· Nov 9, 2020, 10:48 UTC · re: Michal Suchánek · lore

Re: git gc ineffective

On Mon, Nov 09, 2020 at 11:43:27AM +0100, Michal Suchánek wrote:
Show 14 quoted lines
> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
> > On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> > > On Nov 09 2020, Michal Suchánek wrote:
> > > 
> > > > I noticed I am running out of disk space, and one repository taking up
> > > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > > > and git would report it does gc after every commit.
> > > 
> > > Do you have a lot of loose objects?
> > { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> > 53392
> And in the double-size repository it's doubled, too:
>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>  101167

And server-side bare repository has 16G with 30836 loose objects and git 2.26.2.

Thanks
Michal
Andreas Schwab· Nov 9, 2020, 11:17 UTC · re: Michal Suchánek · lore

Re: git gc ineffective

On Nov 09 2020, Michal Suchánek wrote:
Show 14 quoted lines
> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
>> > On Nov 09 2020, Michal Suchánek wrote:
>> > 
>> > > I noticed I am running out of disk space, and one repository taking up
>> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
>> > > and git would report it does gc after every commit.
>> > 
>> > Do you have a lot of loose objects?
>> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>> 53392
> And in the double-size repository it's doubled, too:
>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>  101167
git count-objects also shows the size.
Does it help to prune them --exprire now?
Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Michal Suchánek· Jan 7, 2021, 18:35 UTC · re: Andreas Schwab · lore

Re: git gc ineffective

Hello,
On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote:
Show 18 quoted lines
> On Nov 09 2020, Michal Suchánek wrote:
> 
> > On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
> >> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> >> > On Nov 09 2020, Michal Suchánek wrote:
> >> > 
> >> > > I noticed I am running out of disk space, and one repository taking up
> >> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> >> > > and git would report it does gc after every commit.
> >> > 
> >> > Do you have a lot of loose objects?
> >> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> >> 53392
> > And in the double-size repository it's doubled, too:
> >  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> >  101167
> 
> git count-objects also shows the size.

$ git count-objects 59853 objects, 43249880 kilobytes $ du -hs .git 48G .git $ git gc --aggressive Enumerating objects: 1825080, done. Counting objects: 100% (1825080/1825080), done. Delta compression using up to 4 threads Compressing objects: 100% (1803925/1803925), done. Writing objects: 100% (1825080/1825080), done. Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0 Removing duplicate objects: 100% (256/256), done. Checking connectivity: 2003814, done. Expanding reachable commits in commit graph: 337512, done. $ du -hs .git 172G .git $ git count-objects 178734 objects, 175309572 kilobytes

> Does it help to prune them --exprire now?

$ git prune Checking connectivity: 1825478, done. $ du -hs .git 3.9G .git $ git --version git version 2.26.2

So it is my wrong expectation that 'gc' comand removes garbage. It creates it en masse.

It just makes is in a way that the 'prune' command that really reoves garbage can now remove it.

Thanks
Michal
Johannes Sixt· Jan 7, 2021, 21:48 UTC · re: Michal Suchánek · lore

Re: git gc ineffective

Am 07.01.21 um 19:35 schrieb Michal Suchánek:
Show 54 quoted lines
> Hello,
> 
> On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote:
>> On Nov 09 2020, Michal Suchánek wrote:
>>
>>> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
>>>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
>>>>> On Nov 09 2020, Michal Suchánek wrote:
>>>>>
>>>>>> I noticed I am running out of disk space, and one repository taking up
>>>>>> about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
>>>>>> and git would report it does gc after every commit.
>>>>>
>>>>> Do you have a lot of loose objects?
>>>> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>>>> 53392
>>> And in the double-size repository it's doubled, too:
>>>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>>>  101167
>>
>> git count-objects also shows the size.
> $ git count-objects
> 59853 objects, 43249880 kilobytes
> $ du -hs .git
> 48G     .git
> $ git gc --aggressive
> Enumerating objects: 1825080, done.
> Counting objects: 100% (1825080/1825080), done.
> Delta compression using up to 4 threads
> Compressing objects: 100% (1803925/1803925), done.
> Writing objects: 100% (1825080/1825080), done.
> Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0
> Removing duplicate objects: 100% (256/256), done.
> Checking connectivity: 2003814, done.
> Expanding reachable commits in commit graph: 337512, done.
> $ du -hs .git
> 172G    .git
> $ git count-objects
> 178734 objects, 175309572 kilobytes
> 
>> Does it help to prune them --exprire now?
> 
> $ git prune
> Checking connectivity: 1825478, done.
> $ du -hs .git
> 3.9G    .git
> $ git --version
> git version 2.26.2
> 
> So it is my wrong expectation that 'gc' comand removes garbage. It
> creates it en masse.
> 
> It just makes is in a way that the 'prune' command that really reoves
> garbage can now remove it.

It's an unfortunate default behavior of `git gc`. Set gc.pruneExpire to 'now' to countermand it, but watch out for the caveats.

See https://stackoverflow.com/questions/55414916/how-to-avoid-that-git-gc-generates-garbage-loose-objects for more details.

-- Hannes

← back to recent threads