# git gc ineffective 8 messages from 2020-11-09 to 2021-01-07. Participants: Michal Suchánek, Andreas Schwab, Johannes Sixt. Thread: https://gitlist.dev/t/54604 ## Michal Suchánek, 2020-11-09 09:20 Subject: git gc ineffective Message-ID: <20201109092041.GV29778@kitsune.suse.cz> URL: https://gitlist.dev/e/20201109092041.GV29778%40kitsune.suse.cz ``` Hello, I am running git 2.29.1 I noticed I am running out of disk space, and one repository taking up about 38G. Did git gc --aggressive, and the used space *raised* to 42G, and git would report it does gc after every commit. Did gc without --aggressive, and the used space almost *doubled* to 75G (about 1G is checkout and some utracked files). The expected size is about 20G. Is there some way to debug gc to figure out what went wrong? Thanks Michal ``` ## Andreas Schwab, 2020-11-09 09:49 Subject: Re: git gc ineffective Message-ID: <87361ilv1a.fsf@igel.home> URL: https://gitlist.dev/e/87361ilv1a.fsf%40igel.home In-Reply-To: <20201109092041.GV29778@kitsune.suse.cz> ``` On Nov 09 2020, Michal Suchánek wrote: > I noticed I am running out of disk space, and one repository taking up > about 38G. Did git gc --aggressive, and the used space *raised* to 42G, > and git would report it does gc after every commit. Do you have a lot of loose objects? Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." ``` ## Michal Suchánek, 2020-11-09 10:17 Subject: Re: git gc ineffective Message-ID: <20201109101738.GW29778@kitsune.suse.cz> URL: https://gitlist.dev/e/20201109101738.GW29778%40kitsune.suse.cz In-Reply-To: <87361ilv1a.fsf@igel.home> ``` On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote: > On Nov 09 2020, Michal Suchánek wrote: > > > I noticed I am running out of disk space, and one repository taking up > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G, > > and git would report it does gc after every commit. > > Do you have a lot of loose objects? { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l 53392 > > Andreas. > > -- > Andreas Schwab, schwab@linux-m68k.org > GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 > "And now for something completely different." ``` ## Michal Suchánek, 2020-11-09 10:43 Subject: Re: git gc ineffective Message-ID: <20201109104327.GX29778@kitsune.suse.cz> URL: https://gitlist.dev/e/20201109104327.GX29778%40kitsune.suse.cz In-Reply-To: <20201109101738.GW29778@kitsune.suse.cz> ``` On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote: > On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote: > > On Nov 09 2020, Michal Suchánek wrote: > > > > > I noticed I am running out of disk space, and one repository taking up > > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G, > > > and git would report it does gc after every commit. > > > > Do you have a lot of loose objects? > { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l > 53392 And in the double-size repository it's doubled, too: { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l 101167 ``` ## Michal Suchánek, 2020-11-09 10:48 Subject: Re: git gc ineffective Message-ID: <20201109104854.GY29778@kitsune.suse.cz> URL: https://gitlist.dev/e/20201109104854.GY29778%40kitsune.suse.cz In-Reply-To: <20201109104327.GX29778@kitsune.suse.cz> ``` On Mon, Nov 09, 2020 at 11:43:27AM +0100, Michal Suchánek wrote: > On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote: > > On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote: > > > On Nov 09 2020, Michal Suchánek wrote: > > > > > > > I noticed I am running out of disk space, and one repository taking up > > > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G, > > > > and git would report it does gc after every commit. > > > > > > Do you have a lot of loose objects? > > { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l > > 53392 > And in the double-size repository it's doubled, too: > { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l > 101167 And server-side bare repository has 16G with 30836 loose objects and git 2.26.2. Thanks Michal ``` ## Andreas Schwab, 2020-11-09 11:17 Subject: Re: git gc ineffective Message-ID: <87v9eekcd6.fsf@igel.home> URL: https://gitlist.dev/e/87v9eekcd6.fsf%40igel.home In-Reply-To: <20201109104327.GX29778@kitsune.suse.cz> ``` On Nov 09 2020, Michal Suchánek wrote: > On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote: >> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote: >> > On Nov 09 2020, Michal Suchánek wrote: >> > >> > > I noticed I am running out of disk space, and one repository taking up >> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G, >> > > and git would report it does gc after every commit. >> > >> > Do you have a lot of loose objects? >> { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l >> 53392 > And in the double-size repository it's doubled, too: > { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l > 101167 git count-objects also shows the size. Does it help to prune them --exprire now? Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." ``` ## Michal Suchánek, 2021-01-07 18:35 Subject: Re: git gc ineffective Message-ID: <20210107183531.GB6564@kitsune.suse.cz> URL: https://gitlist.dev/e/20210107183531.GB6564%40kitsune.suse.cz In-Reply-To: <87v9eekcd6.fsf@igel.home> ``` Hello, On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote: > On Nov 09 2020, Michal Suchánek wrote: > > > On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote: > >> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote: > >> > On Nov 09 2020, Michal Suchánek wrote: > >> > > >> > > I noticed I am running out of disk space, and one repository taking up > >> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G, > >> > > and git would report it does gc after every commit. > >> > > >> > Do you have a lot of loose objects? > >> { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l > >> 53392 > > And in the double-size repository it's doubled, too: > > { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l > > 101167 > > git count-objects also shows the size. $ git count-objects 59853 objects, 43249880 kilobytes $ du -hs .git 48G .git $ git gc --aggressive Enumerating objects: 1825080, done. Counting objects: 100% (1825080/1825080), done. Delta compression using up to 4 threads Compressing objects: 100% (1803925/1803925), done. Writing objects: 100% (1825080/1825080), done. Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0 Removing duplicate objects: 100% (256/256), done. Checking connectivity: 2003814, done. Expanding reachable commits in commit graph: 337512, done. $ du -hs .git 172G .git $ git count-objects 178734 objects, 175309572 kilobytes > Does it help to prune them --exprire now? $ git prune Checking connectivity: 1825478, done. $ du -hs .git 3.9G .git $ git --version git version 2.26.2 So it is my wrong expectation that 'gc' comand removes garbage. It creates it en masse. It just makes is in a way that the 'prune' command that really reoves garbage can now remove it. Thanks Michal ``` ## Johannes Sixt, 2021-01-07 21:48 Subject: Re: git gc ineffective Message-ID: <07798280-9818-0d21-5c63-dfc1c621082a@kdbg.org> URL: https://gitlist.dev/e/07798280-9818-0d21-5c63-dfc1c621082a%40kdbg.org In-Reply-To: <20210107183531.GB6564@kitsune.suse.cz> ``` Am 07.01.21 um 19:35 schrieb Michal Suchánek: > Hello, > > On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote: >> On Nov 09 2020, Michal Suchánek wrote: >> >>> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote: >>>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote: >>>>> On Nov 09 2020, Michal Suchánek wrote: >>>>> >>>>>> I noticed I am running out of disk space, and one repository taking up >>>>>> about 38G. Did git gc --aggressive, and the used space *raised* to 42G, >>>>>> and git would report it does gc after every commit. >>>>> >>>>> Do you have a lot of loose objects? >>>> { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l >>>> 53392 >>> And in the double-size repository it's doubled, too: >>> { for i in .git/objects/?? ; do ls $i ; done ; } | wc -l >>> 101167 >> >> git count-objects also shows the size. > $ git count-objects > 59853 objects, 43249880 kilobytes > $ du -hs .git > 48G .git > $ git gc --aggressive > Enumerating objects: 1825080, done. > Counting objects: 100% (1825080/1825080), done. > Delta compression using up to 4 threads > Compressing objects: 100% (1803925/1803925), done. > Writing objects: 100% (1825080/1825080), done. > Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0 > Removing duplicate objects: 100% (256/256), done. > Checking connectivity: 2003814, done. > Expanding reachable commits in commit graph: 337512, done. > $ du -hs .git > 172G .git > $ git count-objects > 178734 objects, 175309572 kilobytes > >> Does it help to prune them --exprire now? > > $ git prune > Checking connectivity: 1825478, done. > $ du -hs .git > 3.9G .git > $ git --version > git version 2.26.2 > > So it is my wrong expectation that 'gc' comand removes garbage. It > creates it en masse. > > It just makes is in a way that the 'prune' command that really reoves > garbage can now remove it. It's an unfortunate default behavior of `git gc`. Set gc.pruneExpire to 'now' to countermand it, but watch out for the caveats. See https://stackoverflow.com/questions/55414916/how-to-avoid-that-git-gc-generates-garbage-loose-objects for more details. -- Hannes ```