{"thread":{"id":"54604","subject":"git gc ineffective","startedAt":"2020-11-09T09:20:45Z","lastAt":"2021-01-07T21:49:55Z","messageCount":8,"participants":["Michal Suchánek","Andreas Schwab","Johannes Sixt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"409378","messageId":"20201109092041.GV29778@kitsune.suse.cz","threadId":"54604","inReplyTo":null,"subject":"git gc ineffective","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2020-11-09T09:20:41Z","receivedAt":"2020-11-09T09:20:45Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"\nHello,\n\nI am running git 2.29.1\n\nI noticed I am running out of disk space, and one repository taking up\nabout 38G. Did git gc --aggressive, and the used space *raised* to 42G,\nand git would report it does gc after every commit. Did gc without\n--aggressive, and the used space almost *doubled* to 75G (about 1G is\ncheckout and some utracked files).\n\nThe expected size is about 20G. Is there some way to debug gc to figure\nout what went wrong?\n\nThanks\n\nMichal\n"},{"id":"409379","messageId":"87361ilv1a.fsf@igel.home","threadId":"54604","inReplyTo":"20201109092041.GV29778@kitsune.suse.cz","subject":"Re: git gc ineffective","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2020-11-09T09:49:21Z","receivedAt":"2020-11-09T09:49:28Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Nov 09 2020, Michal Suchánek wrote:\n\n> I noticed I am running out of disk space, and one repository taking up\n> about 38G. Did git gc --aggressive, and the used space *raised* to 42G,\n> and git would report it does gc after every commit.\n\nDo you have a lot of loose objects?\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"409385","messageId":"20201109101738.GW29778@kitsune.suse.cz","threadId":"54604","inReplyTo":"87361ilv1a.fsf@igel.home","subject":"Re: git gc ineffective","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2020-11-09T10:17:38Z","receivedAt":"2020-11-09T10:17:42Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:\n> On Nov 09 2020, Michal Suchánek wrote:\n> \n> > I noticed I am running out of disk space, and one repository taking up\n> > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,\n> > and git would report it does gc after every commit.\n> \n> Do you have a lot of loose objects?\n{ for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n53392\n\n> \n> Andreas.\n> \n> -- \n> Andreas Schwab, schwab@linux-m68k.org\n> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1\n> \"And now for something completely different.\"\n"},{"id":"409386","messageId":"20201109104327.GX29778@kitsune.suse.cz","threadId":"54604","inReplyTo":"20201109101738.GW29778@kitsune.suse.cz","subject":"Re: git gc ineffective","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2020-11-09T10:43:27Z","receivedAt":"2020-11-09T10:43:30Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:\n> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:\n> > On Nov 09 2020, Michal Suchánek wrote:\n> > \n> > > I noticed I am running out of disk space, and one repository taking up\n> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,\n> > > and git would report it does gc after every commit.\n> > \n> > Do you have a lot of loose objects?\n> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n> 53392\nAnd in the double-size repository it's doubled, too:\n { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n 101167\n"},{"id":"409387","messageId":"20201109104854.GY29778@kitsune.suse.cz","threadId":"54604","inReplyTo":"20201109104327.GX29778@kitsune.suse.cz","subject":"Re: git gc ineffective","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2020-11-09T10:48:54Z","receivedAt":"2020-11-09T10:48:57Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"On Mon, Nov 09, 2020 at 11:43:27AM +0100, Michal Suchánek wrote:\n> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:\n> > On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:\n> > > On Nov 09 2020, Michal Suchánek wrote:\n> > > \n> > > > I noticed I am running out of disk space, and one repository taking up\n> > > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,\n> > > > and git would report it does gc after every commit.\n> > > \n> > > Do you have a lot of loose objects?\n> > { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n> > 53392\n> And in the double-size repository it's doubled, too:\n>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n>  101167\nAnd server-side bare repository has 16G with 30836 loose objects and git\n2.26.2.\n\nThanks\n\nMichal\n"},{"id":"409389","messageId":"87v9eekcd6.fsf@igel.home","threadId":"54604","inReplyTo":"20201109104327.GX29778@kitsune.suse.cz","subject":"Re: git gc ineffective","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2020-11-09T11:17:57Z","receivedAt":"2020-11-09T11:18:02Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Nov 09 2020, Michal Suchánek wrote:\n\n> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:\n>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:\n>> > On Nov 09 2020, Michal Suchánek wrote:\n>> > \n>> > > I noticed I am running out of disk space, and one repository taking up\n>> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,\n>> > > and git would report it does gc after every commit.\n>> > \n>> > Do you have a lot of loose objects?\n>> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n>> 53392\n> And in the double-size repository it's doubled, too:\n>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n>  101167\n\ngit count-objects also shows the size.\n\nDoes it help to prune them --exprire now?\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"413718","messageId":"20210107183531.GB6564@kitsune.suse.cz","threadId":"54604","inReplyTo":"87v9eekcd6.fsf@igel.home","subject":"Re: git gc ineffective","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2021-01-07T18:35:32Z","receivedAt":"2021-01-07T18:36:15Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"Hello,\n\nOn Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote:\n> On Nov 09 2020, Michal Suchánek wrote:\n> \n> > On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:\n> >> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:\n> >> > On Nov 09 2020, Michal Suchánek wrote:\n> >> > \n> >> > > I noticed I am running out of disk space, and one repository taking up\n> >> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,\n> >> > > and git would report it does gc after every commit.\n> >> > \n> >> > Do you have a lot of loose objects?\n> >> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n> >> 53392\n> > And in the double-size repository it's doubled, too:\n> >  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n> >  101167\n> \n> git count-objects also shows the size.\n$ git count-objects\n59853 objects, 43249880 kilobytes\n$ du -hs .git\n48G     .git\n$ git gc --aggressive\nEnumerating objects: 1825080, done.\nCounting objects: 100% (1825080/1825080), done.\nDelta compression using up to 4 threads\nCompressing objects: 100% (1803925/1803925), done.\nWriting objects: 100% (1825080/1825080), done.\nTotal 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0\nRemoving duplicate objects: 100% (256/256), done.\nChecking connectivity: 2003814, done.\nExpanding reachable commits in commit graph: 337512, done.\n$ du -hs .git\n172G    .git\n$ git count-objects\n178734 objects, 175309572 kilobytes\n\n> Does it help to prune them --exprire now?\n\n$ git prune\nChecking connectivity: 1825478, done.\n$ du -hs .git\n3.9G    .git\n$ git --version\ngit version 2.26.2\n\nSo it is my wrong expectation that 'gc' comand removes garbage. It\ncreates it en masse.\n\nIt just makes is in a way that the 'prune' command that really reoves\ngarbage can now remove it.\n\nThanks\n\nMichal\n"},{"id":"413765","messageId":"07798280-9818-0d21-5c63-dfc1c621082a@kdbg.org","threadId":"54604","inReplyTo":"20210107183531.GB6564@kitsune.suse.cz","subject":"Re: git gc ineffective","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2021-01-07T21:48:54Z","receivedAt":"2021-01-07T21:49:55Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 07.01.21 um 19:35 schrieb Michal Suchánek:\n> Hello,\n> \n> On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote:\n>> On Nov 09 2020, Michal Suchánek wrote:\n>>\n>>> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:\n>>>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:\n>>>>> On Nov 09 2020, Michal Suchánek wrote:\n>>>>>\n>>>>>> I noticed I am running out of disk space, and one repository taking up\n>>>>>> about 38G. Did git gc --aggressive, and the used space *raised* to 42G,\n>>>>>> and git would report it does gc after every commit.\n>>>>>\n>>>>> Do you have a lot of loose objects?\n>>>> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n>>>> 53392\n>>> And in the double-size repository it's doubled, too:\n>>>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l\n>>>  101167\n>>\n>> git count-objects also shows the size.\n> $ git count-objects\n> 59853 objects, 43249880 kilobytes\n> $ du -hs .git\n> 48G     .git\n> $ git gc --aggressive\n> Enumerating objects: 1825080, done.\n> Counting objects: 100% (1825080/1825080), done.\n> Delta compression using up to 4 threads\n> Compressing objects: 100% (1803925/1803925), done.\n> Writing objects: 100% (1825080/1825080), done.\n> Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0\n> Removing duplicate objects: 100% (256/256), done.\n> Checking connectivity: 2003814, done.\n> Expanding reachable commits in commit graph: 337512, done.\n> $ du -hs .git\n> 172G    .git\n> $ git count-objects\n> 178734 objects, 175309572 kilobytes\n> \n>> Does it help to prune them --exprire now?\n> \n> $ git prune\n> Checking connectivity: 1825478, done.\n> $ du -hs .git\n> 3.9G    .git\n> $ git --version\n> git version 2.26.2\n> \n> So it is my wrong expectation that 'gc' comand removes garbage. It\n> creates it en masse.\n> \n> It just makes is in a way that the 'prune' command that really reoves\n> garbage can now remove it.\n\nIt's an unfortunate default behavior of `git gc`. Set gc.pruneExpire to\n'now' to countermand it, but watch out for the caveats.\n\nSee\nhttps://stackoverflow.com/questions/55414916/how-to-avoid-that-git-gc-generates-garbage-loose-objects\nfor more details.\n\n-- Hannes\n"}]}