Thank you Brian and Bryan. You both clarified what was happening and now I know what to look for.
I can use a shallow clone for most repos, but there are some I want to keep history for. I don't need a full copy of this repo, but it was a good repo to show the issue I was facing.
Thanks again!
---- On Mon, 08 Mar 2021 23:56:53 +0000 brian m. carlson <sandals@crustytoothpaste.net> wrote ----
> On 2021-03-08 at 22:29:16, Bryan Turner wrote:
> > On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller <anthony@monospace.sh> wrote:
> > >
> > > What did you do before the bug happened? (Steps to reproduce your issue)
> > >
> > > git clone https://github.com/notracking/hosts-blocklists
> > > cd hosts-blocklists
> > > git reflog expire --all --expire=now && git gc --prune=now --aggressive
> >
> > --aggressive tells git gc to discard all of its existing delta chains
> > and go find new ones, and to be fairly aggressive in how it looks for
> > candidates. This is going to be the primary source of the resource
> > usage you see, as well as the time.
> >
> > Aggressive GCs are something you do once in a (very great) while. If
> > you try this without the --aggressive, how does it look?
>
> I should point out that this repository is also rather pathologically
> structured. Almost every commit is an automatic commit updating the
> same five files which are text files ranging from 5 MB to 11 MB.
>
> When you use --aggressive, as Bryan pointed out, you're asking to throw
> away all the deltas and try really hard to compute all of them fresh.
> That's going to use a lot of memory because you're loading many large
> text files into memory. It's also going to use a lot of CPU because
> these files do indeed delta extremely well, and since computing deltas
> on larger files is more expensive, especially when there are many of
> them.
>
> And that's just the blobs. The trees and commits are also going to be
> nearly identically structured and will also delta well with virtually
> every other similar object of their type. Normally Git sorts by size
> which helps pick better candidates, but since these are all going to be
> identically sized, the performance is going to suffer.
>
> Now, I have the advantage in this case of being a person who's sometimes
> on call for the maintenance of Git repositories and in that capacity,
> that this is pathologically structured is obvious to me. But, yeah, I
> would definitely not run --aggressive on this repo unless I needed to
> and I would not expect it to perform well.
> --
> brian m. carlson (he/him or they/them)
> Houston, Texas, US
>