Re: git tag triggers out-of-memory killer

From: Jeff King <peff@peff.net>
Date: Jul 11, 2022, 17:50 UTC
Message-ID: <Ysxiyn0w/4oTQ7ks@coredump.intra.peff.net>
In-Reply-To: <20220711171537.19d058da.olaf@aepfle.de>

On Mon, Jul 11, 2022 at 05:15:37PM +0200, Olaf Hering wrote:

> I ran top(1), all the git processes were competing for memory.
> There was most of the time no memory shared, according to top.

OK, that makes sense if they really are allocating a lot of heap. They _would_ be sharing mmap'd pages, but the memory pressure causes the OS to evict as many of those as possible.

Show 6 quoted lines

> This tool to process a single repository exists since a few years.
> In the past I ran it on a 12cpu/64GB machine. Today it was running the
> first time on a 96cpu/64GB machine. There is a slim chance the issue did
> not show up because the 12 processes always had access to enough memory.
> 
> I will try your suggestions.

In case you haven't seen it, read the end of:

  https://lore.kernel.org/git/YsxiSwQGvLhzNQrt@coredump.intra.peff.net/

I realized my reproduction on linux.git was not traversing a wide enough chunk of history. Fixing that, I do see ~1GB of heap allocation. My patch drops that substantially, but you will still be much better off building a commit-graph file.

-Peff