threads / discuss / 58132
Subject: git tag triggers out-of-memory killer
Olaf HeringWhat knobs exist inside git to restrict the amount of memory for each individual git process?
Running a large number of "git tag --sort=taggerdate --contains " processes in parallel triggers the OOM killer because each one allocates more than one gigabyte resident memory.
I tried to set all knobs from git-config(1) that appear to be related to memory usage, but nothing seems to have an effect.
Thanks, Olaf
Jeff KingRe: git tag triggers out-of-memory killer
On Mon, Jul 11, 2022 at 03:31:35PM +0200, Olaf Hering wrote:
> What knobs exist inside git to restrict the amount of memory for each > individual git process?
You can set GIT_ALLOC_LIMIT in the environment to limit the size of a single heap allocation. But there's no overall limit for all allocations. You'd have to use an OS-level tool like ulimit or cgroups there.
> Running a large number of "git tag --sort=taggerdate --contains " > processes in parallel triggers the OOM killer because each one > allocates more than one gigabyte resident memory.
How did you measure? Are you sure that each one is allocating a gigabyte itself, or might some of it be shared between the processes? Git will mmap the packfiles on disk, which will count against RSS (assuming the memory even gets faulted in). But multiple processes on the same repository will share those read-only pages.
Are you running the command against a large number of distinct repositories? In that case, the pages obviously wouldn't be shared. You _might_ have some success with core.packedGitLimit, which would lower the amount that each process will mmap. But in general I'd expect the OS to happily evict read-only mmap'd pages rather than OOM.
Is there anything about your repository that might be unusual? A large number of commits, or tags? If I run "git tag --sort=taggerdate --contains" against linux.git, measuring with massif shows it uses about 2MB of heap. Digging for a much older commit, like 1da177e4c3, uses about 16MB (there's some per-commit internal bookkeeping for each commit we actually have to traverse).
Those are both with commit-graphs enabled (i.e., running "git commit-graph write --reachable"). Without them, it looks like the heap is closer to 100MB. Probably git-tag should be disabling save_commit_buffer internally. Unlike git-log, it won't end up pretty-printing the commits it walks during the --contains traversal.
So maybe try with this patch:
diff --git a/builtin/tag.c b/builtin/tag.c index 75dece0e4f..8f3a6dffb4 100644 --- a/builtin/tag.c +++ b/builtin/tag.c @@ -486,6 +486,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix) int ret = 0; const char *only_in_list = NULL; + save_commit_buffer = 0; + setup_ref_filter_porcelain_msg(); git_config(git_tag_config, &sorting_options); or with commit graphs written, and see if that improves things for you. Those numbers aren't anywhere near the 1GB you mentioned, but perhaps a sufficient number of 100MB heaps is enough to cause problems for you. -Peff
Olaf HeringRe: git tag triggers out-of-memory killer
Mon, 11 Jul 2022 10:06:32 -0400 Jeff King <peff@peff.net>:
> How did you measure? Are you sure that each one is allocating a > gigabyte itself, or might some of it be shared between the processes? > Git will mmap the packfiles on disk, which will count against RSS > (assuming the memory even gets faulted in). But multiple processes on > the same repository will share those read-only pages.
I ran top(1), all the git processes were competing for memory. There was most of the time no memory shared, according to top.
This tool to process a single repository exists since a few years. In the past I ran it on a 12cpu/64GB machine. Today it was running the first time on a 96cpu/64GB machine. There is a slim chance the issue did not show up because the 12 processes always had access to enough memory.
I will try your suggestions.
Thanks. Olaf
Jeff KingRe: git tag triggers out-of-memory killer
On Mon, Jul 11, 2022 at 05:15:37PM +0200, Olaf Hering wrote:
> I ran top(1), all the git processes were competing for memory. > There was most of the time no memory shared, according to top.
OK, that makes sense if they really are allocating a lot of heap. They _would_ be sharing mmap'd pages, but the memory pressure causes the OS to evict as many of those as possible.
> This tool to process a single repository exists since a few years. > In the past I ran it on a 12cpu/64GB machine. Today it was running the > first time on a 96cpu/64GB machine. There is a slim chance the issue did > not show up because the 12 processes always had access to enough memory. > > I will try your suggestions.
In case you haven't seen it, read the end of:
https://lore.kernel.org/git/YsxiSwQGvLhzNQrt@coredump.intra.peff.net/
I realized my reproduction on linux.git was not traversing a wide enough chunk of history. Fixing that, I do see ~1GB of heap allocation. My patch drops that substantially, but you will still be much better off building a commit-graph file.
-Peff
Olaf HeringRe: git tag triggers out-of-memory killer
Mon, 11 Jul 2022 10:06:32 -0400 Jeff King <peff@peff.net>:
> --- a/builtin/tag.c > +++ b/builtin/tag.c > @@ -486,6 +486,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix) > int ret = 0; > const char *only_in_list = NULL; > > + save_commit_buffer = 0; > + > setup_ref_filter_porcelain_msg(); > > git_config(git_tag_config, &sorting_options); >
Thanks, this patch helps.
The individual git processes start with a maximum of ~320M, and slowly reach around ~600M as maximum.
I was not aware of commit-graph. After writing a commit-graph the processing is much faster, and an unmodified git 2.35.3 uses ~320M as a maximum.
Olaf