{"thread":{"id":"58132","subject":"git tag triggers out-of-memory killer","startedAt":"2022-07-11T13:37:55Z","lastAt":"2022-07-12T10:22:43Z","messageCount":5,"participants":["Olaf Hering","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"458787","messageId":"20220711153135.35b1b135.olaf@aepfle.de","threadId":"58132","inReplyTo":null,"subject":"git tag triggers out-of-memory killer","fromName":"Olaf Hering","fromEmail":"olaf@aepfle.de","sentAt":"2022-07-11T13:31:35Z","receivedAt":"2022-07-11T13:37:55Z","isPatch":false,"sender":{"key":"olaf@aepfle.de","avatar":"https://avatars.githubusercontent.com/u/942324?v=4"},"body":"What knobs exist inside git to restrict the amount of memory for each individual git process?\n\nRunning a large number of \"git tag --sort=taggerdate --contains \" processes in parallel triggers the OOM killer because each one allocates more than one gigabyte resident memory.\n\nI tried to set all knobs from git-config(1) that appear to be related to memory usage, but nothing seems to have an effect.\n\n\nThanks,\nOlaf\n"},{"id":"458789","messageId":"YswuaPx6Mk7YkIim@coredump.intra.peff.net","threadId":"58132","inReplyTo":"20220711153135.35b1b135.olaf@aepfle.de","subject":"Re: git tag triggers out-of-memory killer","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-07-11T14:06:32Z","receivedAt":"2022-07-11T14:06:37Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 11, 2022 at 03:31:35PM +0200, Olaf Hering wrote:\n\n> What knobs exist inside git to restrict the amount of memory for each\n> individual git process?\n\nYou can set GIT_ALLOC_LIMIT in the environment to limit the size of a\nsingle heap allocation. But there's no overall limit for all\nallocations. You'd have to use an OS-level tool like ulimit or cgroups\nthere.\n\n> Running a large number of \"git tag --sort=taggerdate --contains \"\n> processes in parallel triggers the OOM killer because each one\n> allocates more than one gigabyte resident memory.\n\nHow did you measure?  Are you sure that each one is allocating a\ngigabyte itself, or might some of it be shared between the processes?\nGit will mmap the packfiles on disk, which will count against RSS\n(assuming the memory even gets faulted in). But multiple processes on\nthe same repository will share those read-only pages.\n\nAre you running the command against a large number of distinct\nrepositories? In that case, the pages obviously wouldn't be shared. You\n_might_ have some success with core.packedGitLimit, which would lower\nthe amount that each process will mmap. But in general I'd expect the OS\nto happily evict read-only mmap'd pages rather than OOM.\n\nIs there anything about your repository that might be unusual? A large\nnumber of commits, or tags?  If I run \"git tag --sort=taggerdate\n--contains\" against linux.git, measuring with massif shows it uses about\n2MB of heap. Digging for a much older commit, like 1da177e4c3, uses\nabout 16MB (there's some per-commit internal bookkeeping for each commit\nwe actually have to traverse).\n\nThose are both with commit-graphs enabled (i.e., running \"git\ncommit-graph write --reachable\"). Without them, it looks like the heap\nis closer to 100MB. Probably git-tag should be disabling\nsave_commit_buffer internally. Unlike git-log, it won't end up\npretty-printing the commits it walks during the --contains traversal.\n\nSo maybe try with this patch:\n\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 75dece0e4f..8f3a6dffb4 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -486,6 +486,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tint ret = 0;\n \tconst char *only_in_list = NULL;\n \n+\tsave_commit_buffer = 0;\n+\n \tsetup_ref_filter_porcelain_msg();\n \n \tgit_config(git_tag_config, &sorting_options);\n\nor with commit graphs written, and see if that improves things for you.\nThose numbers aren't anywhere near the 1GB you mentioned, but perhaps a\nsufficient number of 100MB heaps is enough to cause problems for you.\n\n-Peff\n"},{"id":"458800","messageId":"20220711171537.19d058da.olaf@aepfle.de","threadId":"58132","inReplyTo":"YswuaPx6Mk7YkIim@coredump.intra.peff.net","subject":"Re: git tag triggers out-of-memory killer","fromName":"Olaf Hering","fromEmail":"olaf@aepfle.de","sentAt":"2022-07-11T15:15:37Z","receivedAt":"2022-07-11T15:15:49Z","isPatch":false,"sender":{"key":"olaf@aepfle.de","avatar":"https://avatars.githubusercontent.com/u/942324?v=4"},"body":"Mon, 11 Jul 2022 10:06:32 -0400 Jeff King <peff@peff.net>:\n\n> How did you measure?  Are you sure that each one is allocating a\n> gigabyte itself, or might some of it be shared between the processes?\n> Git will mmap the packfiles on disk, which will count against RSS\n> (assuming the memory even gets faulted in). But multiple processes on\n> the same repository will share those read-only pages.\n\n\nI ran top(1), all the git processes were competing for memory.\nThere was most of the time no memory shared, according to top.\n\nThis tool to process a single repository exists since a few years.\nIn the past I ran it on a 12cpu/64GB machine. Today it was running the\nfirst time on a 96cpu/64GB machine. There is a slim chance the issue did\nnot show up because the 12 processes always had access to enough memory.\n\n\nI will try your suggestions.\n\nThanks.\nOlaf\n"},{"id":"458807","messageId":"Ysxiyn0w/4oTQ7ks@coredump.intra.peff.net","threadId":"58132","inReplyTo":"20220711171537.19d058da.olaf@aepfle.de","subject":"Re: git tag triggers out-of-memory killer","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-07-11T17:50:02Z","receivedAt":"2022-07-11T17:50:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 11, 2022 at 05:15:37PM +0200, Olaf Hering wrote:\n\n> I ran top(1), all the git processes were competing for memory.\n> There was most of the time no memory shared, according to top.\n\nOK, that makes sense if they really are allocating a lot of heap. They\n_would_ be sharing mmap'd pages, but the memory pressure causes the OS\nto evict as many of those as possible.\n\n> This tool to process a single repository exists since a few years.\n> In the past I ran it on a 12cpu/64GB machine. Today it was running the\n> first time on a 96cpu/64GB machine. There is a slim chance the issue did\n> not show up because the 12 processes always had access to enough memory.\n> \n> I will try your suggestions.\n\nIn case you haven't seen it, read the end of:\n\n  https://lore.kernel.org/git/YsxiSwQGvLhzNQrt@coredump.intra.peff.net/\n\nI realized my reproduction on linux.git was not traversing a wide enough\nchunk of history. Fixing that, I do see ~1GB of heap allocation. My\npatch drops that substantially, but you will still be much better off\nbuilding a commit-graph file.\n\n-Peff\n"},{"id":"458869","messageId":"20220712122226.183fff9b.olaf@aepfle.de","threadId":"58132","inReplyTo":"YswuaPx6Mk7YkIim@coredump.intra.peff.net","subject":"Re: git tag triggers out-of-memory killer","fromName":"Olaf Hering","fromEmail":"olaf@aepfle.de","sentAt":"2022-07-12T10:22:26Z","receivedAt":"2022-07-12T10:22:43Z","isPatch":false,"sender":{"key":"olaf@aepfle.de","avatar":"https://avatars.githubusercontent.com/u/942324?v=4"},"body":"Mon, 11 Jul 2022 10:06:32 -0400 Jeff King <peff@peff.net>:\n\n> --- a/builtin/tag.c\n> +++ b/builtin/tag.c\n> @@ -486,6 +486,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n>  \tint ret = 0;\n>  \tconst char *only_in_list = NULL;\n>  \n> +\tsave_commit_buffer = 0;\n> +\n>  \tsetup_ref_filter_porcelain_msg();\n>  \n>  \tgit_config(git_tag_config, &sorting_options);\n> \n\n\nThanks, this patch helps.\n\nThe individual git processes start with a maximum of ~320M, and slowly reach around ~600M as maximum.\n\nI was not aware of commit-graph. After writing a commit-graph the processing is much faster, and an unmodified git 2.35.3 uses ~320M as a maximum.\n\nOlaf\n"}]}