{"thread":{"id":"62024","subject":"Unpredictable peak memory usage when using `git log` command","startedAt":"2024-08-30T12:20:29Z","lastAt":"2024-09-02T13:08:18Z","messageCount":6,"participants":["Yuri Karnilaev","Jeff King","Junio C Hamano","Patrick Steinhardt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"501918","messageId":"60B730E6-F3C6-4B57-94D6-E5A71754DAF3@gmail.com","threadId":"62024","inReplyTo":null,"subject":"Unpredictable peak memory usage when using `git log` command","fromName":"Yuri Karnilaev","fromEmail":"karnilaev@gmail.com","sentAt":"2024-08-30T12:20:15Z","receivedAt":"2024-08-30T12:20:29Z","isPatch":false,"sender":{"key":"karnilaev@gmail.com","avatar":null},"body":"Hello,\n\nI encountered an issue when using the `git log` command to retrieve commits in large repositories. My task is to iterate over all commits and output them in a specific format. However, my computer has limited memory, so I am looking for a way to reduce the memory consumption of this operation.\n\nI tested two different commands on the `torvalds/linux` repository as an example of a large repository and noticed a significant difference in peak memory usage:\n\n1. Processing all commits in one go:\n```\n/usr/bin/time -l -h -p git log --ignore-missing --pretty=format:%H%x02%P%x02%aN%x02%aE%x02%at%x00 --numstat > 1.txt\n```\nResult:\n```\nreal 594,01\nuser 562,22\nsys 12,43\n 7407976448 maximum resident set size\n 0 average shared memory size\n 0 average unshared data size\n 0 average unshared stack size\n 187437 page reclaims\n 274228 page faults\n 0 swaps\n 0 block input operations\n 0 block output operations\n 0 messages sent\n 0 messages received\n 0 signals received\n 1031 voluntary context switches\n 287056 involuntary context switches\n 5455479398547 instructions retired\n 1828253079874 cycles elapsed\n 135_616_064 peak memory footprint\n```\n\n2. Processing commits in batches:\n```\n/usr/bin/time -l -h -p git log --ignore-missing --pretty=format:%H%x02%P%x02%aN%x02%aE%x02%at%x00 -n 1000 --skip=1000000 --numstat > 1.txt\n```\nResult:\n```\nreal 9,83\nuser 7,48\nsys 0,40\n 2390540288 maximum resident set size\n 0 average shared memory size\n 0 average unshared data size\n 0 average unshared stack size\n 93487 page reclaims\n 52995 page faults\n 0 swaps\n 0 block input operations\n 0 block output operations\n 0 messages sent\n 0 messages received\n 0 signals received\n 634 voluntary context switches\n 14183 involuntary context switches\n 50173495540 instructions retired\n 24906960156 cycles elapsed\n 1_470_935_680 peak memory footprint\n```\n\nAs you can see from the results, the peak memory usage when processing commits in batches is 10 times higher than when processing all commits in one go.\nCan you please explain why this happens? Is there a way to work around this? Or maybe can you fix this in future Git versions?\n\nOperating System: Mac OS 14.6.1 (23G93)\nGit Version: 2.39.3 (Apple Git-146)\n\nBest regards,\nYuri"},{"id":"501936","messageId":"20240830205331.GA1038751@coredump.intra.peff.net","threadId":"62024","inReplyTo":"60B730E6-F3C6-4B57-94D6-E5A71754DAF3@gmail.com","subject":"[PATCH] revision: free commit buffers for skipped commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-30T20:53:31Z","receivedAt":"2024-08-30T20:53:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 30, 2024 at 03:20:15PM +0300, Yuri Karnilaev wrote:\n\n> As you can see from the results, the peak memory usage when processing\n> commits in batches is 10 times higher than when processing all commits\n> in one go.\n> Can you please explain why this happens? Is there a way to work around\n> this? Or maybe can you fix this in future Git versions?\n\nTry this:\n\n-- >8 --\nSubject: [PATCH] revision: free commit buffers for skipped commits\n\nIn git-log we leave the save_commit_buffer flag set to \"1\", which tells\nthe commit parsing code to store the object content after it has parsed\nit to find parents, tree, etc. That lets us reuse the contents for\npretty-printing the commit in the output. And then after printing each\ncommit, we call free_commit_buffer(), since we don't need it anymore.\n\nBut some options may cause us to traverse commits which are not part of\nthe output. And so git-log does not see them at all, and doesn't free\nthem. One such case is something like:\n\n git log -n 1000 --skip=1000000\n\nwhich will churn through a million commits, before showing only a\nthousand. We loop through these inside get_revision(), without freeing\nthe contents. As a result, we end up storing the object data for those\nmillion commits simultaneously.\n\nWe should free the stored buffers (if any) for those commits as we skip\nover them, which is what this patch does. Running the above command in\nlinux.git drops the peak heap usage from ~1.1GB to ~200MB, according to\nvalgrind/massif. (I thought we might get an even bigger improvement, but\nthe remaining memory is going to commit/tree structs, which we do hold\non to forever).\n\nNote that this problem doesn't occur if:\n\n - you're running a git-rev-list without a --format parameter; it turns\n off save_commit_buffer by default, since it only output the object\n id\n\n - you've built a commit-graph file, since in that case we'd use the\n optimized graph data instead of the initial parse, and then do a\n lazy parse for commits we're actually going to output\n\nThere are probably some other option combinations that can likewise\nend up with useless stored commit buffers. For example, if you ask for\n\"foo..bar\", then we'll have to walk down to the merge base, and\neverything on the \"foo\" side won't be shown. Tuning the \"save\" behavior\nto handle that might be tricky (I guess maybe drop buffers for anything\nwe mark as UNINTERESTING?). And in the long run, the right solution here\nis probably to make sure the commit-graph is built (since it fixes the\nmemory problem _and_ drastically reduces CPU usage).\n\nBut since this \"--skip\" case is an easy one-liner, it's worth fixing in\nthe meantime. It should be OK to make this call even if there is no\nsaved buffer (e.g., because save_commit_buffer=0, or because a\ncommit-graph was used), since it's O(1) to look up the buffer and is a\nnoop if it isn't present. I verified by running the above command after\n\"git commit-graph write --reachable\", and it takes the same time with\nand without this patch.\n\nReported-by: Yuri Karnilaev \nSigned-off-by: Jeff King \n---\n revision.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/revision.c b/revision.c\nindex ac94f8d429..2d7ad2bddf 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -4407,6 +4407,7 @@ static struct commit *get_revision_internal(struct rev_info *revs)\n \t\t\t\tc = get_revision_1(revs);\n \t\t\t\tif (!c)\n \t\t\t\t\tbreak;\n+\t\t\t\tfree_commit_buffer(revs->repo->parsed_objects, c);\n \t\t\t}\n \t\t}\n \n-- \n2.46.0.769.g1b22d789e3\n\n"},{"id":"501938","messageId":"20240830210607.GB1038751@coredump.intra.peff.net","threadId":"62024","inReplyTo":"60B730E6-F3C6-4B57-94D6-E5A71754DAF3@gmail.com","subject":"Re: Unpredictable peak memory usage when using `git log` command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-30T21:06:07Z","receivedAt":"2024-08-30T21:06:09Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 30, 2024 at 03:20:15PM +0300, Yuri Karnilaev wrote:\n\n> 2. Processing commits in batches:\n> ```\n> /usr/bin/time -l -h -p git log --ignore-missing --pretty=format:%H%x02%P%x02%aN%x02%aE%x02%at%x00 -n 1000 --skip=1000000 --numstat > 1.txt\n> ```\n> [...]\n> Operating System: Mac OS 14.6.1 (23G93)\n> Git Version: 2.39.3 (Apple Git-146)\n\nI sent a patch which I think should make things better for you, but I\nwanted to mention two things in a more general way:\n\n 1. You should really consider building a commit-graph file with \"git\n commit-graph write --reachable\". That will reduce the memory usage\n for this case, but also improve the CPU quite a bit (we won't have\n to open those million skipped commits to chase their parent\n pointers).\n\n I haven't kept up with the defaults for writing graph files. I\n thought gc.writeCommitGraph defaults to \"true\" these days, though\n that wouldn't help in a freshly cloned repository (arguably we\n should write the commit graph on clone?).\n\n 2. Using \"--skip\" still has to traverse all of those intermediate\n commits. So it's effectively quadratic in the number of commits\n overall (you end up skipping the first 1000 over and over).\n\n It's been a while since I've had to \"paginate\" segments of history\n like this, but a better solution is along the lines of:\n\n - use \"-n 1000\" to get 1000 commits in each chunk\n\n - use \"--boundary\" to report the commits that were queued to be\n\t traversed next but weren't shown\n\n - in invocations after the first one, start the traversal at\n\t those boundary commits, rather than HEAD\n\n You'll probably need to add \"%m\" to your format to show the\n boundaries (or alternatively, you can do the commit selection with\n rev-list, and then output the result to \"log --no-walk --stdin\" to\n do the pretty-printing).\n\n-Peff\n"},{"id":"501942","messageId":"xmqq1q25r99i.fsf@gitster.g","threadId":"62024","inReplyTo":"20240830205331.GA1038751@coredump.intra.peff.net","subject":"Re: [PATCH] revision: free commit buffers for skipped commits","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-30T21:27:53Z","receivedAt":"2024-08-30T21:27:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> But since this \"--skip\" case is an easy one-liner, it's worth fixing in\n> the meantime.\n\nOK.\n\n> diff --git a/revision.c b/revision.c\n> index ac94f8d429..2d7ad2bddf 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -4407,6 +4407,7 @@ static struct commit *get_revision_internal(struct rev_info *revs)\n> \t\t\t\tc = get_revision_1(revs);\n> \t\t\t\tif (!c)\n> \t\t\t\t\tbreak;\n> +\t\t\t\tfree_commit_buffer(revs->repo->parsed_objects, c);\n> \t\t\t}\n\nEven if we freed the buffer and then later need it, we'd read the\nbuffer again anyway, so this is a safe thing to do. And because\ncommits skipped in this separate loop will _never_ be given to the\ncaller of get_revision(), this it a reasonable optimization, too.\n\nWill queue. Thanks.\n"},{"id":"501951","messageId":"13D8CFB5-4073-42CC-BFDD-ADFE8FFE985C@gmail.com","threadId":"62024","inReplyTo":"20240830210607.GB1038751@coredump.intra.peff.net","subject":"Re: Unpredictable peak memory usage when using `git log` command","fromName":"Yuri Karnilaev","fromEmail":"karnilaev@gmail.com","sentAt":"2024-08-31T10:24:35Z","receivedAt":"2024-08-31T10:24:48Z","isPatch":false,"sender":{"key":"karnilaev@gmail.com","avatar":null},"body":"Thanks, Peff!\n\nI will try the recommendations for optimizing memory consumption for my task, that you mentioned.\n\nHave a nice day,\nYuri\n\n> On 31. Aug 2024, at 0.06, Jeff King wrote:\n> \n> On Fri, Aug 30, 2024 at 03:20:15PM +0300, Yuri Karnilaev wrote:\n> \n>> 2. Processing commits in batches:\n>> ```\n>> /usr/bin/time -l -h -p git log --ignore-missing --pretty=format:%H%x02%P%x02%aN%x02%aE%x02%at%x00 -n 1000 --skip=1000000 --numstat > 1.txt\n>> ```\n>> [...]\n>> Operating System: Mac OS 14.6.1 (23G93)\n>> Git Version: 2.39.3 (Apple Git-146)\n> \n> I sent a patch which I think should make things better for you, but I\n> wanted to mention two things in a more general way:\n> \n> 1. You should really consider building a commit-graph file with \"git\n> commit-graph write --reachable\". That will reduce the memory usage\n> for this case, but also improve the CPU quite a bit (we won't have\n> to open those million skipped commits to chase their parent\n> pointers).\n> \n> I haven't kept up with the defaults for writing graph files. I\n> thought gc.writeCommitGraph defaults to \"true\" these days, though\n> that wouldn't help in a freshly cloned repository (arguably we\n> should write the commit graph on clone?).\n> \n> 2. Using \"--skip\" still has to traverse all of those intermediate\n> commits. So it's effectively quadratic in the number of commits\n> overall (you end up skipping the first 1000 over and over).\n> \n> It's been a while since I've had to \"paginate\" segments of history\n> like this, but a better solution is along the lines of:\n> \n> - use \"-n 1000\" to get 1000 commits in each chunk\n> \n> - use \"--boundary\" to report the commits that were queued to be\n> \t traversed next but weren't shown\n> \n> - in invocations after the first one, start the traversal at\n> \t those boundary commits, rather than HEAD\n> \n> You'll probably need to add \"%m\" to your format to show the\n> boundaries (or alternatively, you can do the commit selection with\n> rev-list, and then output the result to \"log --no-walk --stdin\" to\n> do the pretty-printing).\n> \n> -Peff\n\n"},{"id":"501979","messageId":"ZtW4vD3j87oVkWKC@tanuki","threadId":"62024","inReplyTo":"20240830210607.GB1038751@coredump.intra.peff.net","subject":"Re: Unpredictable peak memory usage when using `git log` command","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-02T13:08:12Z","receivedAt":"2024-09-02T13:08:18Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Aug 30, 2024 at 05:06:07PM -0400, Jeff King wrote:\n> On Fri, Aug 30, 2024 at 03:20:15PM +0300, Yuri Karnilaev wrote:\n> \n> > 2. Processing commits in batches:\n> > ```\n> > /usr/bin/time -l -h -p git log --ignore-missing --pretty=format:%H%x02%P%x02%aN%x02%aE%x02%at%x00 -n 1000 --skip=1000000 --numstat > 1.txt\n> > ```\n> > [...]\n> > Operating System: Mac OS 14.6.1 (23G93)\n> > Git Version: 2.39.3 (Apple Git-146)\n> \n> I sent a patch which I think should make things better for you, but I\n> wanted to mention two things in a more general way:\n> \n> 1. You should really consider building a commit-graph file with \"git\n> commit-graph write --reachable\". That will reduce the memory usage\n> for this case, but also improve the CPU quite a bit (we won't have\n> to open those million skipped commits to chase their parent\n> pointers).\n> \n> I haven't kept up with the defaults for writing graph files. I\n> thought gc.writeCommitGraph defaults to \"true\" these days, though\n> that wouldn't help in a freshly cloned repository (arguably we\n> should write the commit graph on clone?).\n\nIt does default to true indeed. There is also an option to write commit\ngraph on fetch via \"fetch.writeCommitGraph\", but that setting is set to\nfalse by default. To the best of my knowledge there is no option to\ngenerate on a clone, but I agree that it would be sensible to have such\na thing.\n\nPatrick\n"}]}