{"thread":{"id":"47839","subject":"Regression in memory consumption of git fsck","startedAt":"2018-02-14T10:23:59Z","lastAt":"2018-02-14T13:55:59Z","messageCount":3,"participants":["Dominic Sacré","SZEDER Gábor","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"339268","messageId":"b1a4d7ed-79b6-1f6b-1f0f-85536e943cef@gmx.de","threadId":"47839","inReplyTo":null,"subject":"Regression in memory consumption of git fsck","fromName":"Dominic Sacré","fromEmail":"dominic.sacre@gmx.de","sentAt":"2018-02-14T10:22:43Z","receivedAt":"2018-02-14T10:23:59Z","isPatch":false,"sender":{"key":"dominic.sacre@gmx.de","avatar":null},"body":"Hi,\n\nI've noticed that recent versions of git consume a lot of memory during\n\"git fsck\", to the point where I've regularly had git fall victim to\nLinux's OOM killer.\n\nFor example, if I clone torvalds/linux.git, and then run \"git fsck\n--connectivity-only\" in the newly cloned repository, git will consume\nmore than 6GB of physical memory, while older versions peak at about 2GB.\n\nI've managed to bisect this down to this commit in v2.14:\n\nad2db4030e42890e569de529e3cd61a8d03de497\nfsck: remove redundant parse_tree() invocation\n\nIf I revert that commit (on top of current master) the memory\nconsumption goes down to 2GB again. The change looks relatively harmless\nto me, so does anyone know what's going on here?\n\n\nThanks,\n\nDominic\n"},{"id":"339273","messageId":"20180214124828.27845-1-szeder.dev@gmail.com","threadId":"47839","inReplyTo":"b1a4d7ed-79b6-1f6b-1f0f-85536e943cef@gmx.de","subject":"Re: Regression in memory consumption of git fsck","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-14T12:48:28Z","receivedAt":"2018-02-14T12:48:48Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> I've noticed that recent versions of git consume a lot of memory during\n> \"git fsck\", to the point where I've regularly had git fall victim to\n> Linux's OOM killer.\n> \n> For example, if I clone torvalds/linux.git, and then run \"git fsck\n> --connectivity-only\" in the newly cloned repository, git will consume\n> more than 6GB of physical memory, while older versions peak at about 2GB.\n> \n> I've managed to bisect this down to this commit in v2.14:\n> \n> ad2db4030e42890e569de529e3cd61a8d03de497\n> fsck: remove redundant parse_tree() invocation\n> \n> If I revert that commit (on top of current master) the memory\n> consumption goes down to 2GB again. The change looks relatively harmless\n> to me, so does anyone know what's going on here?\n\nI could reproduce the increased memory usage even for much smaller\nrepositories.  The patch below seems to fix it for me.\n\n\n -- >8 --\n\nSubject: [PATCH] fsck: plug tree buffer leak\n\nCommit ad2db4030e (fsck: remove redundant parse_tree() invocation,\n2017-07-18), along with that redundant call to parse_tree() in\ntraverse_one_object(), also removed a call to free_tree_buffer() from\nthat function.  This resulted in significantly increased memory usage\nof 'git fsck' because of all the non-freed tree buffers; in case of\ngit.git and '--connectivity-only' it went from around 270MB to over\n1.2GB.\n\nRestore that free_tree_buffer() call to bring down memory usage to the\nprevious level.\n\nReported-by: Dominic Sacré <dominic.sacre@gmx.de>\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n\n builtin/fsck.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 7a8a679d4f..8bc1b59daf 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -180,7 +180,10 @@ static void mark_object_reachable(struct object *obj)\n \n static int traverse_one_object(struct object *obj)\n {\n-\treturn fsck_walk(obj, obj, &fsck_walk_options);\n+\tint result = fsck_walk(obj, obj, &fsck_walk_options);\n+\tif (obj->type == OBJ_TREE)\n+\t\tfree_tree_buffer((struct tree *)obj);\n+\treturn result;\n }\n \n static int traverse_reachable(void)\n-- \n2.16.1.347.gd41f2872c6\n\n"},{"id":"339276","messageId":"20180214135551.GA27513@sigill.intra.peff.net","threadId":"47839","inReplyTo":"20180214124828.27845-1-szeder.dev@gmail.com","subject":"Re: Regression in memory consumption of git fsck","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-14T13:55:52Z","receivedAt":"2018-02-14T13:55:59Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 14, 2018 at 01:48:28PM +0100, SZEDER Gábor wrote:\n\n> > If I revert that commit (on top of current master) the memory\n> > consumption goes down to 2GB again. The change looks relatively harmless\n> > to me, so does anyone know what's going on here?\n> \n> I could reproduce the increased memory usage even for much smaller\n> repositories.  The patch below seems to fix it for me.\n> \n> \n>  -- >8 --\n> \n> Subject: [PATCH] fsck: plug tree buffer leak\n\nI think this is fixed already in ba3a08ca0e (fsck: fix leak when\ntraversing trees, 2018-01-20), which is in 'next' (and marked for \"will\nmerge to master in yesterday's \"what's cooking\").\n\n-Peff\n"}]}