{"thread":{"id":"1833","subject":"Improve git-rev-list memory usage further","startedAt":"2005-09-16T21:32:48Z","lastAt":"2005-09-16T21:42:55Z","messageCount":2,"participants":["Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"8729","messageId":"Pine.LNX.4.58.0509161423400.26803@g5.osdl.org","threadId":"1833","inReplyTo":null,"subject":"Improve git-rev-list memory usage further","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-16T21:32:48Z","receivedAt":"2005-09-16T21:32:48Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nThis avoids keeping tree entries around, and free's them as it traverses\nthe list. This avoids building up a huge memory footprint just for these \nsmall but very common allocations.\n\nBefore:\n\n\t[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l\n\t11.65user 0.38system 0:12.65elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k\n\t0inputs+0outputs (0major+42934minor)pagefaults 0swaps\n\t59124\n\nAfter:\n\n\t[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l\n\t12.28user 0.29system 0:12.57elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n\t0inputs+0outputs (0major+26718minor)pagefaults 0swaps\n\t59124\n\nnote how the minor fault numbers - which ends up being how many pages we \nneeded to map - go down from 42934 (167 MB) to 26718 (104 MB).\n\nThat's still a honking big memory footprint, but it's about half of what\nit was just a day or two ago (and this is the object list for a pretty big\nupdate - almost 60,000 objects. Smaller updates need less memory).\n\nSigned-off-by: Linus Torvalds <torvalds@osdl.org>\n---\ndiff --git a/rev-list.c b/rev-list.c\n--- a/rev-list.c\n+++ b/rev-list.c\n@@ -147,11 +147,16 @@ static struct object_list **process_tree\n \t\tdie(\"bad tree object %s\", sha1_to_hex(obj->sha1));\n \tobj->flags |= SEEN;\n \tp = add_object(obj, p, name);\n-\tfor (entry = tree->entries ; entry ; entry = entry->next) {\n+\tentry = tree->entries;\n+\ttree->entries = NULL;\n+\twhile (entry) {\n+\t\tstruct tree_entry_list *next = entry->next;\n \t\tif (entry->directory)\n \t\t\tp = process_tree(entry->item.tree, p, entry->name);\n \t\telse\n \t\t\tp = process_blob(entry->item.blob, p, entry->name);\n+\t\tfree(entry);\n+\t\tentry = next;\n \t}\n \treturn p;\n }\n@@ -218,12 +223,15 @@ static void mark_tree_uninteresting(stru\n \tif (parse_tree(tree) < 0)\n \t\tdie(\"bad tree %s\", sha1_to_hex(obj->sha1));\n \tentry = tree->entries;\n+\ttree->entries = NULL;\n \twhile (entry) {\n+\t\tstruct tree_entry_list *next = entry->next;\n \t\tif (entry->directory)\n \t\t\tmark_tree_uninteresting(entry->item.tree);\n \t\telse\n \t\t\tmark_blob_uninteresting(entry->item.blob);\n-\t\tentry = entry->next;\n+\t\tfree(entry);\n+\t\tentry = next;\n \t}\n }\n \n"},{"id":"8730","messageId":"Pine.LNX.4.58.0509161434070.26803@g5.osdl.org","threadId":"1833","inReplyTo":"Pine.LNX.4.58.0509161423400.26803@g5.osdl.org","subject":"Re: Improve git-rev-list memory usage further","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-16T21:42:55Z","receivedAt":"2005-09-16T21:42:55Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 16 Sep 2005, Linus Torvalds wrote:\n>\n> Before:\n> \t42934 minor pagefaults\n>\n> After:\n> \n> \t26718 minor pagefaults\n\nIn case it wasn't clear, this is all in _addition_ to the previous fixes. \nIt used to be ~48,000 pagefaults.\n\nStill some low-hanging fruit to go after this. Object refs next, that will \nbring it down to ~18,500 pages for this load.\n\n\t\tLinus\n"}]}