{"thread":{"id":"1834","subject":"Avoid building object ref lists when not needed","startedAt":"2005-09-16T21:55:33Z","lastAt":"2005-09-16T23:00:12Z","messageCount":3,"participants":["Linus Torvalds","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"8731","messageId":"Pine.LNX.4.58.0509161446100.26803@g5.osdl.org","threadId":"1834","inReplyTo":null,"subject":"Avoid building object ref lists when not needed","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-16T21:55:33Z","receivedAt":"2005-09-16T21:55:33Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nThe object parsing code builds a generic \"this object references that \nobject\" because doing a full connectivity check for fsck requires it.\n\nHowever, nothing else really needs it, and it's quite expensive for \ngit-rev-list that can have tons of objects in flight.\n\nSo, exactly like the commit buffer save thing, add a global flag to \ndisable it, and use it in git-rev-list.\n\nBefore:\n\n\t[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l\n\t12.28user 0.29system 0:12.57elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n\t0inputs+0outputs (0major+26718minor)pagefaults 0swaps\n\t59124\n\nAfter this change:\n\n\t[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l\n\t10.33user 0.18system 0:10.54elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n\t0inputs+0outputs (0major+18509minor)pagefaults 0swaps\n\t59124\n\nand note how the number of pages touched by git-rev-list for this\nparticular object list has shrunk from 26,718 (104 MB) to 18,509 (72 MB).\n\nCalculating the total object difference between two git revisions is still\nclearly the most expensive git operation (both in memory and CPU time),\nbut it's now less than 40% of what it used to be.\n\nSigned-off-by: Linus Torvalds <torvalds@osdl.org>\n----\ndiff --git a/object.c b/object.c\n--- a/object.c\n+++ b/object.c\n@@ -9,6 +9,8 @@ struct object **objs;\n int nr_objs;\n static int obj_allocs;\n \n+int track_object_refs = 1;\n+\n static int find_object(const unsigned char *sha1)\n {\n \tint first = 0, last = nr_objs;\n@@ -67,9 +69,12 @@ void created_object(const unsigned char \n \n void add_ref(struct object *refer, struct object *target)\n {\n-\tstruct object_list **pp = &refer->refs;\n-\tstruct object_list *p;\n-\t\n+\tstruct object_list **pp, *p;\n+\n+\tif (!track_object_refs)\n+\t\treturn;\n+\n+\tpp = &refer->refs;\n \twhile ((p = *pp) != NULL) {\n \t\tif (p->item == target)\n \t\t\treturn;\n@@ -87,6 +92,8 @@ void mark_reachable(struct object *obj, \n {\n \tstruct object_list *p = obj->refs;\n \n+\tif (!track_object_refs)\n+\t\tdie(\"cannot do reachability with object refs turned off\");\n \t/* If we've been here already, don't bother */\n \tif (obj->flags & mask)\n \t\treturn;\ndiff --git a/object.h b/object.h\n--- a/object.h\n+++ b/object.h\n@@ -17,6 +17,7 @@ struct object {\n \tvoid *util;\n };\n \n+extern int track_object_refs;\n extern int nr_objs;\n extern struct object **objs;\n \ndiff --git a/rev-list.c b/rev-list.c\n--- a/rev-list.c\n+++ b/rev-list.c\n@@ -581,6 +581,7 @@ int main(int argc, char **argv)\n \t}\n \n \tsave_commit_buffer = verbose_header;\n+\ttrack_object_refs = 0;\n \n \tif (!merge_order) {\t\t\n \t\tsort_by_date(&list);\n"},{"id":"8733","messageId":"Pine.LNX.4.58.0509161505410.26803@g5.osdl.org","threadId":"1834","inReplyTo":"Pine.LNX.4.58.0509161446100.26803@g5.osdl.org","subject":"Re: Avoid building object ref lists when not needed","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-16T22:17:31Z","receivedAt":"2005-09-16T22:17:31Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 16 Sep 2005, Linus Torvalds wrote:\n> \n> and note how the number of pages touched by git-rev-list for this\n> particular object list has shrunk from 26,718 (104 MB) to 18,509 (72 MB).\n\nOne final note: since I'm only looking at the total page footprint for the \nprocess, this _includes_ all the mmap faults of not just the executable, \nbut since my kernel is usign a packed archive, also the actual mapping of \nthe pack itself.\n\nThe pack itself is 76MB in size, but obviously not all of it actually\nneeds to be read in. For example, it doesn't need to actually read any of\nthe objects themselves (that's done by the actual packing phase), so we're\nonly touching the \"commit\" and \"tree\" parts of the pack.\n\nOf the 72MB, valgrind reports that \"only\" about 24MB is now used for\nactual malloc'ed memory. So while the _total_ \"touched pages\" have shrunk\nfrom 187 MB (when I started) to 72 MB (now), the malloc()'ed memory has\nactually shrunk a lot more (all the savings have been by either avoiding \nallocations altogether, or by freeing them as soon as we can).\n\nI'm done for the while. There are small wins still to be had, but they are \nsmaller and harder to reach. Maybe I'll get a spurt of energy one of these \ndays, so if somebody finds a particularly problematic load, tell me.\n\n\t\tLinus\n"},{"id":"8734","messageId":"7vek7ohb43.fsf@assigned-by-dhcp.cox.net","threadId":"1834","inReplyTo":"Pine.LNX.4.58.0509161446100.26803@g5.osdl.org","subject":"Re: Avoid building object ref lists when not needed","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-16T23:00:12Z","receivedAt":"2005-09-16T23:00:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks.  I will push this out along with your other patches when\nmister.kernel.org is back online after relocation.\n"}]}