{"thread":{"id":"4503","subject":"oprofile on svn import","startedAt":"2006-06-14T01:10:19Z","lastAt":"2006-06-14T19:38:09Z","messageCount":10,"participants":["Jon Smirl","Eric Wong","Martin Langhoff","Ryan Anderson","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"21758","messageId":"9e4733910606131810ya6aa585m5d2349f651b01492@mail.gmail.com","threadId":"4503","inReplyTo":null,"subject":"oprofile on svn import","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-14T01:10:19Z","receivedAt":"2006-06-14T01:10:19Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"I'm going back to cvsimport tomorrow. My svn import that had been\nrunning for five days got killed this morning when the city decided to\nmove the telephone pole that provides my electricty.\n\nSome oprofile data, this doesn't make a lot of sense to me. Why is it\nin libcypto so much?\n\n 12632739 30.6077 /lib/libcrypto.so.0.9.8a\n 11762639 28.4995 /home/good/vmlinux\n 6310191 15.2889 /lib/libc-2.4.so\n 2498812 6.0543 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so\n 2079975 5.0395 /usr/local/bin/git-update-index\n 1103116 2.6727 /usr/lib/libz.so.1.2.3\n 617395 1.4959 /usr/lib/libapr-1.so.0.2.2\n 484625 1.1742 /usr/local/bin/git-read-tree\n\nkernel breakdown\n\n2035561 16.4450 copy_page_range\n1110813 8.9741 get_page_from_freelist\n851064 6.8756 check_poison_obj\n759296 6.1342 unmap_vmas\n670659 5.4181 release_pages\n667657 5.3939 page_remove_rmap\n595826 4.8136 page_fault\n241962 1.9548 __copy_from_user_ll\n185876 1.5017 do_wp_page\n176506 1.4260 do_page_fault\n\n\nI reset the statistics and took another snapshot half an hour later.\n\n 2232310 44.3485 /home/good/vmlinux\n 757114 15.0413 /lib/libcrypto.so.0.9.8a\n 507282 10.0780 /lib/libc-2.4.so\n 203440 4.0417 /usr/lib/libz.so.1.2.3\n 179105 3.5582 /usr/lib/libapr-1.so.0.2.2\n 169724 3.3718 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so\n 114384 2.2724 /usr/local/bin/git-update-index\n 102350 2.0334 /usr/lib/libsvn_subr-1.so.0.0.0\n 74673 1.4835 /usr/lib/libaprutil-1.so.0.2.2\n 69987 1.3904 /usr/lib/libsvn_fs_fs-1.so.0.0.0\n\nKernel:\n\n543264 21.2518 copy_page_range\n243383 9.5208 check_poison_obj\n227788 8.9108 unmap_vmas\n161806 6.3296 page_remove_rmap\n153201 5.9930 release_pages\n119092 4.6587 page_fault\n100116 3.9164 get_page_from_freelist\n45014 1.7609 do_wp_page\n42130 1.6481 vm_normal_page\n34804 1.3615 poison_obj\n28231 1.1044 do_page_fault\n27403 1.0720 __handle_mm_fault\n24558 0.9607 __copy_to_user_ll\n20618 0.8066 flush_tlb_page\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21760","messageId":"20060614020108.GB12083@hand.yhbt.net","threadId":"4503","inReplyTo":"9e4733910606131810ya6aa585m5d2349f651b01492@mail.gmail.com","subject":"Re: oprofile on svn import","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2006-06-14T02:01:08Z","receivedAt":"2006-06-14T02:01:08Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jon Smirl wrote:\n> I'm going back to cvsimport tomorrow. My svn import that had been\n> running for five days got killed this morning when the city decided to\n> move the telephone pole that provides my electricty.\n> \n> Some oprofile data, this doesn't make a lot of sense to me. Why is it\n> in libcypto so much?\n\nThe sha1 calculation is done in libcrypto, afaik.\n\nAnybody want to see how my latest patches to git-svn (and using SVN perl\nlibraries) stacks up against the mozilla repo? Speedwise, I don't\nexpect git-svn to be too different than git-svnimport, but it should use\nmuch less memory (I'll probably port the hacks to git-svnimport, too).\n\nI'll see about freeing up one of my machines to test the mozilla repo.\nUnfortunately, all of my hardware is a few years old and not extremely\nfast.\n\n-- \nEric Wong\n"},{"id":"21762","messageId":"9e4733910606131932w362c6ddcx5bf36ea5591feba1@mail.gmail.com","threadId":"4503","inReplyTo":"9e4733910606131810ya6aa585m5d2349f651b01492@mail.gmail.com","subject":"Re: oprofile on svn import","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-14T02:32:53Z","receivedAt":"2006-06-14T02:32:53Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":">From the previous data it is obvious that I had slab debugging\nenabled. I usally never notice having it turned on but in this case it\nmake a lot of difference.\n\nNew numbers without slab debug. Could forking off the git tasks be\ncausing all of this vm load?\n\n[root@jonsmirl jonsmirl]# vmstat 10\nprocs -----------memory---------- ---swap-- -----io---- --system--\n-----cpu------\n r b swpd free buff cache si so bi bo in cs us\nsy id wa st\n 2 0 0 13504 91220 563280 0 0 299 232 244 426 23\n18 53 6 0\n 2 0 0 10900 91344 565128 0 0 169 464 481 737 26\n23 48 2 0\n 2 0 0 10804 91436 564832 0 0 196 650 478 780 25\n24 49 3 0\n 4 0 0 13516 91512 561696 0 0 166 612 474 790 26\n23 49 2 0\n 1 0 0 10928 91632 563548 0 0 124 471 464 789 24\n25 48 2 0\n 1 0 0 12312 91684 562000 0 0 179 688 472 783 26\n23 48 3 0\n 1 0 0 13232 91748 560712 0 0 51 198 445 794 25\n26 48 1 0\n\n 9951967 44.5102 /home/good/vmlinux\n 3192131 14.2768 /lib/libcrypto.so.0.9.8a\n 2207857 9.8747 /lib/libc-2.4.so\n 1587518 7.1002 /usr/lib/libz.so.1.2.3\n 663114 2.9658 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so\n 517463 2.3144 /lib/ld-2.4.so\n 435100 1.9460 /usr/lib/libapr-1.so.0.2.2\n 430292 1.9245 /usr/local/bin/git-update-index\n 285157 1.2754 /usr/local/bin/git-read-tree\n\n2331728 22.8834 copy_page_range\n1076769 10.5673 unmap_vmas\n667975 6.5555 page_remove_rmap\n663844 6.5149 page_fault\n654668 6.4249 release_pages\n440547 4.3235 get_page_from_freelist\n245142 2.4058 do_wp_page\n174656 1.7141 vm_normal_page\n155185 1.5230 __handle_mm_fault\n133584 1.3110 do_page_fault\n131456 1.2901 __d_lookup\n94194 0.9244 __link_path_walk\n92927 0.9120 flush_tlb_page\n91775 0.9007 find_get_page\n85927 0.8433 copy_process\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21763","messageId":"9e4733910606131939h35b2278bvaa296459ea061621@mail.gmail.com","threadId":"4503","inReplyTo":"20060614020108.GB12083@hand.yhbt.net","subject":"Re: oprofile on svn import","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-14T02:39:32Z","receivedAt":"2006-06-14T02:39:32Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/13/06, Eric Wong wrote:\n> Jon Smirl wrote:\n> > I'm going back to cvsimport tomorrow. My svn import that had been\n> > running for five days got killed this morning when the city decided to\n> > move the telephone pole that provides my electricty.\n> >\n> > Some oprofile data, this doesn't make a lot of sense to me. Why is it\n> > in libcypto so much?\n>\n> The sha1 calculation is done in libcrypto, afaik.\n\nThat make sense, but it's eating up 14% of my CPU in a long sample.\n\n> Anybody want to see how my latest patches to git-svn (and using SVN perl\n> libraries) stacks up against the mozilla repo? Speedwise, I don't\n> expect git-svn to be too different than git-svnimport, but it should use\n> much less memory (I'll probably port the hacks to git-svnimport, too).\n\nCan svnimport be rewritten to avoid calling fork? If I am reading the\noprofiles correctly that fork is very expensive especially when the\nsvnimport task grows to 600MB.\n\nI have an import running but post your code when it is ready and I can\ntry it on the next run. They always seem to fail so there will\nprobably be another run.\n\n> I'll see about freeing up one of my machines to test the mozilla repo.\n> Unfortunately, all of my hardware is a few years old and not extremely\n> fast.\n>\n> --\n> Eric Wong\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21764","messageId":"20060614030258.GA23077@hand.yhbt.net","threadId":"4503","inReplyTo":"9e4733910606131939h35b2278bvaa296459ea061621@mail.gmail.com","subject":"Re: oprofile on svn import","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2006-06-14T03:02:58Z","receivedAt":"2006-06-14T03:02:58Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Linus: I hope I'm right on [1] (the stuff about fork).\n\nJon Smirl wrote:\n> On 6/13/06, Eric Wong wrote:\n> >Jon Smirl wrote:\n> >> I'm going back to cvsimport tomorrow. My svn import that had been\n> >> running for five days got killed this morning when the city decided to\n> >> move the telephone pole that provides my electricty.\n> >>\n> >> Some oprofile data, this doesn't make a lot of sense to me. Why is it\n> >> in libcypto so much?\n> >\n> >The sha1 calculation is done in libcrypto, afaik.\n> \n> That make sense, but it's eating up 14% of my CPU in a long sample.\n> \n> >Anybody want to see how my latest patches to git-svn (and using SVN perl\n> >libraries) stacks up against the mozilla repo? Speedwise, I don't\n> >expect git-svn to be too different than git-svnimport, but it should use\n> >much less memory (I'll probably port the hacks to git-svnimport, too).\n> \n> Can svnimport be rewritten to avoid calling fork? If I am reading the\n> oprofiles correctly that fork is very expensive especially when the\n> svnimport task grows to 600MB.\n\nI think the problem is the process growing to 600MB, and not the fork :)\ngit-svn avoids process growth pretty well from my tests with the gcc\nrepo.\n\nSee the fetch_lib() function in this patch on how I avoid process\ngrowth by _using_ fork():\n\nSubject: [PATCH 12/13] git-svn: add support for Perl SVN::* libraries\n\t(<115022175180-git-send-email-normalperson@yhbt.net>)\n\nPerl processes (at least on my machines (5.8.x, Linux x86) don't like to\nrelease memory back to the OS when they're done using it (although it\ncan reuse the memory within the process itself). This is why SVN::Pool\nisn't very effective in many cases.\n\nfork() will only duplicate memory for the pages that are changed by the\nchild, not the entire process[1]. So I fork children that run temporarily\nto avoid accumulating memory usage inside the process.\n\nThis technique should probably be added to git-svnimport as well.\n\n> I have an import running but post your code when it is ready and I can\n> try it on the next run. They always seem to fail so there will\n> probably be another run.\n\nI've posted a two series of patches the past few days that have yet\nto be merged by Junio:\n\nSubject: [PATCH] git-svn: bug fixes (some resends)\n\t<11500094252972-git-send-email-normalperson@yhbt.net>\nSubject: [PATCH 0/13] git-svn: better branch support, SVN:: lib usage, feature additions\n\t<11502217352245-git-send-email-normalperson@yhbt.net>\n\n-- \nEric Wong\n"},{"id":"21765","messageId":"46a038f90606132032o7ebfb688u81500d78f4240162@mail.gmail.com","threadId":"4503","inReplyTo":"9e4733910606131810ya6aa585m5d2349f651b01492@mail.gmail.com","subject":"Re: oprofile on svn import","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-06-14T03:32:58Z","receivedAt":"2006-06-14T03:32:58Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 6/14/06, Jon Smirl wrote:\n> I'm going back to cvsimport tomorrow. My svn import that had been\n\nFor best results, make sure you remove the -a from the git-repack\nline. Once it's done, run git-repack -a -d manually.\n\ncheers,\n\n\nmartin\n"},{"id":"21766","messageId":"20060614044802.GE30825@h4x0r5.com","threadId":"4503","inReplyTo":"20060614020108.GB12083@hand.yhbt.net","subject":"Re: oprofile on svn import","fromName":"Ryan Anderson","fromEmail":"ryan@michonline.com","sentAt":"2006-06-14T04:48:02Z","receivedAt":"2006-06-14T04:48:02Z","isPatch":false,"sender":{"key":"ryan@michonline.com","avatar":null},"body":"On Tue, Jun 13, 2006 at 07:01:08PM -0700, Eric Wong wrote:\n> Anybody want to see how my latest patches to git-svn (and using SVN perl\n> libraries) stacks up against the mozilla repo? Speedwise, I don't\n> expect git-svn to be too different than git-svnimport, but it should use\n> much less memory (I'll probably port the hacks to git-svnimport, too).\n\nI've got access to a pretty good machine to run this on - where can I\ngrab the svn repo from?\n(I can just grab the CVS one and convert it, first, as well, just point\nme at that, if that's got more bandwidth.)\n\n-- \n\nRyan Anderson\n sometimes Pug Majere\n"},{"id":"21767","messageId":"9e4733910606132226u720c74b6tb614df2702a819c5@mail.gmail.com","threadId":"4503","inReplyTo":"20060614044802.GE30825@h4x0r5.com","subject":"Re: oprofile on svn import","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-14T05:26:57Z","receivedAt":"2006-06-14T05:26:57Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/14/06, Ryan Anderson wrote:\n> On Tue, Jun 13, 2006 at 07:01:08PM -0700, Eric Wong wrote:\n> > Anybody want to see how my latest patches to git-svn (and using SVN perl\n> > libraries) stacks up against the mozilla repo? Speedwise, I don't\n> > expect git-svn to be too different than git-svnimport, but it should use\n> > much less memory (I'll probably port the hacks to git-svnimport, too).\n>\n> I've got access to a pretty good machine to run this on - where can I\n> grab the svn repo from?\n> (I can just grab the CVS one and convert it, first, as well, just point\n> me at that, if that's got more bandwidth.)\n\nrsync -az cvs-mirror.mozilla.org::mozilla ~/mozilla/cvs-mirror\nIt took about three days for my machine to convert that cvs to svn.\n\nI have the converted repo local but it is 8.2GB and I have 256kb up.\n\nThere is no real purpose in converting mozilla cvs to svn to git other\nthan to test the tools. My last attempt at svn to git ran five days\nbefore I lost power. Towards the end it was getting significantly slow\nimplying some kind of n squared problem in the import process. The\nidea was to see if cvsimport and svnimport both end up with the same\noutput.\n\nI am going to use git-cvsimport on the mozilla repo but that tool\nneeds to 2GB+ physical RAM to run. I ordered 2GB more and it will be\nhere tomorrow. I have just been playing with the svn conversion while\nI wait five days for my 2nd day air package to show up.\n\n\n>\n> --\n>\n> Ryan Anderson\n> sometimes Pug Majere\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21794","messageId":"9e4733910606141225n11b406fte6229ea9993825dd@mail.gmail.com","threadId":"4503","inReplyTo":"9e4733910606131932w362c6ddcx5bf36ea5591feba1@mail.gmail.com","subject":"Re: oprofile on svn import","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-14T19:25:46Z","receivedAt":"2006-06-14T19:25:46Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"Stats after 18 hours into git-svnimport. Process is now stuck in the\nkernel 64% of the time. All of the kernel time is in page management.\nPerl svnimport process is 290MB now.\n\nMy top candidates for causing the problem are the fork in the perl\ncode or the execing of a million tiny git processes.\n\nThe key low level git functions could be made into a library to avoid\nthe need to exec them continuously. The svn functions are libraries\nand they hardly show up.\n\n 606218 2.4143 /usr/local/bin/git-update-index\n 127170 0.5065 /usr/local/bin/git-write-tree\n 81153 0.3232 /usr/local/bin/git-read-tree\n 13065 0.0520 /usr/local/bin/git-ls-files\n 2624 0.0105 /usr/local/bin/git-hash-object\n 754 0.0030 /usr/local/bin/git-commit-tree\n 462 0.0018 /usr/local/bin/git-ls-tree\n 398 0.0016 /usr/local/bin/git-rev-parse\n\nversus\n\n 102784 0.3641 /usr/lib/libsvn_subr-1.so.0.0.0\n 70235 0.2488 /usr/lib/libsvn_fs_fs-1.so.0.0.0\n 67081 0.2376 /usr/lib/libsvn_delta-1.so.0.0.0\n 848 0.0030 /usr/lib/libsvn_swig_perl-1.so.0.0.0\n 512 0.0018 /usr/lib/libsvn_ra_local-1.so.0.0.0\n 350 0.0012 /usr/lib/libsvn_fs-1.so.0.0.0\n 222 7.9e-04 /usr/lib/libsvn_repos-1.so.0.0.0\n 124 4.4e-04 /usr/lib/libsvn_ra-1.so.0.0.0\n\n------------------------------------------------------------------------------------------------------------\n\n 4093890 64.3711 /home/good/vmlinux\n 906014 14.2459 /lib/libcrypto.so.0.9.8a\n 435744 6.8515 /lib/libc-2.4.so\n 158325 2.4895 /usr/lib/libz.so.1.2.3\n 139995 2.2012 /usr/local/bin/git-update-index\n 75322 1.1843 /nvidia\n 64349 1.0118 /usr/bin/oprofiled\n 52825 0.8306 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so\n 51930 0.8165 /usr/lib/libapr-1.so.0.2.2\n 42771 0.6725 /usr/local/bin/git-read-tree\n 37774 0.5939 /lib/ld-2.4.so\n 34761 0.5466 /usr/local/bin/git-write-tree\n 29560 0.4648 /usr/lib/libsvn_subr-1.so.0.0.0\n 28210 0.4436 /usr/lib/libaprutil-1.so.0.2.2\n\n-----------------------------------------------------------------------------------------------------------------\n\n2471826 32.8741 copy_page_range\n375260 18.2903 unmap_vmas\n574208 7.6367 release_pages\n572189 7.6098 page_remove_rmap\n233367 3.1037 free_pages_and_swap_cache\n191051 2.5409 get_page_from_freelist\n169058 2.2484 unlock_page\n162027 2.1549 vm_normal_page\n155691 2.0706 swap_info_get\n136324 1.8130 swap_duplicate\n119227 1.5857 page_fault\n99729 1.3263 page_waitqueue\n49288 0.6555 remove_exclusive_swap_page\n39611 0.5268 do_wp_page\n39142 0.5206 __wake_up_bit\n34384 0.4573 __copy_from_user_ll\n31111 0.4138 __handle_mm_fault\n29990 0.3989 find_get_page\n29682 0.3948 do_page_fault\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21796","messageId":"e6poil$e8l$1@sea.gmane.org","threadId":"4503","inReplyTo":"9e4733910606141225n11b406fte6229ea9993825dd@mail.gmail.com","subject":"Re: oprofile on svn import","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-06-14T19:38:09Z","receivedAt":"2006-06-14T19:38:09Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jon Smirl wrote:\n\n> Stats after 18 hours into git-svnimport. Process is now stuck in the\n> kernel 64% of the time. All of the kernel time is in page management.\n> Perl svnimport process is 290MB now.\n> \n> My top candidates for causing the problem are the fork in the perl\n> code or the execing of a million tiny git processes.\n> \n> The key low level git functions could be made into a library to avoid\n> the need to exec them continuously. The svn functions are libraries\n> and they hardly show up.\n\nThere is ongoing effort to translate git functions into builtins.\nStill you would need to translate git-svnimport Perl code into C,\nor somehow access git library from Perl.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"}]}