Re: git svn's performance issue and strange pauses, and other thing
- From
- Hin-Tak Leung <htl10@users.sourceforge.net>
- Date
- Oct 7, 2014, 18:20 UTC
- Message-ID
- <1412706046.90413.YahooMailBasic@web172303.mail.ir2.yahoo.com>
------------------------------ On Tue, Oct 7, 2014 00:51 BST Hin-Tak Leung wrote:
>------------------------------ >On Sun, Oct 5, 2014 02:02 BST Eric Wong wrote:
<snipped>
>>Hin-Tak: have you tried Jakob's patches? I've taken another look, >>signed-off and pushed to my master.
... Then
Show 21 quoted lines
>I changed my mind, and decided what the hell, let's clone the whole >thing again :-). So I made a new directory, run 'git init', just copy >.git/config from the old reop and am doing 'git svn fetch --all' in the new empty >directory again. > >So far it seems to be good. But I am only at revision 35700-ish at the moment, >and the whole thing is 66700-ish. Oh, I forgot to mention that the strange >pauses seem to be followed by messages like these: > >W:svn cherry-pick ignored (/branches/R-2-12-branch:52939,54476,55265) - missing 492 commit(s) (eg 9bf20dca6a8b05dff28e6486b1613f10825972c9) >W:svn cherry-pick ignored (/branches/R-2-13-branch:55265,55432) - missing 231 commit(s) (eg 9290cf6ce2d7f6cca168cf326eed6e9fe760895f) >W:svn cherry-pick ignored (/branches/R-2-15-branch:58894,59717) - missing 405 commit(s) (eg ed84a373b33f728949edf3371829fc3414c343a8) >W:svn cherry-pick ignored (/branches/R-3-0-branch:62497) - missing 154 commit(s) (eg 9e4742d201771c9658417c2d2f83838e550e3162) >W:svn cherry-pick ignored (/trunk: > >So presumably I'd only see interesting behavior when there are a number of branches. >It seems the first branches are around revision 48000-ish, so I might have >to wait a bit. > >So far, the new clone hasn't created ".git/svn/.caches/" yet; and memory consumption seems >okay also.
The changes definitely improve, as far as my impression goes. There was only one notable pause around r50651, and it is probably because the rather large "Checking svn:mergeinfo changes since r15413" from r15413? That took about 12 minutes. Other instances of "W:svn cherry-pick ignored" though do take a while, are in the seconds region - before the code changes they could be minutes, if memory serves.
<-- M src/library/tools/R/toHTML.R r50650 = bed91d435c535f2643cf0d48623fecf86d264bd9 (refs/remotes/trunk) M src/modules/X11/rotated.c M src/modules/X11/dataentry.c Checking svn:mergeinfo changes since r15413: 1 sources, 1 changed W:svn cherry-pick ignored (/trunk:28840) - missing 9372 commit(s) (eg cea6142c76300539a0d0c9c743738e31a9f7d523) r50651 = ad139a5bf91f9ad6690ff5fb4a3f71cea591a944 (refs/remotes/R-uthreads) -->
The new clone has:
<-- $ ls -ltr .git/svn/.caches/ total 144788 -rw-rw-r--. 1 Hin-Tak Hin-Tak 1166138 Oct 7 13:44 lookup_svn_merge.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 72849741 Oct 7 13:48 check_cherry_pick.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 1133855 Oct 7 13:49 has_no_changes.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 73109005 Oct 7 13:53 _rev_list.yaml -->
The old clone has:
<--- $ ls -ltr .git/svn/.caches/ total 318824 -rw-rw-r--. 1 Hin-Tak Hin-Tak 5711724 Jul 24 2012 lookup_svn_merge.db -rw-rw-r--. 1 Hin-Tak Hin-Tak 30523628 Jul 24 2012 check_cherry_pick.db -rw-rw-r--. 1 Hin-Tak Hin-Tak 296592 Jul 24 2012 has_no_changes.db -rw-rw-r--. 1 Hin-Tak Hin-Tak 40241189 Oct 5 16:42 lookup_svn_merge.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 225323456 Oct 5 16:49 check_cherry_pick.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 242547 Oct 5 16:49 has_no_changes.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 24120007 Oct 5 16:50 _rev_list.yaml -->
I had to suspend somewhat around r59000 - but it is interesting to see that the max memory consumption of the later part is almost double? and it also runs at 100% rather than 60% overall; I don't know what to make of that - probably just smaller changes versus larger ones, or different time of day and network loads (yes, I guess it is just bandwidth-limited?, since the bulk of CPU time is in system rather than user).
I am somwhat worry about the dramatic difference between the two .svn/.caches - check_cherry_pick.yaml is 225MB in one and 73MB in the other, and also _rev_list.yaml is opposite - 24MB vs 73MB. How do I reconcile that?
<-- M src/main/dotcode.c M doc/NEWS.Rd r59140 = b6014a226aebf9e016c89c0bd1aca1979796a057 (refs/remotes/trunk) M src/main/dotcode.c M doc/NEWS.Rd Checking svn:mergeinfo changes since r59138: 4 sources, 1 changed W:svn cherry-pick ignored (/trunk:59137,59140) - missing 369 commit(s) (eg 8a2a36083ba39be27fc9940acc3f51eab6a7a0c3) r59141 = 38c6d05f164d34e4b5cc545bda387be9d910f748 (refs/remotes/R-2-15-branch) Connection timed out: Connection timed out at /usr/share/perl5/vendor_perl/Git/SVN/Ra.pm line 290.
Command exited with non-zero status 1 Command being timed: "git svn fetch --all" User time (seconds): 5642.19 System time (seconds): 23552.44 Percent of CPU this job got: 57% Elapsed (wall clock) time (h:mm:ss or m:ss): 14:06:58 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 349324 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 39 Minor (reclaiming a frame) page faults: 744713614 Voluntary context switches: 4761489 Involuntary context switches: 8595950 Swaps: 0 File system inputs: 7712 File system outputs: 121404296 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 1 --> <-- M src/include/Defn.h r66719 = 1e3288d3ae4cfb15f6e4e4116f18d38b3efc5bb5 (refs/remotes/trunk) M doc/NEWS.Rd r66720 = 1c184e5fc2b71a27767215a45a1270f3edbc616f (refs/remotes/trunk) Checked out HEAD: https://svn.r-project.org/R/trunk r66720 creating empty directory: tests/Pkgs/exNSS4/man Command being timed: "git svn fetch --all" User time (seconds): 2126.00 System time (seconds): 7852.44 Percent of CPU this job got: 96% Elapsed (wall clock) time (h:mm:ss or m:ss): 2:52:38 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 755256 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 6 Minor (reclaiming a frame) page faults: 142730534 Voluntary context switches: 898725 Involuntary context switches: 1842056 Swaps: 0 File system inputs: 1800 File system outputs: 28606392 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 -->