Re: git svn's performance issue and strange pauses, and other thing

------------------------------ On Tue, Oct 7, 2014 00:51 BST Hin-Tak Leung wrote:

<snipped>

... Then

The changes definitely improve, as far as my impression goes. There was only one notable pause around r50651, and it is probably because the rather large "Checking svn:mergeinfo changes since r15413" from r15413? That took about 12 minutes. Other instances of "W:svn cherry-pick ignored" though do take a while, are in the seconds region - before the code changes they could be minutes, if memory serves.

<--
	M	src/library/tools/R/toHTML.R
r50650 = bed91d435c535f2643cf0d48623fecf86d264bd9 (refs/remotes/trunk)
	M	src/modules/X11/rotated.c
	M	src/modules/X11/dataentry.c
Checking svn:mergeinfo changes since r15413: 1 sources, 1 changed
W:svn cherry-pick ignored (/trunk:28840) - missing 9372 commit(s) (eg cea6142c76300539a0d0c9c743738e31a9f7d523)
r50651 = ad139a5bf91f9ad6690ff5fb4a3f71cea591a944 (refs/remotes/R-uthreads)
-->

The new clone has:

<-- $ ls -ltr .git/svn/.caches/ total 144788 -rw-rw-r--. 1 Hin-Tak Hin-Tak 1166138 Oct 7 13:44 lookup_svn_merge.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 72849741 Oct 7 13:48 check_cherry_pick.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 1133855 Oct 7 13:49 has_no_changes.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 73109005 Oct 7 13:53 _rev_list.yaml -->

The old clone has:

<--- $ ls -ltr .git/svn/.caches/ total 318824 -rw-rw-r--. 1 Hin-Tak Hin-Tak 5711724 Jul 24 2012 lookup_svn_merge.db -rw-rw-r--. 1 Hin-Tak Hin-Tak 30523628 Jul 24 2012 check_cherry_pick.db -rw-rw-r--. 1 Hin-Tak Hin-Tak 296592 Jul 24 2012 has_no_changes.db -rw-rw-r--. 1 Hin-Tak Hin-Tak 40241189 Oct 5 16:42 lookup_svn_merge.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 225323456 Oct 5 16:49 check_cherry_pick.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 242547 Oct 5 16:49 has_no_changes.yaml -rw-rw-r--. 1 Hin-Tak Hin-Tak 24120007 Oct 5 16:50 _rev_list.yaml -->

I had to suspend somewhat around r59000 - but it is interesting to see that the max memory consumption of the later part is almost double? and it also runs at 100% rather than 60% overall; I don't know what to make of that - probably just smaller changes versus larger ones, or different time of day and network loads (yes, I guess it is just bandwidth-limited?, since the bulk of CPU time is in system rather than user).

I am somwhat worry about the dramatic difference between the two .svn/.caches - check_cherry_pick.yaml is 225MB in one and 73MB in the other, and also _rev_list.yaml is opposite - 24MB vs 73MB. How do I reconcile that?

<--
	M	src/main/dotcode.c
	M	doc/NEWS.Rd
r59140 = b6014a226aebf9e016c89c0bd1aca1979796a057 (refs/remotes/trunk)
	M	src/main/dotcode.c
	M	doc/NEWS.Rd
Checking svn:mergeinfo changes since r59138: 4 sources, 1 changed
W:svn cherry-pick ignored (/trunk:59137,59140) - missing 369 commit(s) (eg 8a2a36083ba39be27fc9940acc3f51eab6a7a0c3)
r59141 = 38c6d05f164d34e4b5cc545bda387be9d910f748 (refs/remotes/R-2-15-branch)
Connection timed out: Connection timed out at /usr/share/perl5/vendor_perl/Git/SVN/Ra.pm line 290.

Command exited with non-zero status 1
	Command being timed: "git svn fetch --all"
	User time (seconds): 5642.19
	System time (seconds): 23552.44
	Percent of CPU this job got: 57%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 14:06:58
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 349324
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 39
	Minor (reclaiming a frame) page faults: 744713614
	Voluntary context switches: 4761489
	Involuntary context switches: 8595950
	Swaps: 0
	File system inputs: 7712
	File system outputs: 121404296
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 1
-->
<--
	M	src/include/Defn.h
r66719 = 1e3288d3ae4cfb15f6e4e4116f18d38b3efc5bb5 (refs/remotes/trunk)
	M	doc/NEWS.Rd
r66720 = 1c184e5fc2b71a27767215a45a1270f3edbc616f (refs/remotes/trunk)
Checked out HEAD:
  https://svn.r-project.org/R/trunk r66720
creating empty directory: tests/Pkgs/exNSS4/man
	Command being timed: "git svn fetch --all"
	User time (seconds): 2126.00
	System time (seconds): 7852.44
	Percent of CPU this job got: 96%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 2:52:38
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 755256
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 6
	Minor (reclaiming a frame) page faults: 142730534
	Voluntary context switches: 898725
	Involuntary context switches: 1842056
	Swaps: 0
	File system inputs: 1800
	File system outputs: 28606392
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
-->

Hin-Tak Leung <htl10@users.sourceforge.net> wrote:

<snip>

git-svn memory usage is insane, and we need to reduce it.
(on Linux, fork() performance is reduced as memory size of the parent
 grows, and I don't think we can easily call vfork() from Perl)

Calling patterns changed, and it looks like Jakob's changes avoided some
calls.  The main thing to care about:
	Does the repository history look right?

The check_cherry_pick cache can be made smaller, too:
----------------------- 8< -----------------------------
From: Eric Wong <normalperson@yhbt.net>
Subject: [PATCH] git-svn: reduce check_cherry_pick cache overhead

We do not need to store entire lists of commits, only the number of incomplete and the first commit for reference. This reduces the amount of data we need to store in memory and on disk stores.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 perl/Git/SVN.pm | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

On Oct 18, 2014, at 21:12, Eric Wong <normalperson@yhbt.net> wrote:

It is possible that those functions don't need to be memoized any more. My patch is trying to avoid calling them with the same arguments over and over, and memoizing doesn't help when arguments are changing.

Thanks, /jakob