threads / discuss / 37586

git svn's performance issue and strange pauses, and other thing

Subject: git svn's performance issue and strange pauses, and other thing

## tl;dr

4 messages between Sep 18, 2014 and Oct 5, 2014.

replies: 3people: 3as markdown or json

Hin-Tak Leung· Sep 18, 2014, 07:39 UTC · lore

(I am not on the list - please CC)

Thanks for git-svn - I use it instead of subversion itself for many years now.

Just thought I'd ask/report a few issues I noticed for some time now, of tracking development of a particular subversion-based development project. Broadly speaking, I think there are 3 problems, especially noticeable against a particular repository, but to a lesser extent with some others too.

- just doing "git svn fetch --all" seems to consume a lot of memory,
for very little actual fetched changes. (in the 2GB+ region, sometimes).

- "git svn fetch --all" also seems to take a long time too, for certain
fetched changes. (in the minutes region).

-  I know I can probably just "read the source", but I'd like to know
why .git/svn/.caches is even larger than .git/objects (which supposedly
contains everything that's of interest)? I hope this can be documented
towards the end of the man-page, for example, of important parts
of .git/svn (and what not to do with them...), without needing to
'read the source'. Here is part of "du" from a couple of days ago:

254816 .git/objects 307056 .git/svn/.caches 332452 .git/svn 588064 .git

The actual .git/config is here - this should be sufficient info for somebody looking into experiencing the issues I mentioned above.

--------
$ more .git/config 
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[svn-remote "svn"]
	url = https://svn.r-project.org/R
	fetch = trunk:refs/remotes/trunk
	branches = branches/*:refs/remotes/*
	tags = tags/*:refs/remotes/tags/*
[pack]
	threads = 1
------------

Eric Wong· Sep 19, 2014, 08:25 UTC · re: Hin-Tak Leung · lore

Re: git svn's performance issue and strange pauses, and other thing

Hin-Tak Leung <htl10@users.sourceforge.net> wrote:

> (I am not on the list - please CC)

Done, it is standard practice for git :)

Show 13 quoted lines

> Thanks for git-svn - I use it instead of subversion itself for many years now.
> 
> Just thought I'd ask/report a few issues I noticed for some time
> now, of tracking development of a particular subversion-based
> development project. Broadly speaking, I think there are 3 problems,
> especially noticeable against a particular repository, but 
> to a lesser extent with some others too.
> 
> - just doing "git svn fetch --all" seems to consume a lot of memory,
> for very little actual fetched changes. (in the 2GB+ region, sometimes).
> 
> - "git svn fetch --all" also seems to take a long time too, for certain
> fetched changes. (in the minutes region).

Jakob sent some patches a few months ago which seem to address the issue. Unfortunately we forgot about them :x

Can you take a look at the following two "mergeinfo-speedups" in my repo? (git://bogomips.org/git-svn)

Jakob Stoklund Olesen (2):
      git-svn: only look at the new parts of svn:mergeinfo
      git-svn: only look at the root path for svn:mergeinfo

Also downloadable here:

http://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535 http://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a

Can you please give them a try?

Show 14 quoted lines

> -  I know I can probably just "read the source", but I'd like to know
> why .git/svn/.caches is even larger than .git/objects (which supposedly
> contains everything that's of interest)? I hope this can be documented
> towards the end of the man-page, for example, of important parts
> of .git/svn (and what not to do with them...), without needing to
> 'read the source'. Here is part of "du" from a couple of days ago:
> 
> 254816	.git/objects
> 307056	.git/svn/.caches
> 332452	.git/svn
> 588064	.git
> 
> The actual .git/config is here - this should be sufficient info for
> somebody looking into experiencing the issues I mentioned above.

IIRC, the caching is unique to mergeinfo, so perhaps Jakob's patches help, there, too.

Sorry I don't understand the mergeinfo stuff more, I've never worked on a project which uses it.

Show 15 quoted lines

> --------
> $ more .git/config 
> [core]
> 	repositoryformatversion = 0
> 	filemode = true
> 	bare = false
> 	logallrefupdates = true
> [svn-remote "svn"]
> 	url = https://svn.r-project.org/R
> 	fetch = trunk:refs/remotes/trunk
> 	branches = branches/*:refs/remotes/*
> 	tags = tags/*:refs/remotes/tags/*
> [pack]
> 	threads = 1
> ------------

Jakob Stoklund Olesen· Sep 19, 2014, 13:44 UTC · re: Eric Wong · lore

Re: git svn's performance issue and strange pauses, and other thing

Show 21 quoted lines

> On Sep 19, 2014, at 1:25, Eric Wong <normalperson@yhbt.net> wrote:
> 
> Hin-Tak Leung <htl10@users.sourceforge.net> wrote:
> 
>> -  I know I can probably just "read the source", but I'd like to know
>> why .git/svn/.caches is even larger than .git/objects (which supposedly
>> contains everything that's of interest)? I hope this can be documented
>> towards the end of the man-page, for example, of important parts
>> of .git/svn (and what not to do with them...), without needing to
>> 'read the source'. Here is part of "du" from a couple of days ago:
>> 
>> 254816    .git/objects
>> 307056    .git/svn/.caches
>> 332452    .git/svn
>> 588064    .git
>> 
>> The actual .git/config is here - this should be sufficient info for
>> somebody looking into experiencing the issues I mentioned above.
> 
> IIRC, the caching is unique to mergeinfo, so perhaps Jakob's patches
> help, there, too.

IIRC the caches are used for memoization, and with my two patches applied it doesn't improve performance much.

You could try removing the memoization after applying my patches.

Thanks, /Jakob

Eric Wong· Oct 5, 2014, 01:02 UTC · re: Eric Wong · lore

Re: git svn's performance issue and strange pauses, and other thing

Eric Wong <normalperson@yhbt.net> wrote:

> Jakob sent some patches a few months ago which seem to address the
> issue.  Unfortunately we forgot about them :x

Hin-Tak: have you tried Jakob's patches?  I've taken another look,
signed-off and pushed to my master.

Show 13 quoted lines

> Can you take a look at the following two "mergeinfo-speedups"
> in my repo?  (git://bogomips.org/git-svn)
> 
> Jakob Stoklund Olesen (2):
>       git-svn: only look at the new parts of svn:mergeinfo
>       git-svn: only look at the root path for svn:mergeinfo
> 
> Also downloadable here:
> 
> http://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535
> http://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a
> 
> Can you please give them a try?