# git svn's performance issue and strange pauses, and other thing 4 messages from 2014-09-18 to 2014-10-05. Participants: Hin-Tak Leung, Eric Wong, Jakob Stoklund Olesen. Thread: https://gitlist.dev/t/37586 ## Hin-Tak Leung, 2014-09-18 07:39 Subject: git svn's performance issue and strange pauses, and other thing Message-ID: <1411025993.80693.YahooMailBasic@web172304.mail.ir2.yahoo.com> URL: https://gitlist.dev/e/1411025993.80693.YahooMailBasic%40web172304.mail.ir2.yahoo.com ``` (I am not on the list - please CC) Thanks for git-svn - I use it instead of subversion itself for many years now. Just thought I'd ask/report a few issues I noticed for some time now, of tracking development of a particular subversion-based development project. Broadly speaking, I think there are 3 problems, especially noticeable against a particular repository, but to a lesser extent with some others too. - just doing "git svn fetch --all" seems to consume a lot of memory, for very little actual fetched changes. (in the 2GB+ region, sometimes). - "git svn fetch --all" also seems to take a long time too, for certain fetched changes. (in the minutes region). - I know I can probably just "read the source", but I'd like to know why .git/svn/.caches is even larger than .git/objects (which supposedly contains everything that's of interest)? I hope this can be documented towards the end of the man-page, for example, of important parts of .git/svn (and what not to do with them...), without needing to 'read the source'. Here is part of "du" from a couple of days ago: 254816 .git/objects 307056 .git/svn/.caches 332452 .git/svn 588064 .git The actual .git/config is here - this should be sufficient info for somebody looking into experiencing the issues I mentioned above. -------- $ more .git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [svn-remote "svn"] url = https://svn.r-project.org/R fetch = trunk:refs/remotes/trunk branches = branches/*:refs/remotes/* tags = tags/*:refs/remotes/tags/* [pack] threads = 1 ------------ ``` ## Eric Wong, 2014-09-19 08:25 Subject: Re: git svn's performance issue and strange pauses, and other thing Message-ID: <20140919082529.GA32459@dcvr.yhbt.net> URL: https://gitlist.dev/e/20140919082529.GA32459%40dcvr.yhbt.net In-Reply-To: <1411025993.80693.YahooMailBasic@web172304.mail.ir2.yahoo.com> ``` Hin-Tak Leung wrote: > (I am not on the list - please CC) Done, it is standard practice for git :) > Thanks for git-svn - I use it instead of subversion itself for many years now. > > Just thought I'd ask/report a few issues I noticed for some time > now, of tracking development of a particular subversion-based > development project. Broadly speaking, I think there are 3 problems, > especially noticeable against a particular repository, but > to a lesser extent with some others too. > > - just doing "git svn fetch --all" seems to consume a lot of memory, > for very little actual fetched changes. (in the 2GB+ region, sometimes). > > - "git svn fetch --all" also seems to take a long time too, for certain > fetched changes. (in the minutes region). Jakob sent some patches a few months ago which seem to address the issue. Unfortunately we forgot about them :x Can you take a look at the following two "mergeinfo-speedups" in my repo? (git://bogomips.org/git-svn) Jakob Stoklund Olesen (2): git-svn: only look at the new parts of svn:mergeinfo git-svn: only look at the root path for svn:mergeinfo Also downloadable here: http://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535 http://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a Can you please give them a try? > - I know I can probably just "read the source", but I'd like to know > why .git/svn/.caches is even larger than .git/objects (which supposedly > contains everything that's of interest)? I hope this can be documented > towards the end of the man-page, for example, of important parts > of .git/svn (and what not to do with them...), without needing to > 'read the source'. Here is part of "du" from a couple of days ago: > > 254816 .git/objects > 307056 .git/svn/.caches > 332452 .git/svn > 588064 .git > > The actual .git/config is here - this should be sufficient info for > somebody looking into experiencing the issues I mentioned above. IIRC, the caching is unique to mergeinfo, so perhaps Jakob's patches help, there, too. Sorry I don't understand the mergeinfo stuff more, I've never worked on a project which uses it. > -------- > $ more .git/config > [core] > repositoryformatversion = 0 > filemode = true > bare = false > logallrefupdates = true > [svn-remote "svn"] > url = https://svn.r-project.org/R > fetch = trunk:refs/remotes/trunk > branches = branches/*:refs/remotes/* > tags = tags/*:refs/remotes/tags/* > [pack] > threads = 1 > ------------ ``` ## Jakob Stoklund Olesen, 2014-09-19 13:44 Subject: Re: git svn's performance issue and strange pauses, and other thing Message-ID: URL: https://gitlist.dev/e/DE2A36B7-69F2-46CA-89FC-C2038F08179A%402pi.dk In-Reply-To: <20140919082529.GA32459@dcvr.yhbt.net> ``` > On Sep 19, 2014, at 1:25, Eric Wong wrote: > > Hin-Tak Leung wrote: > >> - I know I can probably just "read the source", but I'd like to know >> why .git/svn/.caches is even larger than .git/objects (which supposedly >> contains everything that's of interest)? I hope this can be documented >> towards the end of the man-page, for example, of important parts >> of .git/svn (and what not to do with them...), without needing to >> 'read the source'. Here is part of "du" from a couple of days ago: >> >> 254816 .git/objects >> 307056 .git/svn/.caches >> 332452 .git/svn >> 588064 .git >> >> The actual .git/config is here - this should be sufficient info for >> somebody looking into experiencing the issues I mentioned above. > > IIRC, the caching is unique to mergeinfo, so perhaps Jakob's patches > help, there, too. IIRC the caches are used for memoization, and with my two patches applied it doesn't improve performance much. You could try removing the memoization after applying my patches. Thanks, /Jakob ``` ## Eric Wong, 2014-10-05 01:02 Subject: Re: git svn's performance issue and strange pauses, and other thing Message-ID: <20141005010243.GA2926@dcvr.yhbt.net> URL: https://gitlist.dev/e/20141005010243.GA2926%40dcvr.yhbt.net In-Reply-To: <20140919082529.GA32459@dcvr.yhbt.net> ``` Eric Wong wrote: > Jakob sent some patches a few months ago which seem to address the > issue. Unfortunately we forgot about them :x Hin-Tak: have you tried Jakob's patches? I've taken another look, signed-off and pushed to my master. > Can you take a look at the following two "mergeinfo-speedups" > in my repo? (git://bogomips.org/git-svn) > > Jakob Stoklund Olesen (2): > git-svn: only look at the new parts of svn:mergeinfo > git-svn: only look at the root path for svn:mergeinfo > > Also downloadable here: > > http://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535 > http://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a > > Can you please give them a try? ```