{"thread":{"id":"37586","subject":"git svn's performance issue and strange pauses, and other thing","startedAt":"2014-09-18T07:39:53Z","lastAt":"2014-10-05T01:02:43Z","messageCount":4,"participants":["Hin-Tak Leung","Eric Wong","Jakob Stoklund Olesen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"249628","messageId":"1411025993.80693.YahooMailBasic@web172304.mail.ir2.yahoo.com","threadId":"37586","inReplyTo":null,"subject":"git svn's performance issue and strange pauses, and other thing","fromName":"Hin-Tak Leung","fromEmail":"htl10@users.sourceforge.net","sentAt":"2014-09-18T07:39:53Z","receivedAt":"2014-09-18T07:39:53Z","isPatch":false,"sender":{"key":"htl10@users.sourceforge.net","avatar":null},"body":"(I am not on the list - please CC)\n\nThanks for git-svn - I use it instead of subversion itself for many years now.\n\nJust thought I'd ask/report a few issues I noticed for some time\nnow, of tracking development of a particular subversion-based\ndevelopment project. Broadly speaking, I think there are 3 problems,\nespecially noticeable against a particular repository, but \nto a lesser extent with some others too.\n\n- just doing \"git svn fetch --all\" seems to consume a lot of memory,\nfor very little actual fetched changes. (in the 2GB+ region, sometimes).\n\n- \"git svn fetch --all\" also seems to take a long time too, for certain\nfetched changes. (in the minutes region).\n\n- I know I can probably just \"read the source\", but I'd like to know\nwhy .git/svn/.caches is even larger than .git/objects (which supposedly\ncontains everything that's of interest)? I hope this can be documented\ntowards the end of the man-page, for example, of important parts\nof .git/svn (and what not to do with them...), without needing to\n'read the source'. Here is part of \"du\" from a couple of days ago:\n\n254816\t.git/objects\n307056\t.git/svn/.caches\n332452\t.git/svn\n588064\t.git\n\nThe actual .git/config is here - this should be sufficient info for\nsomebody looking into experiencing the issues I mentioned above.\n\n--------\n$ more .git/config \n[core]\n\trepositoryformatversion = 0\n\tfilemode = true\n\tbare = false\n\tlogallrefupdates = true\n[svn-remote \"svn\"]\n\turl = https://svn.r-project.org/R\n\tfetch = trunk:refs/remotes/trunk\n\tbranches = branches/*:refs/remotes/*\n\ttags = tags/*:refs/remotes/tags/*\n[pack]\n\tthreads = 1\n------------\n"},{"id":"249654","messageId":"20140919082529.GA32459@dcvr.yhbt.net","threadId":"37586","inReplyTo":"1411025993.80693.YahooMailBasic@web172304.mail.ir2.yahoo.com","subject":"Re: git svn's performance issue and strange pauses, and other thing","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-09-19T08:25:29Z","receivedAt":"2014-09-19T08:25:29Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hin-Tak Leung wrote:\n> (I am not on the list - please CC)\n\nDone, it is standard practice for git :)\n\n> Thanks for git-svn - I use it instead of subversion itself for many years now.\n> \n> Just thought I'd ask/report a few issues I noticed for some time\n> now, of tracking development of a particular subversion-based\n> development project. Broadly speaking, I think there are 3 problems,\n> especially noticeable against a particular repository, but \n> to a lesser extent with some others too.\n> \n> - just doing \"git svn fetch --all\" seems to consume a lot of memory,\n> for very little actual fetched changes. (in the 2GB+ region, sometimes).\n> \n> - \"git svn fetch --all\" also seems to take a long time too, for certain\n> fetched changes. (in the minutes region).\n\nJakob sent some patches a few months ago which seem to address the\nissue. Unfortunately we forgot about them :x\n\nCan you take a look at the following two \"mergeinfo-speedups\"\nin my repo? (git://bogomips.org/git-svn)\n\nJakob Stoklund Olesen (2):\n git-svn: only look at the new parts of svn:mergeinfo\n git-svn: only look at the root path for svn:mergeinfo\n\nAlso downloadable here:\n\nhttp://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535\nhttp://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a\n\nCan you please give them a try?\n\n> - I know I can probably just \"read the source\", but I'd like to know\n> why .git/svn/.caches is even larger than .git/objects (which supposedly\n> contains everything that's of interest)? I hope this can be documented\n> towards the end of the man-page, for example, of important parts\n> of .git/svn (and what not to do with them...), without needing to\n> 'read the source'. Here is part of \"du\" from a couple of days ago:\n> \n> 254816\t.git/objects\n> 307056\t.git/svn/.caches\n> 332452\t.git/svn\n> 588064\t.git\n> \n> The actual .git/config is here - this should be sufficient info for\n> somebody looking into experiencing the issues I mentioned above.\n\nIIRC, the caching is unique to mergeinfo, so perhaps Jakob's patches\nhelp, there, too.\n\nSorry I don't understand the mergeinfo stuff more, I've never worked on\na project which uses it.\n\n> --------\n> $ more .git/config \n> [core]\n> \trepositoryformatversion = 0\n> \tfilemode = true\n> \tbare = false\n> \tlogallrefupdates = true\n> [svn-remote \"svn\"]\n> \turl = https://svn.r-project.org/R\n> \tfetch = trunk:refs/remotes/trunk\n> \tbranches = branches/*:refs/remotes/*\n> \ttags = tags/*:refs/remotes/tags/*\n> [pack]\n> \tthreads = 1\n> ------------\n"},{"id":"249658","messageId":"DE2A36B7-69F2-46CA-89FC-C2038F08179A@2pi.dk","threadId":"37586","inReplyTo":"20140919082529.GA32459@dcvr.yhbt.net","subject":"Re: git svn's performance issue and strange pauses, and other thing","fromName":"Jakob Stoklund Olesen","fromEmail":"stoklund@2pi.dk","sentAt":"2014-09-19T13:44:00Z","receivedAt":"2014-09-19T13:44:00Z","isPatch":false,"sender":{"key":"stoklund@2pi.dk","avatar":"https://avatars.githubusercontent.com/u/12660495?v=4"},"body":"\n\n> On Sep 19, 2014, at 1:25, Eric Wong wrote:\n> \n> Hin-Tak Leung wrote:\n> \n>> - I know I can probably just \"read the source\", but I'd like to know\n>> why .git/svn/.caches is even larger than .git/objects (which supposedly\n>> contains everything that's of interest)? I hope this can be documented\n>> towards the end of the man-page, for example, of important parts\n>> of .git/svn (and what not to do with them...), without needing to\n>> 'read the source'. Here is part of \"du\" from a couple of days ago:\n>> \n>> 254816 .git/objects\n>> 307056 .git/svn/.caches\n>> 332452 .git/svn\n>> 588064 .git\n>> \n>> The actual .git/config is here - this should be sufficient info for\n>> somebody looking into experiencing the issues I mentioned above.\n> \n> IIRC, the caching is unique to mergeinfo, so perhaps Jakob's patches\n> help, there, too.\n\nIIRC the caches are used for memoization, and with my two patches applied it doesn't improve performance much.\n\nYou could try removing the memoization after applying my patches.\n\nThanks,\n/Jakob"},{"id":"250253","messageId":"20141005010243.GA2926@dcvr.yhbt.net","threadId":"37586","inReplyTo":"20140919082529.GA32459@dcvr.yhbt.net","subject":"Re: git svn's performance issue and strange pauses, and other thing","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-05T01:02:43Z","receivedAt":"2014-10-05T01:02:43Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Eric Wong wrote:\n> Jakob sent some patches a few months ago which seem to address the\n> issue. Unfortunately we forgot about them :x\n\nHin-Tak: have you tried Jakob's patches? I've taken another look,\nsigned-off and pushed to my master.\n\n> Can you take a look at the following two \"mergeinfo-speedups\"\n> in my repo? (git://bogomips.org/git-svn)\n> \n> Jakob Stoklund Olesen (2):\n> git-svn: only look at the new parts of svn:mergeinfo\n> git-svn: only look at the root path for svn:mergeinfo\n> \n> Also downloadable here:\n> \n> http://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535\n> http://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a\n> \n> Can you please give them a try?\n"}]}