Re: Excruciatingly slow git-svn imports

From: Eric Wong <normalperson@yhbt.net>
Date: May 6, 2008, 04:25 UTC
Message-ID: <20080506042508.GA23465@untitled>
In-Reply-To: <32541b130805052056g450b69cfg46693bc3c0c5a1ed@mail.gmail.com>

Avery Pennarun <apenwarr@gmail.com> wrote:

Show 18 quoted lines

> On 5/5/08, Eric Wong <normalperson@yhbt.net> wrote:
> > Interesting.  By  "These commits seemed all to have thousands of files",
> >  you mean the first 35K that took up most of the time?  If so, yes,
> >  that's definitely a problem...
> >
> >  git-svn requests a log from SVN containing a list of all paths modified
> >  in each revision.  By default, git-svn only requests log entries for up
> >  to 100 revisions at a time to reduce memory usage.  However, having
> >  thousands of files modified for each revision would still be
> >  problematic, as would having insanely long commit messages.
> 
> On my system, any branch that was created using "svn cp" of a toplevel
> directory seems to cause git-svn to (rather slowly) download every
> single file in the entire branch for the first commit on that branch,
> giving a symptom that sounds a lot like the above "commits with
> thousands of files".  I assumed this was just an intentional design
> decision in git-svn, to be slow and safe instead of fast and loose.
> Is it actually supposed to do something smarter than that?

When using "svn cp" on a top-level directory, it *should* just show up as a single file change in the log entry. Something like:

  A /project/branch/my-new-branch (from /project/trunk:1234)

This would not take much memory at all. However, I've also occasionally seen stuff like this:

  A /project/branch/my-new-branch
  A /project/branch/my-new-branch/file1 (from /project/trunk/file1:1234)
  A /project/branch/my-new-branch/file2 (from /project/trunk/file2:1234)
  A /project/branch/my-new-branch/file3 (from /project/trunk/file3:1234)
  .... many more files and directories along the same lines ...

This is what I suspect Geert is seeing in his repository and causing problems. Perhaps something caused by cvs2svn importing those tags into SVN originally?

But the symptom you're seeing with git-svn downloading every file seems to be the result of using a pre-1.4.3 version of the Perl SVN bindings which lacked a working do_switch() function. I fallback to using do_update() and checking out a new tree for SVN 1.4.2 and before. So yes, I'm definitely safe, slow and _lazy_ by falling back to do_update() instead of doing something fancy to workaround something that's already fixed in SVN :)

-- 
Eric Wong