# [PATCH] SVN import: Use one log call

7 messages from 2005-11-29 to 2005-11-30. Participants: Matthias Urlichs, Junio C Hamano, Nicolas Pitre.
Thread: https://gitlist.dev/t/2706

## Matthias Urlichs, 2005-11-29 07:13

Subject: [PATCH] SVN import: Use one log call
Message-ID: <pan.2005.11.29.07.13.02.145977@smurf.noris.de>
URL: https://gitlist.dev/e/pan.2005.11.29.07.13.02.145977%40smurf.noris.de

```
One "svn log" (or its equivalent) per revision adds delay and server load.
Instead, open two SVN connections -- one for the log, and one for the files.

Positive side effect: Only those log entries which actually contain data
are committed => no more empty commits.

Also, change the "-l" option to set the maximum revision to be pulled,
not the number of revisions.

---

... or pull from http://netz.smurf.noris.de/git/git.git#svnup.

 Documentation/git-svnimport.txt |   15 ++++++---------
 git-svnimport.perl              |   37 ++++++++++++++++---------------------
 2 files changed, 22 insertions(+), 30 deletions(-)

applies-to: 744acb3ef516240503d9a13cf1adb6a9670f905e
81da2a98762ef4965e6410c1c83600f2d80dc67a
diff --git a/Documentation/git-svnimport.txt b/Documentation/git-svnimport.txt
index fcc79fa..f8dbee7 100644
--- a/Documentation/git-svnimport.txt
+++ b/Documentation/git-svnimport.txt
@@ -10,7 +10,7 @@ git-svnimport - Import a SVN repository 
 SYNOPSIS
 --------
 'git-svnimport' [ -o <branch-for-HEAD> ] [ -h ] [ -v ] [ -d | -D ]
-			[ -C <GIT_repository> ] [ -i ] [ -u ] [-l limit_nr_changes]
+			[ -C <GIT_repository> ] [ -i ] [ -u ] [-l limit_rev]
 			[ -b branch_subdir ] [ -t trunk_subdir ] [ -T tag_subdir ]
 			[ -s start_chg ] [ -m ] [ -M regex ]
 			<SVN_repository_URL> [ <path> ]
@@ -71,14 +71,11 @@ When importing incementally, you might n
 	regex. It can be used with -m to also see the default regexes.
 	You must escape forward slashes.
 
--l <max_num_changes>::
-	Limit the number of SVN changesets we pull before quitting.
-	This option is necessary because the SVN library has serious memory
-	leaks; the recommended value for nontrivial imports is 100.
-
-	git-svnimport will still exit with a zero exit code. You can check
-	the size of the file ".git/svn2git" to determine whether to call
-	the importer again.
+-l <max_rev>::
+	Specify a maximum revision number to pull.
+
+	Formerly, this option controlled how many revisions to pull, due to
+	SVN memory leaks. (These have been worked around.)
 
 -v::
 	Verbosity: let 'svnimport' report what it is doing.
diff --git a/git-svnimport.perl b/git-svnimport.perl
index af13fdd..640d675 100755
--- a/git-svnimport.perl
+++ b/git-svnimport.perl
@@ -35,7 +35,7 @@ our($opt_h,$opt_o,$opt_v,$opt_u,$opt_C,$
 sub usage() {
 	print STDERR <<END;
 Usage: ${\basename $0}     # fetch/update GIT from SVN
-       [-o branch-for-HEAD] [-h] [-v] [-l max_num_changes]
+       [-o branch-for-HEAD] [-h] [-v] [-l max_rev]
        [-C GIT_repository] [-t tagname] [-T trunkname] [-b branchname]
        [-d|-D] [-i] [-u] [-s start_chg] [-m] [-M regex] [SVN_URL]
 END
@@ -126,8 +126,9 @@ sub file {
 package main;
 use URI;
 
-my $svn = $svn_url;
+our $svn = $svn_url;
 $svn .= "/$svn_dir" if defined $svn_dir;
+my $svn2 = SVNconn->new($svn);
 $svn = SVNconn->new($svn);
 
 my $lwp_ua;
@@ -198,7 +199,7 @@ $ENV{GIT_INDEX_FILE} = $git_index;
 my $maxnum = 0;
 my $last_rev = "";
 my $last_branch;
-my $current_rev = $opt_s-1;
+my $current_rev = $opt_s || 1;
 unless(-d $git_dir) {
 	system("git-init-db");
 	die "Cannot init the GIT db at $git_tree: $?\n" if $?;
@@ -250,7 +251,7 @@ EOM
 		my($num,$branch,$ref) = split;
 		$branches{$branch}{$num} = $ref;
 		$branches{$branch}{"LAST"} = $ref;
-		$current_rev = $num if $current_rev < $num;
+		$current_rev = $num+1 if $current_rev <= $num;
 	}
 	close($B);
 }
@@ -704,17 +705,17 @@ sub commit {
 	print "DONE: $revision $dest $cid\n" if $opt_v;
 }
 
-my ($changed_paths, $revision, $author, $date, $message, $pool) = @_;
-sub _commit_all {
-	($changed_paths, $revision, $author, $date, $message, $pool) = @_;
+sub commit_all {
+	# Recursive use of the SVN connection does not work
+	local $svn = $svn2;
+
+	my ($changed_paths, $revision, $author, $date, $message, $pool) = @_;
 	my %p;
 	while(my($path,$action) = each %$changed_paths) {
 		$p{$path} = [ $action->action,$action->copyfrom_path, $action->copyfrom_rev, $path ];
 	}
 	$changed_paths = \%p;
-}
 
-sub commit_all {
 	my %done;
 	my @col;
 	my $pref;
@@ -730,18 +731,12 @@ sub commit_all {
 	}
 }
 
-while(++$current_rev <= $svn->{'maxrev'}) {
-	if (defined $opt_l) {
-		$opt_l--;
-		if ($opt_l < 0) {
-			last;
-		}
-	}
-	my $pool=SVN::Pool->new;
-	$svn->{'svn'}->get_log("/",$current_rev,$current_rev,1,1,1,\&_commit_all,$pool);
-	$pool->clear;
-	commit_all();
-}
+$opt_l = $svn->{'maxrev'} if not defined $opt_l or $opt_l > $svn->{'maxrev'};
+print "Fetching from $current_rev to $opt_l ...\n" if $opt_v;
+
+my $pool=SVN::Pool->new;
+$svn->{'svn'}->get_log("/",$current_rev,$opt_l,0,1,1,\&commit_all,$pool);
+$pool->clear;
 
 
 unlink($git_index);
---


-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
There was a young peasant named Gorse Who fell madly in love with his horse.
Said his wife, "You rapscallion, That horse is a stallion -- This constitutes
grounds for divorce."

```

## Junio C Hamano, 2005-11-29 20:22

Subject: Re: [PATCH] SVN import: Use one log call
Message-ID: <7vhd9vdx1o.fsf@assigned-by-dhcp.cox.net>
URL: https://gitlist.dev/e/7vhd9vdx1o.fsf%40assigned-by-dhcp.cox.net
In-Reply-To: <pan.2005.11.29.07.13.02.145977@smurf.noris.de>

```
Matthias Urlichs <smurf@smurf.noris.de> writes:

> One "svn log" (or its equivalent) per revision adds delay and server load.
> Instead, open two SVN connections -- one for the log, and one for the files.

Thanks, applied and pushed out.

BTW, I've never successfully managed to run svnimport from my
private svn repository.  Admittedly the repository does not
follow the recommended layout and that probably is the major
cause (it started its life when svn documentation recommended
{trunk,branches,tags}/{projectA,projectB,projectC} layout.
{projectA,projectB,projectC}/{trunk,branches,tags} is the layout
they recommend these days, I think [*1*]).

It does not use any branches (it is primarily a random
collection of small throwaway scripts).  The repository hosts
many unrelated pieces ("http://127.0.0.1/svn/private/" is the
root level of the repository), organized like this:

    $ svn ls http://127.0.0.1/svn/private/
    attic/
    main/
    tags/

   main/ is the active one (trunk), and it has bunch of
   unrelated subdirectories.

   attic/ is where I prepared to "svn mv" things from main/ that
   are no longer needed, but is empty.

   tags/ have one tree that is a copy of one of the subtrees
   under main/ from distant past.

I wanted to convert one of the subsubdirectory of main to git.

    $ svn ls http://127.0.0.1/svn/private/main/sources/photocat
    Makefile
    Notes
    cmdmason.pm
    ...

What is the svnimport command line I should give?  Luckily, I do
not have "tags" or "branches" under private/main/sources/, so I
tried to cheat like this, hoping it would mistake "photocat" is the trunk of 
"main/sources" project in the repository.  No such luck.

$ cd /var/tmp && rm -fr try0 && mkdir try0 && cd try0
$ git svnimport -v -i -t photocat http://127.0.0.1/svn/private main/sources
1: Unrecognized path: /main/sources
1: Unrecognized path: /main/in-place
1: Unrecognized path: /main
...
1500: Unrecognized path: /main/sources/photocat/db/catalog.sql
1501: Unrecognized path: /main/sources/photocat/data/035-maribon-making.yaml
DONE; creating master branch
cp: cannot stat `/var/tmp/try0/.git/refs/heads/origin': No such file or directory
fatal: master: not a valid SHA1
$ 

If your answer is "your repository layout is too weird and
nonstandard, you are screwed", that is perfectly fine.  I do not
want you to bend over backwards to butcher the import script to
support it, if it is too nonstandard.  I already converted what
I wanted to convert manually already; history being linear
without branches, that was easy enough.

But I thought it would never hurt to ask ;-).

[Footnote]

*1*

http://svnbook.red-bean.com/en/1.1/svn-book.html#svn-ch-5-sect-6.1
shows two layouts, one with {trunk,tags,branches} at the top
level of each project, another with nested projects (if you look
at "utils" in the picture as a project with two subcomponents
"calc" and "calendar"), with {trunk,tags,branches} under each
subproject.  I think the current code should import from
"calendar" or "calc" level just fine, but I wonder if we want to
support importing from "utils" level.

```

## Matthias Urlichs, 2005-11-30 02:46

Subject: Re: [PATCH] SVN import: Use one log call
Message-ID: <20051130024610.GX20418@kiste.smurf.noris.de>
URL: https://gitlist.dev/e/20051130024610.GX20418%40kiste.smurf.noris.de
In-Reply-To: <7vhd9vdx1o.fsf@assigned-by-dhcp.cox.net>

```
Hi,

Junio C Hamano:
> $ git svnimport -v -i -t photocat http://127.0.0.1/svn/private main/sources
> ...
> 1500: Unrecognized path: /main/sources/photocat/db/catalog.sql

Hmmm, "git-svnimport -T main/sources" should work. Somewhat.

>>/
>>   utils/
>>      calc/
>>         trunk/
>>         tags/
>>         branches/
>>      calendar/
>>         trunk/
>>         tags/
>>         branches/
>>      …

>                                       but I wonder if we want to
> support importing from "utils" level.
> 
That'd require somewhat more flexible support for reordering the
repository path elements than we have now, I'm afraid, thus the answer
to that particular question is "import both separately, then merge
into a common git repo if you really need to". We *can* do that, after
all. ;-)

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
You goddamn cornhuskers are all alike.
					-- Jim Thompson

```

## Nicolas Pitre, 2005-11-30 03:54

Subject: Re: [PATCH] SVN import: Use one log call
Message-ID: <Pine.LNX.4.64.0511292247591.3122@localhost.localdomain>
URL: https://gitlist.dev/e/Pine.LNX.4.64.0511292247591.3122%40localhost.localdomain
In-Reply-To: <7vhd9vdx1o.fsf@assigned-by-dhcp.cox.net>

```
On Tue, 29 Nov 2005, Junio C Hamano wrote:

> BTW, I've never successfully managed to run svnimport from my
> private svn repository.

Same here, with the following public repo: svn://mielke.cc/main/brltty

> $ git svnimport -v -i -t photocat http://127.0.0.1/svn/private main/sources
> 1: Unrecognized path: /main/sources
> 1: Unrecognized path: /main/in-place
> 1: Unrecognized path: /main
> ...
> 1500: Unrecognized path: /main/sources/photocat/db/catalog.sql
> 1501: Unrecognized path: /main/sources/photocat/data/035-maribon-making.yaml
> DONE; creating master branch
> cp: cannot stat `/var/tmp/try0/.git/refs/heads/origin': No such file or directory
> fatal: master: not a valid SHA1
> $ 

And I only get similar results.

> If your answer is "your repository layout is too weird and
> nonstandard, you are screwed", that is perfectly fine.  I do not
> want you to bend over backwards to butcher the import script to
> support it, if it is too nonstandard.

Thing is the above repository is not _that_ weird.  And with the real 
svn it produces a proper source tree of course, without any special 
options.  So I would think git-svnimport should be able to do the same.  
No?


Nicolas

```

## Matthias Urlichs, 2005-11-30 06:52

Subject: Re: [PATCH] SVN import: Use one log call
Message-ID: <20051130065241.GY20418@kiste.smurf.noris.de>
URL: https://gitlist.dev/e/20051130065241.GY20418%40kiste.smurf.noris.de
In-Reply-To: <Pine.LNX.4.64.0511292247591.3122@localhost.localdomain>

```
Hi,

Nicolas Pitre:
> Same here, with the following public repo: svn://mielke.cc/main/brltty
> 
Same thing -- should work with a "-T main/brltty" option.

> Thing is the above repository is not _that_ weird.  And with the real 
> svn it produces a proper source tree of course, without any special 
> options.  So I would think git-svnimport should be able to do the same.  
> No?

Sure. The problem is that svn doesn't have tags, branches, or different
projects; it's all just subdirectories with linked files.

So you need a good mapping from one to the other. In principle that's
simple, it's just that nobody has written the code for a more flexible
mapping from svn paths to tags / branch+subpath yet.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
"Adam may not have been so perfect after the 'fall," but he was
 not so big a fool."
      [Lemuel K. Washburn, _Is The Bible Worth Reading
                             And Other Essays_]

```

## Junio C Hamano, 2005-11-30 07:54

Subject: Re: [PATCH] SVN import: Use one log call
Message-ID: <7vwtiqwoy7.fsf@assigned-by-dhcp.cox.net>
URL: https://gitlist.dev/e/7vwtiqwoy7.fsf%40assigned-by-dhcp.cox.net
In-Reply-To: <20051130065241.GY20418@kiste.smurf.noris.de>

```
Matthias Urlichs <smurf@smurf.noris.de> writes:

> Nicolas Pitre:
>> Same here, with the following public repo: svn://mielke.cc/main/brltty
>> 
> Same thing -- should work with a "-T main/brltty" option.

So would the full command line be like this?

	$ rm -rf try1; mkdir try1; cd try1
        $ git-svnimport -T main/brltty svn://mielke.cc/main/brltty

I realize my first attempt was incorrectly using "-t" (tags)
instead of "-T" (trunk), which was obviously incorrect, but
replacing that lowercase -t with -T did not work well.  However,
as you told me:

	$ git svnimport -T main/sources \
	    http://127.0.0.1/svn/private/main/sources/photocat

seems to be doing something.  But I do not understand why.  How
did you decide "-T main/sources" is good for my case while "-T
main/brltty" is good for Nicolas?  I would have guessed you may
say "-T main" for Nicolas, dropping the last path component.

```

## Matthias Urlichs, 2005-11-30 08:34

Subject: Re: [PATCH] SVN import: Use one log call
Message-ID: <20051130083436.GZ20418@kiste.smurf.noris.de>
URL: https://gitlist.dev/e/20051130083436.GZ20418%40kiste.smurf.noris.de
In-Reply-To: <7vwtiqwoy7.fsf@assigned-by-dhcp.cox.net>

```
Hi,

Junio C Hamano:
> 	$ git svnimport -T main/sources \
> 	    http://127.0.0.1/svn/private/main/sources/photocat
> 
> seems to be doing something.  But I do not understand why.  How
> did you decide "-T main/sources" is good for my case while "-T
> main/brltty" is good for Nicolas?  I would have guessed you may
> say "-T main" for Nicolas, dropping the last path component.
> 
What git-svnimport does right now is rather simple. It pulls the log
from the SVN URL, drops everything that does not have a prefix of either
of the -t tags -T trunk -b branches options. The prefix is then thrown
away. If it's a trunk, import the rest to the "origin" branch; otherwise
remove the next path component and use that as either the branch name
or the tag name.

The point is that the picture in the SVN manual is incomplete. 
There's not just a branches/ subdirectory alongside the trunk/ one;
trunk/ directly contains the project's files, while branches/ contains
subdirectories which are named for the branch, and *they* contain the
files.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
If you're looking ahead long-term, and what you see looks like
science fiction, it might be wrong. But if it doesn't look like
science fiction, it's definitely wrong...
		-- Christine Peterson,

```