threads / discuss / 8003

importing multi-project svn repositories

Subject: importing multi-project svn repositories

## tl;dr

9 messages between May 6, 2007 and May 13, 2007.

replies: 8people: 5as markdown or json

David Hanson· May 6, 2007, 18:56 UTC · lore

What's the recommended way to import the full history from an svn repository organized as a group of projects? E.g., as described at http://svnbook.red-bean.com/nightly/en/ svn.reposadmin.planning.html#svn.reposadmin.projects.chooselayout:

/
    calc/
       trunk/
       tags/
       branches/
    calendar/
       trunk/
       tags/
       branches/
    spreadsheet/
       trunk/
       tags/
       branches/
    …

I'd like to import calc at the top level, put calc/tags/foo in git's tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto for calendar, spreadsheet, etc.

I suspect there's a way to use git-svnimport repeatedly with appropriate editing of .git/svn2git. thanks, dave h

Steven Grimm· May 7, 2007, 18:06 UTC · re: David Hanson · lore

Re: importing multi-project svn repositories

David Hanson wrote:
> I'd like to import calc at the top level, put calc/tags/foo in git's 
> tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto for 
> calendar, spreadsheet, etc.

Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer really maintained. The former will do what you want pretty easily. You just run it like

git svn clone --prefix=calc/ -T trunk -t tags -b branches \
    http://svn/repo/url/calc gitrepo

and it'll create a git repository called "gitrepo" with the calc project's trunk/tags/branches as git tags. Then do

git svn clone --prefix=spreadsheet/ -T trunk -t tags -b branches \
    http://svn/repo/url/spreadsheet gitrepo

and it will add the spreadsheet branches to the same repo. The "--prefix" option is needed to keep the branch namespaces from overlapping. git-svn will make an attempt to figure out the correct history of the branches and tags relative to the trunk so they look like sane git branches.

If you want to be able to check out calc and spreadsheet simultaneously, then you probably need to use git's submodule support, which is kind of a work in progress at the moment.

-Steve
Dave Hanson· May 8, 2007, 03:48 UTC · re: Steven Grimm · lore

Re: importing multi-project svn repositories

On 5/7/07, Steven Grimm <koreth@midwinter.com> wrote:
Show 7 quoted lines
> David Hanson wrote:
> > I'd like to import calc at the top level, put calc/tags/foo in git's
> > tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto for
> > calendar, spreadsheet, etc.
>
> Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer
> really maintained. ...

Thanks for the tip. I was running git 1.5.0, which didn't support "clone" to git-svn. I upgraded to 1.5.1.3 and followed your suggestions.

If I decide to flush the svn repository in favor of git, I'll probably do some manual rearrangement so that commands like "git tag -l" gives useful results. thanks, dave h

Steven Grimm· May 8, 2007, 07:16 UTC · re: Dave Hanson · lore

Deprecate git-svnimport?

On 5/7/07, Steven Grimm <koreth@midwinter.com> wrote:
> Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer
> really maintained. ...

I said that because in the time I've been using git and reading this list, I think I've seen a grand total of one patch to git-svnimport, a ton of them to git-svn, and an occasional new user posting about git-svnimport failing to do something that git-svn would have done if they'd used it instead.

It looks like git-svnimport hasn't been updated to the 1.5.x remote branch naming convention (it still calls the svn head tracking branch "origin" if its manpage is to be believed).

Originally it looked like git-svn was what you used if you wanted to track just one svn branch bidirectionally, and git-svnimport was how you tracked multiple svn branches read-only. But now that git-svn has good handling of multi-branch svn repositories, I wonder if git-svnimport still serves a purpose. Should it perhaps be deprecated and removed in a future release? Or is it still useful for things that git-svn doesn't handle?

-Steve
Junio C Hamano· May 8, 2007, 09:58 UTC · re: Steven Grimm · lore

minimize_url in git-svn?

I was trying to run git-svn against this:
	https://repo.socialtext.net:8999/svn/socialtext/trunk

This is an open source project [*1*] and the trunk is supposed to be readable by everybody, but it seems that anything outside that area needs authentication. If I mimick the example in git-svn.txt manual page to clone from there, it creates trunk, trunk/.git, and then asks for password:

	$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk
	$ git-svn clone $URL
        Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN
	Password for 'junio': ^C

I've narrowed it down to this part of git-svn. If I tell it not to bother "minimiz"ing the URL, it seems to import without stepping outside of the URL it was given.

--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1038,7 +1038,8 @@
 		}
 		$self->{repo_id} = $existing;
 	} else {
-		my $min_url = Git::SVN::Ra->new($url)->minimize_url;
+		my $ra = Git::SVN::Ra->new($url);
+		my $min_url = $url; # $ra->minimize_url;
 		$existing = find_existing_remote($min_url, $r);
 		if ($existing) {
 			unless ($no_write) {

Two and half questions.

 * What does minimize do, and why is it necessary?

 * The resulting git-svn remote tracking branch (and 'master')
   seems to check out fine, but I do not know what damage the
   hack to avoid minimizing is causing.  Are there any?  I see
   many 0{40} lines in trunk/.git/svn/git-svn/.rev_db.* file,
   and also many lines in unhandled.log file (+empty_dir,
   +file_prop, and +dir_prop).  Are these something to worry
   about?

 * Assuming there aren't any damage, or maybe some damage that
   would cause minor decreased functionality/interoperability,
   would it perhaps make sense to optionally allow skipping the
   minimizing to avoid this problem?  Would it make sense, or is
   the setting at socialtext site too esoteric and it isn't
   worth to worry about?


[Footnote]

*1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env
Johannes Schindelin· May 8, 2007, 11:28 UTC · re: Junio C Hamano · lore

Re: minimize_url in git-svn?

Hi,
On Tue, 8 May 2007, Junio C Hamano wrote:
Show 14 quoted lines
> I was trying to run git-svn against this:
> 
> 	https://repo.socialtext.net:8999/svn/socialtext/trunk
> 
> This is an open source project [*1*] and the trunk is supposed
> to be readable by everybody, but it seems that anything outside
> that area needs authentication.  If I mimick the example in
> git-svn.txt manual page to clone from there, it creates trunk,
> trunk/.git, and then asks for password:
> 
> 	$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk
> 	$ git-svn clone $URL
>         Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN
> 	Password for 'junio': ^C
FWIW I encountered the same problem with tailor.

Ciao, Dscho

Eric Wong· May 8, 2007, 19:34 UTC · re: Junio C Hamano · lore

Re: minimize_url in git-svn?

Junio C Hamano <junkio@cox.net> wrote:
Show 31 quoted lines
> I was trying to run git-svn against this:
> 
> 	https://repo.socialtext.net:8999/svn/socialtext/trunk
> 
> This is an open source project [*1*] and the trunk is supposed
> to be readable by everybody, but it seems that anything outside
> that area needs authentication.  If I mimick the example in
> git-svn.txt manual page to clone from there, it creates trunk,
> trunk/.git, and then asks for password:
> 
> 	$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk
> 	$ git-svn clone $URL
>         Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN
> 	Password for 'junio': ^C
> 
> I've narrowed it down to this part of git-svn.  If I tell it not
> to bother "minimiz"ing the URL, it seems to import without
> stepping outside of the URL it was given.
> 
> --- a/git-svn.perl
> +++ b/git-svn.perl
> @@ -1038,7 +1038,8 @@
>  		}
>  		$self->{repo_id} = $existing;
>  	} else {
> -		my $min_url = Git::SVN::Ra->new($url)->minimize_url;
> +		my $ra = Git::SVN::Ra->new($url);
> +		my $min_url = $url; # $ra->minimize_url;
>  		$existing = find_existing_remote($min_url, $r);
>  		if ($existing) {
>  			unless ($no_write) {
That should be fine.
> Two and half questions.
> 
>  * What does minimize do, and why is it necessary?

I try to connect to the root (or closer to the root) of the repository. This allows branches and tags to be tracked more effectively without needing reconnects. There's a reparent function in SVN 1.4, but it doesn't work correctly with svn:// repos last I checked (1.4.3)

Show 7 quoted lines
>  * The resulting git-svn remote tracking branch (and 'master')
>    seems to check out fine, but I do not know what damage the
>    hack to avoid minimizing is causing.  Are there any?  I see
>    many 0{40} lines in trunk/.git/svn/git-svn/.rev_db.* file,
>    and also many lines in unhandled.log file (+empty_dir,
>    +file_prop, and +dir_prop).  Are these something to worry
>    about?
Nope.  unhandled.log is strictly informational.

.rev_db is offset-based database. Revision numbers to git commits can be looked up using (SVN revision * 41).

If the project has really high revision numbers (like gcc) or lots of tags, it's a space-killer. I've been meaning to add an optional SQLite alternative to .rev_db for people tracking those projects. Patches welcome :)

Show 6 quoted lines
>  * Assuming there aren't any damage, or maybe some damage that
>    would cause minor decreased functionality/interoperability,
>    would it perhaps make sense to optionally allow skipping the
>    minimizing to avoid this problem?  Would it make sense, or is
>    the setting at socialtext site too esoteric and it isn't
>    worth to worry about?

It *should* be automatically detecting the highest level up it can access and stop there. In your case, there's obviously something broken in my code :(

I've definitely tested this as working against Seth Falcon's hedgehog repo (URL is somewhere in the archives). I also setup a test repository somewhere that I can double-check against.

> [Footnote]
> 
> *1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env

I'll try to take a look at that in the next few days. I also have segfaults to fix that I haven't gotten to :(

-- 
Eric Wong
Junio C Hamano· May 9, 2007, 04:56 UTC · re: Eric Wong · lore

Re: minimize_url in git-svn?

Eric Wong <normalperson@yhbt.net> writes:
> .rev_db is offset-based database.  Revision numbers to git commits can
> be looked up using (SVN revision * 41).

Ah, that explains why the file looks so sparse. Their trunk/ seems to be somewhat inactive and activities elsewhere in the branch/ namespace we cannot view may advance SVN revisions.

Show 10 quoted lines
>>  * Assuming there aren't any damage, or maybe some damage that
>>    would cause minor decreased functionality/interoperability,
>>    would it perhaps make sense to optionally allow skipping the
>>    minimizing to avoid this problem?  Would it make sense, or is
>>    the setting at socialtext site too esoteric and it isn't
>>    worth to worry about?
>
> It *should* be automatically detecting the highest level up it can
> access and stop there.  In your case, there's obviously something
> broken in my code :(

Perhaps, but if the user has the proper user identity to access the full site, I suspect you would be better off getting the credential and connect to the higher level. In other words, the definition of "the highest level up it can access" depends on the user and what the user is trying to do (i.e. a user who does have an account with that project may not be interested in getting tags/ nor branches/, in which case the user may want to be able to say "I am not interested in uplevel" upfront, so that the program does not even have to ask for the password. On the other hand the user may want to access tags/ and branches/ in which case the user would appreciate the current behaviour, asking for the password and go up after authenticated).

Show 10 quoted lines
> I've definitely tested this as working against Seth Falcon's hedgehog
> repo (URL is somewhere in the archives).  I also setup a test repository
> somewhere that I can double-check against.
>
>> [Footnote]
>> 
>> *1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env
>
> I'll try to take a look at that in the next few days.  I also have
> segfaults to fix that I haven't gotten to :(

Thanks. If I type <Return> instead of ^C in the transcript at the beginning of my message, I get a segfault. It may or may not be the same one you have been chasing.

Eric Wong· May 13, 2007, 16:58 UTC · re: Eric Wong · lore

[PATCH] git-svn: don't attempt to minimize URLs by default

For tracking branches and tags, git-svn prefers to connect to the root of the repository or at least the level that houses branches and tags as well as trunk. However, users that are accustomed to tracking a single directory have no use for this feature.

As pointed out by Junio, users may not have permissions to connect to connect to a higher-level path in the repository.

While the current minimize_url() function detects lack of permissions to certain paths _after_ successful logins, it cannot effectively determine if it is trying to access a login-only portion of a repo when the user expects to connect to a part where anonymous access is allowed.

For people used to the git-svnimport switches of --trunk, --tags, --branches, they'll already pass the repository root (or root+subdirectory), so minimize URL isn't of too much use to them, either.

For people *not* used to git-svnimport, git-svn also supports:

 git svn init --minimize-url \
  --trunk http://repository-root/foo/trunk \
  --branches http://repository-root/foo/branches \
  --tags http://repository-root/foo/tags
And this is where the new --minimize-url command-line switch
comes in to allow for this behavior to continue working.
---
 git-svn.perl                         |    5 +++--
 t/t9100-git-svn-basic.sh             |    2 +-
 t/t9104-git-svn-follow-parent.sh     |   13 +++++++------
 t/t9105-git-svn-commit-diff.sh       |    2 +-
 t/t9110-git-svn-use-svm-props.sh     |    7 ++++---
 t/t9111-git-svn-use-svnsync-props.sh |    6 +++---
 6 files changed, 19 insertions(+), 16 deletions(-)
diff --git a/git-svn.perl b/git-svn.perl
index 5352470..e38811a 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -80,6 +80,7 @@ my %icv;
 my %init_opts = ( 'template=s' => \$_template, 'shared:s' => \$_shared,
                   'trunk|T=s' => \$_trunk, 'tags|t=s' => \$_tags,
                   'branches|b=s' => \$_branches, 'prefix=s' => \$_prefix,
+                  'minimize-url|m' => \$Git::SVN::_minimize_url,
 		  'no-metadata' => sub { $icv{noMetadata} = 1 },
 		  'use-svm-props' => sub { $icv{useSvmProps} = 1 },
 		  'use-svnsync-props' => sub { $icv{useSvnsyncProps} = 1 },
@@ -820,7 +821,7 @@ use strict;
 use warnings;
 use vars qw/$default_repo_id $default_ref_id $_no_metadata $_follow_parent
             $_repack $_repack_flags $_use_svm_props $_head
-            $_use_svnsync_props $no_reuse_existing/;
+            $_use_svnsync_props $no_reuse_existing $_minimize_url/;
 use Carp qw/croak/;
 use File::Path qw/mkpath/;
 use File::Copy qw/copy/;
@@ -1037,7 +1038,7 @@ sub init_remote_config {
 				     "[svn-remote \"$existing\"]\n";
 		}
 		$self->{repo_id} = $existing;
-	} else {
+	} elsif ($_minimize_url) {
 		my $min_url = Git::SVN::Ra->new($url)->minimize_url;
 		$existing = find_existing_remote($min_url, $r);
 		if ($existing) {
diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
index eb628fe..70c3669 100755
--- a/t/t9100-git-svn-basic.sh
+++ b/t/t9100-git-svn-basic.sh
@@ -229,7 +229,7 @@ test_expect_failure 'exit if init-ing a would clobber a URL' "
 
 test_expect_success \
   'init allows us to connect to another directory in the same repo' "
-        git-svn init -i bar $svnrepo/bar &&
+        git-svn init --minimize-url -i bar $svnrepo/bar &&
         git config --get svn-remote.svn.fetch \
                               '^bar:refs/remotes/bar$' &&
         git config --get svn-remote.svn.fetch \
diff --git a/t/t9104-git-svn-follow-parent.sh b/t/t9104-git-svn-follow-parent.sh
index bd4f366..35aa45c 100755
--- a/t/t9104-git-svn-follow-parent.sh
+++ b/t/t9104-git-svn-follow-parent.sh
@@ -28,7 +28,7 @@ test_expect_success 'initialize repo' "
 	"
 
 test_expect_success 'init and fetch a moved directory' "
-	git-svn init -i thunk $svnrepo/thunk &&
+	git-svn init --minimize-url -i thunk $svnrepo/thunk &&
 	git-svn fetch -i thunk &&
 	test \"\`git-rev-parse --verify refs/remotes/thunk@2\`\" \
            = \"\`git-rev-parse --verify refs/remotes/thunk~1\`\" &&
@@ -68,7 +68,8 @@ test_expect_success 'follow larger parent' "
         echo hi > import/trunk/thunk/bump/thud/file &&
         svn import -m 'import a larger parent' import $svnrepo/larger-parent &&
         svn cp -m 'hi' $svnrepo/larger-parent $svnrepo/another-larger &&
-        git-svn init -i larger $svnrepo/another-larger/trunk/thunk/bump/thud &&
+        git-svn init --minimize-url -i larger \
+          $svnrepo/another-larger/trunk/thunk/bump/thud &&
         git-svn fetch -i larger &&
         git-rev-parse --verify refs/remotes/larger &&
         git-rev-parse --verify \
@@ -90,14 +91,14 @@ test_expect_success 'follow higher-level parent' "
                 cd ..
         svn mkdir -m 'new glob at top level' $svnrepo/glob &&
         svn mv -m 'move blob down a level' $svnrepo/blob $svnrepo/glob/blob &&
-        git-svn init -i blob $svnrepo/glob/blob &&
+        git-svn init --minimize-url -i blob $svnrepo/glob/blob &&
         git-svn fetch -i blob
         "
 
 test_expect_success 'follow deleted directory' "
 	svn mv -m 'bye!' $svnrepo/glob/blob/hi $svnrepo/glob/blob/bye &&
 	svn rm -m 'remove glob' $svnrepo/glob &&
-	git-svn init -i glob $svnrepo/glob &&
+	git-svn init --minimize-url -i glob $svnrepo/glob &&
 	git-svn fetch -i glob &&
 	test \"\`git cat-file blob refs/remotes/glob:blob/bye\`\" = hi &&
 	test \"\`git ls-tree refs/remotes/glob | wc -l \`\" -eq 1
@@ -127,7 +128,7 @@ test_expect_success 'follow-parent avoids deleting relevant info' "
 	  poke native/t/c.t &&
 	  svn commit -m 'reorg test' &&
 	cd .. &&
-	git-svn init -i r9270-t \
+	git-svn init --minimize-url -i r9270-t \
 	  $svnrepo/r9270/trunk/subversion/bindings/swig/perl/native/t &&
 	git-svn fetch -i r9270-t &&
 	test \`git rev-list r9270-t | wc -l\` -eq 2 &&
@@ -137,7 +138,7 @@ test_expect_success 'follow-parent avoids deleting relevant info' "
 
 test_expect_success "track initial change if it was only made to parent" "
 	svn cp -m 'wheee!' $svnrepo/r9270/trunk $svnrepo/r9270/drunk &&
-	git-svn init -i r9270-d \
+	git-svn init --minimize-url -i r9270-d \
 	  $svnrepo/r9270/drunk/subversion/bindings/swig/perl/native/t &&
 	git-svn fetch -i r9270-d &&
 	test \`git rev-list r9270-d | wc -l\` -eq 3 &&
diff --git a/t/t9105-git-svn-commit-diff.sh b/t/t9105-git-svn-commit-diff.sh
index c668dd1..318e172 100755
--- a/t/t9105-git-svn-commit-diff.sh
+++ b/t/t9105-git-svn-commit-diff.sh
@@ -33,7 +33,7 @@ test_expect_success 'test the commit-diff command' "
 
 test_expect_success 'commit-diff to a sub-directory (with git-svn config)' "
 	svn import -m 'sub-directory' import $svnrepo/subdir &&
-	git-svn init $svnrepo/subdir &&
+	git-svn init --minimize-url $svnrepo/subdir &&
 	git-svn fetch &&
 	git-svn commit-diff -r3 '$prev' '$head' &&
 	svn cat $svnrepo/subdir/readme > readme.2 &&
diff --git a/t/t9110-git-svn-use-svm-props.sh b/t/t9110-git-svn-use-svm-props.sh
index 9db0d8f..59e17f2 100755
--- a/t/t9110-git-svn-use-svm-props.sh
+++ b/t/t9110-git-svn-use-svm-props.sh
@@ -9,9 +9,10 @@ test_description='git-svn useSvmProps test'
 
 test_expect_success 'load svm repo' "
 	svnadmin load -q $rawsvnrepo < ../t9110/svm.dump &&
-	git-svn init -R arr -i bar $svnrepo/mirror/arr &&
-	git-svn init -R argh -i dir $svnrepo/mirror/argh &&
-	git-svn init -R argh -i e $svnrepo/mirror/argh/a/b/c/d/e &&
+	git-svn init --minimize-url -R arr -i bar $svnrepo/mirror/arr &&
+	git-svn init --minimize-url -R argh -i dir $svnrepo/mirror/argh &&
+	git-svn init --minimize-url -R argh -i e \
+	  $svnrepo/mirror/argh/a/b/c/d/e &&
 	git-config svn.useSvmProps true &&
 	git-svn fetch --all
 	"
diff --git a/t/t9111-git-svn-use-svnsync-props.sh b/t/t9111-git-svn-use-svnsync-props.sh
index 483d7f8..e523214 100755
--- a/t/t9111-git-svn-use-svnsync-props.sh
+++ b/t/t9111-git-svn-use-svnsync-props.sh
@@ -9,9 +9,9 @@ test_description='git-svn useSvnsyncProps test'
 
 test_expect_success 'load svnsync repo' "
 	svnadmin load -q $rawsvnrepo < ../t9111/svnsync.dump &&
-	git-svn init -R arr -i bar $svnrepo/bar &&
-	git-svn init -R argh -i dir $svnrepo/dir &&
-	git-svn init -R argh -i e $svnrepo/dir/a/b/c/d/e &&
+	git-svn init --minimize-url -R arr -i bar $svnrepo/bar &&
+	git-svn init --minimize-url -R argh -i dir $svnrepo/dir &&
+	git-svn init --minimize-url -R argh -i e $svnrepo/dir/a/b/c/d/e &&
 	git-config svn.useSvnsyncProps true &&
 	git-svn fetch --all
 	"
-- 
Eric Wong

← back to recent threads