threads / patch / 15560

patchgit-svn: do a partial rebuild if rev_map is out-of-date

Subject: [PATCH] git-svn: do a partial rebuild if rev_map is out-of-date

## tl;dr

5 messages between Sep 17, 2008 and Sep 22, 2008. Diffs are folded; open one to read it.

replies: 4people: 2as markdown or json

Deskin Miller· Sep 17, 2008, 03:13 UTC · lore

Suppose you're using git-svn to work with a certain SVN repository. Since you don't like 'git-svn fetch' to take forever, and you don't want to accidentally interrupt it and end up corrupting your repository, you set up a remote Git repository to mirror the SVN repository, which does its own 'git-svn fetch' on a cronjob; now you can 'git-fetch' from the Git mirror into your local repository, and still dcommit to SVN when you have changes to push.

After you do this, though, git-svn will get very confused if you ever try to do 'git-svn fetch' in your local repository again, since its rev_map will differ from the branch's head, and it will be unable to fetch new commits from SVN because of the metadata conflict. But all the necessary metadata are there in the Git commit message; git-svn already knows how to rebuild rev_map files that get blown away, by using the metadata.

This commit will have git-svn do a partial rebuild of the rev_map to match the true state of the branch, if it ever is used to fetch again.

This will only work for projects not using either noMetadata or useSvmProps configuration options; if you are using these options, git-svn will fall back to the previous behaviour.

Signed-off-by: Deskin Miller <deskinm@umich.edu>
---
It's possible that this can be extended to work with useSvmProps, but I don't
know how to do so.
 git-svn.perl |   23 ++++++++++++++++++-----
 1 files changed, 18 insertions(+), 5 deletions(-)
Show changes to git-svn.perl +18 −5
diff --git a/git-svn.perl b/git-svn.perl
index 88066c9..2e7a8ce 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -2626,9 +2626,9 @@ sub rebuild_from_rev_db {
 sub rebuild {
 	my ($self) = @_;
 	my $map_path = $self->map_path;
-	return if (-e $map_path && ! -z $map_path);
+	my $partial = (-e $map_path && ! -z $map_path);
 	return unless ::verify_ref($self->refname.'^0');
-	if ($self->use_svm_props || $self->no_metadata) {
+	if (!$partial && ($self->use_svm_props || $self->no_metadata)) {
 		my $rev_db = $self->rev_db_path;
 		$self->rebuild_from_rev_db($rev_db);
 		if ($self->use_svm_props) {
@@ -2638,10 +2638,12 @@ sub rebuild {
 		$self->unlink_rev_db_symlink;
 		return;
 	}
-	print "Rebuilding $map_path ...\n";
+	print "Rebuilding $map_path ...\n" if (!$partial);
+	my ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) :
+		(undef, undef));
 	my ($log, $ctx) =
 	    command_output_pipe(qw/rev-list --pretty=raw --no-color --reverse/,
-	                        $self->refname, '--');
+				($head ? "^$head" : ""), $self->refname, '--');
 	my $metadata_url = $self->metadata_url;
 	remove_username($metadata_url);
 	my $svn_uuid = $self->ra_uuid;
@@ -2664,12 +2666,17 @@ sub rebuild {
 		    ($metadata_url && $url && ($url ne $metadata_url))) {
 			next;
 		}
+		if ($partial && $head) {
+			print "Partial-rebuilding $map_path ...\n";
+			print "Currently at $base_rev = $head\n";
+			$head = undef;
+		}
 
 		$self->rev_map_set($rev, $c);
 		print "r$rev = $c\n";
 	}
 	command_close_pipe($log, $ctx);
-	print "Done rebuilding $map_path\n";
+	print "Done rebuilding $map_path\n" if (!$partial || !$head);
 	my $rev_db_path = $self->rev_db_path;
 	if (-f $self->rev_db_path) {
 		unlink $self->rev_db_path or croak "unlink: $!";
@@ -2809,6 +2816,12 @@ sub rev_map_set {
 sub rev_map_max {
 	my ($self, $want_commit) = @_;
 	$self->rebuild;
+	my ($r, $c) = $self->rev_map_max_norebuild($want_commit);
+	$want_commit ? ($r, $c) : $r;
+}
+
+sub rev_map_max_norebuild {
+	my ($self, $want_commit) = @_;
 	my $map_path = $self->map_path;
 	stat $map_path or return $want_commit ? (0, undef) : 0;
 	sysopen(my $fh, $map_path, O_RDONLY) or croak "open: $!";
-- 
1.5.4.3
Eric Wong· Sep 18, 2008, 06:38 UTC · re: Deskin Miller · lore

Re: [PATCH] git-svn: do a partial rebuild if rev_map is out-of-date

Deskin Miller <deskinm@umich.edu> wrote:
Show 9 quoted lines
> This commit will have git-svn do a partial rebuild of the rev_map to
> match the true state of the branch, if it ever is used to fetch again.
> 
> This will only work for projects not using either noMetadata or
> useSvmProps configuration options; if you are using these options,
> git-svn will fall back to the previous behaviour.
> 
> Signed-off-by: Deskin Miller <deskinm@umich.edu>
> ---
Hi Deskin,
This seems to break the following test case for me:
*** t9107-git-svn-migrate.sh ***
*   ok 1: setup old-looking metadata
*   ok 2: git-svn-HEAD is a real HEAD
*   ok 3: initialize old-style (v0) git svn layout
*   ok 4: initialize a multi-repository repo
*   ok 5: multi-fetch works on partial urls + paths
*   ok 6: migrate --minimize on old inited layout
* FAIL 7: .rev_db auto-converted to .rev_map.UUID

I haven't had time to diagnose it. Also, can you add a test that demonstrates this functionality (and ensures things keeps working when future work is done on git-svn?)

Thanks.
-- 
Eric Wong
Deskin Miller· Sep 21, 2008, 02:45 UTC · re: Eric Wong · lore

[PATCH v2 0/1] git-svn: testcase for partial rebuild

>From 45c9876a04ff0aac141300dd10fca50d6db30522 Mon Sep 17 00:00:00 2001
From: Deskin Miler <deskinm@umich.edu>
Date: Thu, 18 Sep 2008 17:55:14 -0400
Signed-off-by: Deskin Miller <deskinm@umich.edu>
---
On Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:
Show 14 quoted lines
> This seems to break the following test case for me:
> 
> *** t9107-git-svn-migrate.sh ***
> *   ok 1: setup old-looking metadata
> *   ok 2: git-svn-HEAD is a real HEAD
> *   ok 3: initialize old-style (v0) git svn layout
> *   ok 4: initialize a multi-repository repo
> *   ok 5: multi-fetch works on partial urls + paths
> *   ok 6: migrate --minimize on old inited layout
> * FAIL 7: .rev_db auto-converted to .rev_map.UUID
> 
> I haven't had time to diagnose it.  Also, can you add a test that
> demonstrates this functionality (and ensures things keeps working when
> future work is done on git-svn?)

Thanks for the response; I had a bug in my Perl that my testing hadn't caught. Gave me an opportunity to learn how the git testsuites work!

This testcase fails for me when applied to master, and passes with patch 1/1 in the series.

 t/t9126-git-svn-partial-rebuild.sh |   53 ++++++++++++++++++++++++++++++++++++
 1 files changed, 53 insertions(+), 0 deletions(-)
 create mode 100755 t/t9126-git-svn-partial-rebuild.sh
Show changes to t/t9126-git-svn-partial-rebuild.sh +53 −0
diff --git a/t/t9126-git-svn-partial-rebuild.sh b/t/t9126-git-svn-partial-rebuild.sh
new file mode 100755
index 0000000..9a94866
--- /dev/null
+++ b/t/t9126-git-svn-partial-rebuild.sh
@@ -0,0 +1,53 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 Deskin Miller
+#
+
+test_description='git svn partial-rebuild tests'
+. ./lib-git-svn.sh
+
+test_expect_success \
+    'initialize svnrepo' '
+	mkdir import &&
+	cd import &&
+	mkdir trunk branches tags &&
+	cd trunk &&
+	echo foo > foo &&
+	cd .. &&
+	svn import -m "import for git-svn" . "$svnrepo" >/dev/null &&
+	svn copy "$svnrepo"/trunk "$svnrepo"/branches/a \
+		-m "created branch a" &&
+	cd .. &&
+	rm -rf import &&
+	svn co "$svnrepo"/trunk trunk &&
+	cd trunk &&
+	echo bar >> foo &&
+	svn ci -m "updated trunk" &&
+	cd .. &&
+	svn co "$svnrepo"/branches/a a &&
+	cd a &&
+	echo baz >> a &&
+	svn add a &&
+	svn ci -m "updated a" &&
+	cd .. &&
+	git svn init --stdlayout "$svnrepo"'
+
+test_expect_success 'import an early SVN revision into git' \
+	'git svn fetch -r1:2'
+
+test_expect_success 'make full git mirror of SVN' \
+	'mkdir mirror &&
+	cd mirror &&
+	git init &&
+	git svn init --stdlayout "$svnrepo" &&
+	git svn fetch &&
+	cd ..'
+
+test_expect_success 'fetch from git mirror and partial-rebuild' \
+	'git config --add remote.origin.url "file://$PWD/mirror/.git" &&
+	git config --add remote.origin.fetch refs/remotes/*:refs/remotes/* &&
+	git fetch origin &&
+	git svn fetch
+	'
+
+test_done
-- 
1.6.0.2.GIT
Eric Wong· Sep 22, 2008, 04:12 UTC · re: Deskin Miller · lore

Re: [PATCH v2 0/1] git-svn: testcase for partial rebuild

Deskin Miller <deskinm@umich.edu> wrote:
Show 22 quoted lines
> ---
> On Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:
> > This seems to break the following test case for me:
> > 
> > *** t9107-git-svn-migrate.sh ***
> > *   ok 1: setup old-looking metadata
> > *   ok 2: git-svn-HEAD is a real HEAD
> > *   ok 3: initialize old-style (v0) git svn layout
> > *   ok 4: initialize a multi-repository repo
> > *   ok 5: multi-fetch works on partial urls + paths
> > *   ok 6: migrate --minimize on old inited layout
> > * FAIL 7: .rev_db auto-converted to .rev_map.UUID
> > 
> > I haven't had time to diagnose it.  Also, can you add a test that
> > demonstrates this functionality (and ensures things keeps working when
> > future work is done on git-svn?)
> 
> Thanks for the response; I had a bug in my Perl that my testing hadn't caught.
> Gave me an opportunity to learn how the git testsuites work!
> 
> This testcase fails for me when applied to master, and passes with patch 1/1 in
> the series.
Thanks Deskin, this series is
  Acked-by: Eric Wong <normalperson@yhbt.net>
Junio: I would apply the patch series in reverse order to not break
tests on potential bisections, however.
-- 
Eric Wong
Deskin Miller· Sep 21, 2008, 02:45 UTC · re: Eric Wong · lore

[PATCH v2 1/1] git-svn: do a partial rebuild if rev_map is out-of-date

>From 11ef3a043bb9cf89cd87d7030c684a1ee566a87a Mon Sep 17 00:00:00 2001
From: Deskin Miller <deskinm@umich.edu>
Date: Mon, 15 Sep 2008 21:12:58 -0400

Suppose you're using git-svn to work with a certain SVN repository. Since you don't like 'git-svn fetch' to take forever, and you don't want to accidentally interrupt it and end up corrupting your repository, you set up a remote Git repository to mirror the SVN repository, which does its own 'git-svn fetch' on a cronjob; now you can 'git-fetch' from the Git mirror into your local repository, and still dcommit to SVN when you have changes to push.

After you do this, though, git-svn will get very confused if you ever try to do 'git-svn fetch' in your local repository again, since its rev_map will differ from the branch's head, and it will be unable to fetch new commits from SVN because of the metadata conflict. But all the necessary metadata are there in the Git commit message; git-svn already knows how to rebuild rev_map files that get blown away, by using the metadata.

This commit will have git-svn do a partial rebuild of the rev_map to match the true state of the branch, if it ever is used to fetch again.

This will only work for projects not using either noMetadata or useSvmProps configuration options; if you are using these options, git-svn will fall back to the previous behaviour.

Signed-off-by: Deskin Miller <deskinm@umich.edu>
---
On Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:
Show 14 quoted lines
> This seems to break the following test case for me:
> 
> *** t9107-git-svn-migrate.sh ***
> *   ok 1: setup old-looking metadata
> *   ok 2: git-svn-HEAD is a real HEAD
> *   ok 3: initialize old-style (v0) git svn layout
> *   ok 4: initialize a multi-repository repo
> *   ok 5: multi-fetch works on partial urls + paths
> *   ok 6: migrate --minimize on old inited layout
> * FAIL 7: .rev_db auto-converted to .rev_map.UUID
> 
> I haven't had time to diagnose it.  Also, can you add a test that
> demonstrates this functionality (and ensures things keeps working when
> future work is done on git-svn?)
Here is the new, fixed version.  It allows the test in patch 0/1 to pass.
 git-svn.perl |   24 +++++++++++++++++++-----
 1 files changed, 19 insertions(+), 5 deletions(-)
Show changes to git-svn.perl +19 −5
diff --git a/git-svn.perl b/git-svn.perl
index af8279a..80a5728 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -2626,9 +2626,9 @@ sub rebuild_from_rev_db {
 sub rebuild {
 	my ($self) = @_;
 	my $map_path = $self->map_path;
-	return if (-e $map_path && ! -z $map_path);
+	my $partial = (-e $map_path && ! -z $map_path);
 	return unless ::verify_ref($self->refname.'^0');
-	if ($self->use_svm_props || $self->no_metadata) {
+	if (!$partial && ($self->use_svm_props || $self->no_metadata)) {
 		my $rev_db = $self->rev_db_path;
 		$self->rebuild_from_rev_db($rev_db);
 		if ($self->use_svm_props) {
@@ -2638,10 +2638,13 @@ sub rebuild {
 		$self->unlink_rev_db_symlink;
 		return;
 	}
-	print "Rebuilding $map_path ...\n";
+	print "Rebuilding $map_path ...\n" if (!$partial);
+	my ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) :
+		(undef, undef));
 	my ($log, $ctx) =
 	    command_output_pipe(qw/rev-list --pretty=raw --no-color --reverse/,
-	                        $self->refname, '--');
+				($head ? "$head.." : "") . $self->refname,
+				'--');
 	my $metadata_url = $self->metadata_url;
 	remove_username($metadata_url);
 	my $svn_uuid = $self->ra_uuid;
@@ -2664,12 +2667,17 @@ sub rebuild {
 		    ($metadata_url && $url && ($url ne $metadata_url))) {
 			next;
 		}
+		if ($partial && $head) {
+			print "Partial-rebuilding $map_path ...\n";
+			print "Currently at $base_rev = $head\n";
+			$head = undef;
+		}
 
 		$self->rev_map_set($rev, $c);
 		print "r$rev = $c\n";
 	}
 	command_close_pipe($log, $ctx);
-	print "Done rebuilding $map_path\n";
+	print "Done rebuilding $map_path\n" if (!$partial || !$head);
 	my $rev_db_path = $self->rev_db_path;
 	if (-f $self->rev_db_path) {
 		unlink $self->rev_db_path or croak "unlink: $!";
@@ -2809,6 +2817,12 @@ sub rev_map_set {
 sub rev_map_max {
 	my ($self, $want_commit) = @_;
 	$self->rebuild;
+	my ($r, $c) = $self->rev_map_max_norebuild($want_commit);
+	$want_commit ? ($r, $c) : $r;
+}
+
+sub rev_map_max_norebuild {
+	my ($self, $want_commit) = @_;
 	my $map_path = $self->map_path;
 	stat $map_path or return $want_commit ? (0, undef) : 0;
 	sysopen(my $fh, $map_path, O_RDONLY) or croak "open: $!";
-- 
1.6.0.2.GIT

← back to recent threads