{"thread":{"id":"15560","subject":"[PATCH] git-svn: do a partial rebuild if rev_map is out-of-date","startedAt":"2008-09-17T03:13:04Z","lastAt":"2008-09-22T04:12:34Z","messageCount":5,"participants":["Deskin Miller","Eric Wong"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"90895","messageId":"20080917031304.GA2505@riemann.deskinm.fdns.net","threadId":"15560","inReplyTo":null,"subject":"[PATCH] git-svn: do a partial rebuild if rev_map is out-of-date","fromName":"Deskin Miller","fromEmail":"deskinm@umich.edu","sentAt":"2008-09-17T03:13:04Z","receivedAt":"2008-09-17T03:13:04Z","isPatch":true,"sender":{"key":"deskinm@umich.edu","avatar":"https://gravatar.com/avatar/d340a0e612cdf0a79535c71863c0b4c535e9aba63b42032226ae903e638b64f9?d=mp&s=160"},"body":"Suppose you're using git-svn to work with a certain SVN repository.\nSince you don't like 'git-svn fetch' to take forever, and you don't want\nto accidentally interrupt it and end up corrupting your repository, you\nset up a remote Git repository to mirror the SVN repository, which does\nits own 'git-svn fetch' on a cronjob; now you can 'git-fetch' from the\nGit mirror into your local repository, and still dcommit to SVN when you\nhave changes to push.\n\nAfter you do this, though, git-svn will get very confused if you ever\ntry to do 'git-svn fetch' in your local repository again, since its\nrev_map will differ from the branch's head, and it will be unable to\nfetch new commits from SVN because of the metadata conflict.  But all\nthe necessary metadata are there in the Git commit message; git-svn\nalready knows how to rebuild rev_map files that get blown away, by\nusing the metadata.\n\nThis commit will have git-svn do a partial rebuild of the rev_map to\nmatch the true state of the branch, if it ever is used to fetch again.\n\nThis will only work for projects not using either noMetadata or\nuseSvmProps configuration options; if you are using these options,\ngit-svn will fall back to the previous behaviour.\n\nSigned-off-by: Deskin Miller <deskinm@umich.edu>\n---\nIt's possible that this can be extended to work with useSvmProps, but I don't\nknow how to do so.\n\n git-svn.perl |   23 ++++++++++++++++++-----\n 1 files changed, 18 insertions(+), 5 deletions(-)\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex 88066c9..2e7a8ce 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -2626,9 +2626,9 @@ sub rebuild_from_rev_db {\n sub rebuild {\n \tmy ($self) = @_;\n \tmy $map_path = $self->map_path;\n-\treturn if (-e $map_path && ! -z $map_path);\n+\tmy $partial = (-e $map_path && ! -z $map_path);\n \treturn unless ::verify_ref($self->refname.'^0');\n-\tif ($self->use_svm_props || $self->no_metadata) {\n+\tif (!$partial && ($self->use_svm_props || $self->no_metadata)) {\n \t\tmy $rev_db = $self->rev_db_path;\n \t\t$self->rebuild_from_rev_db($rev_db);\n \t\tif ($self->use_svm_props) {\n@@ -2638,10 +2638,12 @@ sub rebuild {\n \t\t$self->unlink_rev_db_symlink;\n \t\treturn;\n \t}\n-\tprint \"Rebuilding $map_path ...\\n\";\n+\tprint \"Rebuilding $map_path ...\\n\" if (!$partial);\n+\tmy ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) :\n+\t\t(undef, undef));\n \tmy ($log, $ctx) =\n \t    command_output_pipe(qw/rev-list --pretty=raw --no-color --reverse/,\n-\t                        $self->refname, '--');\n+\t\t\t\t($head ? \"^$head\" : \"\"), $self->refname, '--');\n \tmy $metadata_url = $self->metadata_url;\n \tremove_username($metadata_url);\n \tmy $svn_uuid = $self->ra_uuid;\n@@ -2664,12 +2666,17 @@ sub rebuild {\n \t\t    ($metadata_url && $url && ($url ne $metadata_url))) {\n \t\t\tnext;\n \t\t}\n+\t\tif ($partial && $head) {\n+\t\t\tprint \"Partial-rebuilding $map_path ...\\n\";\n+\t\t\tprint \"Currently at $base_rev = $head\\n\";\n+\t\t\t$head = undef;\n+\t\t}\n \n \t\t$self->rev_map_set($rev, $c);\n \t\tprint \"r$rev = $c\\n\";\n \t}\n \tcommand_close_pipe($log, $ctx);\n-\tprint \"Done rebuilding $map_path\\n\";\n+\tprint \"Done rebuilding $map_path\\n\" if (!$partial || !$head);\n \tmy $rev_db_path = $self->rev_db_path;\n \tif (-f $self->rev_db_path) {\n \t\tunlink $self->rev_db_path or croak \"unlink: $!\";\n@@ -2809,6 +2816,12 @@ sub rev_map_set {\n sub rev_map_max {\n \tmy ($self, $want_commit) = @_;\n \t$self->rebuild;\n+\tmy ($r, $c) = $self->rev_map_max_norebuild($want_commit);\n+\t$want_commit ? ($r, $c) : $r;\n+}\n+\n+sub rev_map_max_norebuild {\n+\tmy ($self, $want_commit) = @_;\n \tmy $map_path = $self->map_path;\n \tstat $map_path or return $want_commit ? (0, undef) : 0;\n \tsysopen(my $fh, $map_path, O_RDONLY) or croak \"open: $!\";\n-- \n1.5.4.3\n"},{"id":"90999","messageId":"20080918063754.GA13328@untitled","threadId":"15560","inReplyTo":"20080917031304.GA2505@riemann.deskinm.fdns.net","subject":"Re: [PATCH] git-svn: do a partial rebuild if rev_map is out-of-date","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-09-18T06:38:04Z","receivedAt":"2008-09-18T06:38:04Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Deskin Miller <deskinm@umich.edu> wrote:\n> This commit will have git-svn do a partial rebuild of the rev_map to\n> match the true state of the branch, if it ever is used to fetch again.\n> \n> This will only work for projects not using either noMetadata or\n> useSvmProps configuration options; if you are using these options,\n> git-svn will fall back to the previous behaviour.\n> \n> Signed-off-by: Deskin Miller <deskinm@umich.edu>\n> ---\n\nHi Deskin,\n\nThis seems to break the following test case for me:\n\n*** t9107-git-svn-migrate.sh ***\n*   ok 1: setup old-looking metadata\n*   ok 2: git-svn-HEAD is a real HEAD\n*   ok 3: initialize old-style (v0) git svn layout\n*   ok 4: initialize a multi-repository repo\n*   ok 5: multi-fetch works on partial urls + paths\n*   ok 6: migrate --minimize on old inited layout\n* FAIL 7: .rev_db auto-converted to .rev_map.UUID\n\nI haven't had time to diagnose it.  Also, can you add a test that\ndemonstrates this functionality (and ensures things keeps working when\nfuture work is done on git-svn?)\n\nThanks.\n\n-- \nEric Wong\n"},{"id":"91210","messageId":"20080921024538.GB2505@riemann.deskinm.fdns.net","threadId":"15560","inReplyTo":"20080918063754.GA13328@untitled","subject":"[PATCH v2 0/1] git-svn: testcase for partial rebuild","fromName":"Deskin Miller","fromEmail":"deskinm@umich.edu","sentAt":"2008-09-21T02:45:38Z","receivedAt":"2008-09-21T02:45:38Z","isPatch":true,"sender":{"key":"deskinm@umich.edu","avatar":"https://gravatar.com/avatar/d340a0e612cdf0a79535c71863c0b4c535e9aba63b42032226ae903e638b64f9?d=mp&s=160"},"body":">From 45c9876a04ff0aac141300dd10fca50d6db30522 Mon Sep 17 00:00:00 2001\nFrom: Deskin Miler <deskinm@umich.edu>\nDate: Thu, 18 Sep 2008 17:55:14 -0400\n\nSigned-off-by: Deskin Miller <deskinm@umich.edu>\n---\nOn Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:\n> This seems to break the following test case for me:\n> \n> *** t9107-git-svn-migrate.sh ***\n> *   ok 1: setup old-looking metadata\n> *   ok 2: git-svn-HEAD is a real HEAD\n> *   ok 3: initialize old-style (v0) git svn layout\n> *   ok 4: initialize a multi-repository repo\n> *   ok 5: multi-fetch works on partial urls + paths\n> *   ok 6: migrate --minimize on old inited layout\n> * FAIL 7: .rev_db auto-converted to .rev_map.UUID\n> \n> I haven't had time to diagnose it.  Also, can you add a test that\n> demonstrates this functionality (and ensures things keeps working when\n> future work is done on git-svn?)\n\nThanks for the response; I had a bug in my Perl that my testing hadn't caught.\nGave me an opportunity to learn how the git testsuites work!\n\nThis testcase fails for me when applied to master, and passes with patch 1/1 in\nthe series.\n\n t/t9126-git-svn-partial-rebuild.sh |   53 ++++++++++++++++++++++++++++++++++++\n 1 files changed, 53 insertions(+), 0 deletions(-)\n create mode 100755 t/t9126-git-svn-partial-rebuild.sh\n\ndiff --git a/t/t9126-git-svn-partial-rebuild.sh b/t/t9126-git-svn-partial-rebuild.sh\nnew file mode 100755\nindex 0000000..9a94866\n--- /dev/null\n+++ b/t/t9126-git-svn-partial-rebuild.sh\n@@ -0,0 +1,53 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2008 Deskin Miller\n+#\n+\n+test_description='git svn partial-rebuild tests'\n+. ./lib-git-svn.sh\n+\n+test_expect_success \\\n+    'initialize svnrepo' '\n+\tmkdir import &&\n+\tcd import &&\n+\tmkdir trunk branches tags &&\n+\tcd trunk &&\n+\techo foo > foo &&\n+\tcd .. &&\n+\tsvn import -m \"import for git-svn\" . \"$svnrepo\" >/dev/null &&\n+\tsvn copy \"$svnrepo\"/trunk \"$svnrepo\"/branches/a \\\n+\t\t-m \"created branch a\" &&\n+\tcd .. &&\n+\trm -rf import &&\n+\tsvn co \"$svnrepo\"/trunk trunk &&\n+\tcd trunk &&\n+\techo bar >> foo &&\n+\tsvn ci -m \"updated trunk\" &&\n+\tcd .. &&\n+\tsvn co \"$svnrepo\"/branches/a a &&\n+\tcd a &&\n+\techo baz >> a &&\n+\tsvn add a &&\n+\tsvn ci -m \"updated a\" &&\n+\tcd .. &&\n+\tgit svn init --stdlayout \"$svnrepo\"'\n+\n+test_expect_success 'import an early SVN revision into git' \\\n+\t'git svn fetch -r1:2'\n+\n+test_expect_success 'make full git mirror of SVN' \\\n+\t'mkdir mirror &&\n+\tcd mirror &&\n+\tgit init &&\n+\tgit svn init --stdlayout \"$svnrepo\" &&\n+\tgit svn fetch &&\n+\tcd ..'\n+\n+test_expect_success 'fetch from git mirror and partial-rebuild' \\\n+\t'git config --add remote.origin.url \"file://$PWD/mirror/.git\" &&\n+\tgit config --add remote.origin.fetch refs/remotes/*:refs/remotes/* &&\n+\tgit fetch origin &&\n+\tgit svn fetch\n+\t'\n+\n+test_done\n-- \n1.6.0.2.GIT\n"},{"id":"91211","messageId":"20080921024550.GC2505@riemann.deskinm.fdns.net","threadId":"15560","inReplyTo":"20080918063754.GA13328@untitled","subject":"[PATCH v2 1/1] git-svn: do a partial rebuild if rev_map is out-of-date","fromName":"Deskin Miller","fromEmail":"deskinm@umich.edu","sentAt":"2008-09-21T02:45:50Z","receivedAt":"2008-09-21T02:45:50Z","isPatch":true,"sender":{"key":"deskinm@umich.edu","avatar":"https://gravatar.com/avatar/d340a0e612cdf0a79535c71863c0b4c535e9aba63b42032226ae903e638b64f9?d=mp&s=160"},"body":">From 11ef3a043bb9cf89cd87d7030c684a1ee566a87a Mon Sep 17 00:00:00 2001\nFrom: Deskin Miller <deskinm@umich.edu>\nDate: Mon, 15 Sep 2008 21:12:58 -0400\n\nSuppose you're using git-svn to work with a certain SVN repository.\nSince you don't like 'git-svn fetch' to take forever, and you don't want\nto accidentally interrupt it and end up corrupting your repository, you\nset up a remote Git repository to mirror the SVN repository, which does\nits own 'git-svn fetch' on a cronjob; now you can 'git-fetch' from the\nGit mirror into your local repository, and still dcommit to SVN when you\nhave changes to push.\n\nAfter you do this, though, git-svn will get very confused if you ever\ntry to do 'git-svn fetch' in your local repository again, since its\nrev_map will differ from the branch's head, and it will be unable to\nfetch new commits from SVN because of the metadata conflict.  But all\nthe necessary metadata are there in the Git commit message; git-svn\nalready knows how to rebuild rev_map files that get blown away, by\nusing the metadata.\n\nThis commit will have git-svn do a partial rebuild of the rev_map to\nmatch the true state of the branch, if it ever is used to fetch again.\n\nThis will only work for projects not using either noMetadata or\nuseSvmProps configuration options; if you are using these options,\ngit-svn will fall back to the previous behaviour.\n\nSigned-off-by: Deskin Miller <deskinm@umich.edu>\n---\nOn Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:\n> This seems to break the following test case for me:\n> \n> *** t9107-git-svn-migrate.sh ***\n> *   ok 1: setup old-looking metadata\n> *   ok 2: git-svn-HEAD is a real HEAD\n> *   ok 3: initialize old-style (v0) git svn layout\n> *   ok 4: initialize a multi-repository repo\n> *   ok 5: multi-fetch works on partial urls + paths\n> *   ok 6: migrate --minimize on old inited layout\n> * FAIL 7: .rev_db auto-converted to .rev_map.UUID\n> \n> I haven't had time to diagnose it.  Also, can you add a test that\n> demonstrates this functionality (and ensures things keeps working when\n> future work is done on git-svn?)\n\nHere is the new, fixed version.  It allows the test in patch 0/1 to pass.\n\n git-svn.perl |   24 +++++++++++++++++++-----\n 1 files changed, 19 insertions(+), 5 deletions(-)\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex af8279a..80a5728 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -2626,9 +2626,9 @@ sub rebuild_from_rev_db {\n sub rebuild {\n \tmy ($self) = @_;\n \tmy $map_path = $self->map_path;\n-\treturn if (-e $map_path && ! -z $map_path);\n+\tmy $partial = (-e $map_path && ! -z $map_path);\n \treturn unless ::verify_ref($self->refname.'^0');\n-\tif ($self->use_svm_props || $self->no_metadata) {\n+\tif (!$partial && ($self->use_svm_props || $self->no_metadata)) {\n \t\tmy $rev_db = $self->rev_db_path;\n \t\t$self->rebuild_from_rev_db($rev_db);\n \t\tif ($self->use_svm_props) {\n@@ -2638,10 +2638,13 @@ sub rebuild {\n \t\t$self->unlink_rev_db_symlink;\n \t\treturn;\n \t}\n-\tprint \"Rebuilding $map_path ...\\n\";\n+\tprint \"Rebuilding $map_path ...\\n\" if (!$partial);\n+\tmy ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) :\n+\t\t(undef, undef));\n \tmy ($log, $ctx) =\n \t    command_output_pipe(qw/rev-list --pretty=raw --no-color --reverse/,\n-\t                        $self->refname, '--');\n+\t\t\t\t($head ? \"$head..\" : \"\") . $self->refname,\n+\t\t\t\t'--');\n \tmy $metadata_url = $self->metadata_url;\n \tremove_username($metadata_url);\n \tmy $svn_uuid = $self->ra_uuid;\n@@ -2664,12 +2667,17 @@ sub rebuild {\n \t\t    ($metadata_url && $url && ($url ne $metadata_url))) {\n \t\t\tnext;\n \t\t}\n+\t\tif ($partial && $head) {\n+\t\t\tprint \"Partial-rebuilding $map_path ...\\n\";\n+\t\t\tprint \"Currently at $base_rev = $head\\n\";\n+\t\t\t$head = undef;\n+\t\t}\n \n \t\t$self->rev_map_set($rev, $c);\n \t\tprint \"r$rev = $c\\n\";\n \t}\n \tcommand_close_pipe($log, $ctx);\n-\tprint \"Done rebuilding $map_path\\n\";\n+\tprint \"Done rebuilding $map_path\\n\" if (!$partial || !$head);\n \tmy $rev_db_path = $self->rev_db_path;\n \tif (-f $self->rev_db_path) {\n \t\tunlink $self->rev_db_path or croak \"unlink: $!\";\n@@ -2809,6 +2817,12 @@ sub rev_map_set {\n sub rev_map_max {\n \tmy ($self, $want_commit) = @_;\n \t$self->rebuild;\n+\tmy ($r, $c) = $self->rev_map_max_norebuild($want_commit);\n+\t$want_commit ? ($r, $c) : $r;\n+}\n+\n+sub rev_map_max_norebuild {\n+\tmy ($self, $want_commit) = @_;\n \tmy $map_path = $self->map_path;\n \tstat $map_path or return $want_commit ? (0, undef) : 0;\n \tsysopen(my $fh, $map_path, O_RDONLY) or croak \"open: $!\";\n-- \n1.6.0.2.GIT\n"},{"id":"91277","messageId":"20080922041128.GA23848@untitled","threadId":"15560","inReplyTo":"20080921024538.GB2505@riemann.deskinm.fdns.net","subject":"Re: [PATCH v2 0/1] git-svn: testcase for partial rebuild","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-09-22T04:12:34Z","receivedAt":"2008-09-22T04:12:34Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Deskin Miller <deskinm@umich.edu> wrote:\n> ---\n> On Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:\n> > This seems to break the following test case for me:\n> > \n> > *** t9107-git-svn-migrate.sh ***\n> > *   ok 1: setup old-looking metadata\n> > *   ok 2: git-svn-HEAD is a real HEAD\n> > *   ok 3: initialize old-style (v0) git svn layout\n> > *   ok 4: initialize a multi-repository repo\n> > *   ok 5: multi-fetch works on partial urls + paths\n> > *   ok 6: migrate --minimize on old inited layout\n> > * FAIL 7: .rev_db auto-converted to .rev_map.UUID\n> > \n> > I haven't had time to diagnose it.  Also, can you add a test that\n> > demonstrates this functionality (and ensures things keeps working when\n> > future work is done on git-svn?)\n> \n> Thanks for the response; I had a bug in my Perl that my testing hadn't caught.\n> Gave me an opportunity to learn how the git testsuites work!\n> \n> This testcase fails for me when applied to master, and passes with patch 1/1 in\n> the series.\n\nThanks Deskin, this series is\n  Acked-by: Eric Wong <normalperson@yhbt.net>\n\nJunio: I would apply the patch series in reverse order to not break\ntests on potential bisections, however.\n\n-- \nEric Wong\n"}]}