{"thread":{"id":"13426","subject":"[PATCH] Teach git-svn how to catch up with its tracking branches","startedAt":"2008-05-08T01:39:56Z","lastAt":"2008-05-11T08:27:27Z","messageCount":16,"participants":["Steven Grimm","Junio C Hamano","Chris Shoemaker","Asheesh Laroia","Karl Hasselström","Eric Wong"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"76339","messageId":"20080508013956.GA24956@midwinter.com","threadId":"13426","inReplyTo":null,"subject":"[PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2008-05-08T01:39:56Z","receivedAt":"2008-05-08T01:39:56Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"In environments where a lot of people are sharing an svn repository using\ngit-svn, everyone has identical, but individually maintained, tracking\nbranches. If the svn repository is very active, it can take a while to\nrun \"git svn fetch\" (which has to individually construct each revision\nby querying the svn server). It's much faster to run \"git fetch\" against\nanother git-svn repository to grab the exact same git revisions you'd get\nfrom \"git svn fetch\". But until now, git-svn was confused by this because\nit didn't know how to incrementally rebuild its map of revision IDs.\nThe only choice was to completely remove the map file and rebuild it\nfrom scratch, possibly a lengthy operation when there's a lot of history.\n\nWith this change, git-svn will try to do an incremental update of its\nrevision map if it sees that its tracking branch has svn revisions that\naren't in the map yet.\n\nSigned-off-by: Steven Grimm <koreth@midwinter.com>\n---\n git-svn.perl |   62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 files changed, 58 insertions(+), 4 deletions(-)\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex e47b1ea..87b104b 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -1382,6 +1382,7 @@ sub fetch_all {\n \t\t\t\t$base = $lr if ($lr < $base);\n \t\t\t}\n \t\t\tpush @gs, $gs;\n+\t\t\t$gs->sync_rev_map_with_commits;\n \t\t}\n \t}\n \n@@ -2114,6 +2115,44 @@ sub gc {\n \tcommand_noisy('gc', '--auto');\n };\n \n+# sync_rev_map_with_commits:\n+# If are commits on the tracking branch that aren't present in our revision\n+# map (e.g., because the user has done a git fetch from another git-svn repo\n+# rather than a git svn fetch), bring our revision map up to date. This is\n+# a no-op if the revision map is already up to date.\n+sub sync_rev_map_with_commits {\n+\tmy ($self) = @_;\n+\t# If we can't pull metadata out of log messages, there's nothing\n+\t# to import.\n+\treturn if $self->use_svm_props || $self->no_metadata;\n+\t# If there isn't a revision DB yet, we'll rebuild it from scratch\n+\t# elsewhere, so don't do anything here.\n+\treturn if ! -e $self->map_path || -z $self->map_path;\n+\t# Look at the most recent commit with a git-svn-id line.\n+\tmy ($log, $ctx) =\n+\t    command_output_pipe(qw/rev-list --pretty=raw --no-color /,\n+\t\t\t\t'--grep=^ *git-svn-id:',\n+\t\t\t\t'--max-count=1',\n+\t\t\t\t$self->refname, '--');\n+\tmy ($url, $rev, $uuid, $c);\n+\twhile (<$log>) {\n+\t\tif ( m{^commit ($::sha1)$} ) {\n+\t\t\t$c = $1;\n+\t\t\tnext;\n+\t\t}\n+\t\tnext unless s{^\\s*(git-svn-id:)}{$1};\n+\t\t($url, $rev, $uuid) = ::extract_metadata($_);\n+\t}\n+\tmy ($rev_commit) = $self->rev_map_get($rev, $uuid);\n+\tif (!$rev_commit) {\n+\t\t# The most recent commit in the branch isn't in our\n+\t\t# rev map. Pull in data from the revisions between the\n+\t\t# highest commit in our map and the head of the branch.\n+\t\tmy ($max_rev, $max_commit) = $self->rev_map_max(1);\n+\t\t$self->rebuild($max_commit);\n+\t}\n+}\n+\n sub do_git_commit {\n \tmy ($self, $log_entry) = @_;\n \tmy $lr = $self->last_rev;\n@@ -2489,6 +2528,7 @@ sub make_log_entry {\n sub fetch {\n \tmy ($self, $min_rev, $max_rev, @parents) = @_;\n \tmy ($last_rev, $last_commit) = $self->last_rev_commit;\n+\t$self->sync_rev_map_with_commits;\n \tmy ($base, $head) = $self->get_fetch_range($min_rev, $max_rev);\n \t$self->ra->gs_fetch_loop_common($base, $head, [$self]);\n }\n@@ -2535,10 +2575,17 @@ sub rebuild_from_rev_db {\n \tunlink $path or croak \"unlink: $!\";\n }\n \n+# rebuild:\n+# Reconstructs a revision map from the available metadata. If $min_git_rev\n+# is specified, this is an incremental rebuild that should stop when it hits\n+# the revision in question.\n+#\n+# Incremental rebuilding is only supported when commits contain git-svn\n+# metadata (the default) and not with use_svm_props or no_metadata.\n sub rebuild {\n-\tmy ($self) = @_;\n+\tmy ($self, $min_git_rev) = @_;\n \tmy $map_path = $self->map_path;\n-\treturn if (-e $map_path && ! -z $map_path);\n+\treturn if (!defined $min_git_rev && -e $map_path && ! -z $map_path);\n \treturn unless ::verify_ref($self->refname.'^0');\n \tif ($self->use_svm_props || $self->no_metadata) {\n \t\tmy $rev_db = $self->rev_db_path;\n@@ -2550,10 +2597,17 @@ sub rebuild {\n \t\t$self->unlink_rev_db_symlink;\n \t\treturn;\n \t}\n-\tprint \"Rebuilding $map_path ...\\n\";\n+\tmy $revs_to_scan;\n+\tif (defined $min_git_rev) {\n+\t\tprint \"Updating $map_path ...\\n\";\n+\t\t$revs_to_scan = $min_git_rev . \"..\" . $self->refname;\n+\t} else {\n+\t\tprint \"Rebuilding $map_path ...\\n\";\n+\t\t$revs_to_scan = $self->refname;\n+\t}\n \tmy ($log, $ctx) =\n \t    command_output_pipe(qw/rev-list --pretty=raw --no-color --reverse/,\n-\t                        $self->refname, '--');\n+\t                        $revs_to_scan, '--');\n \tmy $full_url = $self->full_url;\n \tremove_username($full_url);\n \tmy $svn_uuid = $self->ra_uuid;\n-- \n1.5.5.49.gf43e2\n"},{"id":"76340","messageId":"7v63tpd0fj.fsf@gitster.siamese.dyndns.org","threadId":"13426","inReplyTo":"20080508013956.GA24956@midwinter.com","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-08T01:55:28Z","receivedAt":"2008-05-08T01:55:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Steven Grimm <koreth@midwinter.com> writes:\n\n> In environments where a lot of people are sharing an svn repository using\n> git-svn, everyone has identical, but individually maintained, tracking\n> branches. If the svn repository is very active, it can take a while to\n> run \"git svn fetch\" (which has to individually construct each revision\n> by querying the svn server). It's much faster to run \"git fetch\" against\n> another git-svn repository to grab the exact same git revisions you'd get\n> from \"git svn fetch\". But until now, git-svn was confused by this because\n> it didn't know how to incrementally rebuild its map of revision IDs.\n> The only choice was to completely remove the map file and rebuild it\n> from scratch, possibly a lengthy operation when there's a lot of history.\n>\n> With this change, git-svn will try to do an incremental update of its\n> revision map if it sees that its tracking branch has svn revisions that\n> aren't in the map yet.\n\nBeing able to have a shared git-svn managed git repository that mirrors\nsvn is something people have asked often enough, but the recommended\npractice has always been for each to have his own copy.\n\nAlthough I do not use git-svn heavily myself, I like this addition.  We\nwould probably want to update the in-tree doc to cover a recommended\npattern of interacting multiple git repositories with a single svn\nrepository on the other side?\n"},{"id":"76341","messageId":"20080508015806.GA759@pe.Belkin","threadId":"13426","inReplyTo":"20080508013956.GA24956@midwinter.com","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Chris Shoemaker","fromEmail":"c.shoemaker@cox.net","sentAt":"2008-05-08T01:58:06Z","receivedAt":"2008-05-08T01:58:06Z","isPatch":true,"sender":{"key":"c.shoemaker@cox.net","avatar":null},"body":"On Wed, May 07, 2008 at 06:39:56PM -0700, Steven Grimm wrote:\n> In environments where a lot of people are sharing an svn repository using\n> git-svn, everyone has identical, but individually maintained, tracking\n> branches. If the svn repository is very active, it can take a while to\n> run \"git svn fetch\" (which has to individually construct each revision\n> by querying the svn server). It's much faster to run \"git fetch\" against\n> another git-svn repository to grab the exact same git revisions you'd get\n> from \"git svn fetch\". But until now, git-svn was confused by this because\n> it didn't know how to incrementally rebuild its map of revision IDs.\n> The only choice was to completely remove the map file and rebuild it\n> from scratch, possibly a lengthy operation when there's a lot of history.\n> \n> With this change, git-svn will try to do an incremental update of its\n> revision map if it sees that its tracking branch has svn revisions that\n> aren't in the map yet.\n\nSince I'm not qualified to review the patch technically , I'll just\noffer encouragement, comment and question.  First, nice work, this\nseems like a very helpful feature.  It might go quite a way toward\nenabling a semi-distributed workflow with an authoritative svn\nupstream.\n\nSecond, what will happen when different developers have svn URLs with\ndifferent schemes, e.g. http vs. svn+ssh?\n\nThird, I think such a feature surely deserves a mention in\ngit-svn.txt.\n\n-chris\n"},{"id":"76342","messageId":"064B1E1A-9C5C-49A4-AD08-0397FE4C517E@midwinter.com","threadId":"13426","inReplyTo":"20080508015806.GA759@pe.Belkin","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2008-05-08T02:08:50Z","receivedAt":"2008-05-08T02:08:50Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"On May 7, 2008, at 6:58 PM, Chris Shoemaker wrote:\n> Second, what will happen when different developers have svn URLs with\n> different schemes, e.g. http vs. svn+ssh?\n\nThat will cause the commit messages to be different, which means you  \nwon't have the same commit hashes, so this pretty much won't be  \nuseful. (You'd end up fetching the remote repo's entire svn history if  \nyou tried to do git fetch.)\n\nThe assumption here is that you have exactly the same revision history  \nin your tracking branches as the repo you're fetching from.\n\n-Steve\n"},{"id":"76343","messageId":"2F346570-C474-49A2-AB38-4D64792D1DB0@midwinter.com","threadId":"13426","inReplyTo":"7v63tpd0fj.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2008-05-08T02:17:09Z","receivedAt":"2008-05-08T02:17:09Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"On May 7, 2008, at 6:55 PM, Junio C Hamano wrote:\n> Being able to have a shared git-svn managed git repository that  \n> mirrors\n> svn is something people have asked often enough, but the recommended\n> practice has always been for each to have his own copy.\n\nWhich is still sort of the case here -- it would be even better if you  \ncould share the revision map, but that's a bigger change. All this  \nreally does is let you skip talking to the svn server for update  \noperations.\n\n> Although I do not use git-svn heavily myself, I like this addition.   \n> We\n> would probably want to update the in-tree doc to cover a recommended\n> pattern of interacting multiple git repositories with a single svn\n> repository on the other side?\n\nI'll write something up. This is still pretty new (it was an itch I  \nhad time to scratch today) so I honestly don't know yet what the  \noptimal workflow is going to be. But at the very least I can document  \nmy setup as an example of something that works.\n\n-Steve\n"},{"id":"76344","messageId":"20080508022504.GA931@pe.Belkin","threadId":"13426","inReplyTo":"064B1E1A-9C5C-49A4-AD08-0397FE4C517E@midwinter.com","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Chris Shoemaker","fromEmail":"c.shoemaker@cox.net","sentAt":"2008-05-08T02:25:04Z","receivedAt":"2008-05-08T02:25:04Z","isPatch":true,"sender":{"key":"c.shoemaker@cox.net","avatar":null},"body":"On Wed, May 07, 2008 at 07:08:50PM -0700, Steven Grimm wrote:\n> On May 7, 2008, at 6:58 PM, Chris Shoemaker wrote:\n>> Second, what will happen when different developers have svn URLs with\n>> different schemes, e.g. http vs. svn+ssh?\n>\n> That will cause the commit messages to be different, which means you won't \n> have the same commit hashes, so this pretty much won't be useful. (You'd \n> end up fetching the remote repo's entire svn history if you tried to do git \n> fetch.)\n\nYes, indeed.  But git fetching even the entire svn history is probably\noften faster than git-svn fetching even one commit!  I guess the\nquestion is really, if I replace my remote tracking branch with\nsomeone else's remote tracking branch, and it invalidates old map\nentries, what breaks?  Note, that even a slight difference in\nsvn.authorsfile would have the same effect.\n\n> The assumption here is that you have exactly the same revision history in \n> your tracking branches as the repo you're fetching from.\n\nIn that case, it would be helpful to enumerate exactly how two\ndevelopers can ensure that they are creating the same revision\nhistory.  At the very least, svn URL scheme and svnauthors file have\nto be the same.\n\n-chris\n"},{"id":"76349","messageId":"20080508041948.GA1095@midwinter.com","threadId":"13426","inReplyTo":"20080508013956.GA24956@midwinter.com","subject":"[PATCH v2] Teach git-svn how to catch up with its tracking branches","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2008-05-08T04:19:48Z","receivedAt":"2008-05-08T04:19:48Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"In environments where a lot of people are sharing an svn repository using\ngit-svn, everyone has identical, but individually maintained, tracking\nbranches. If the svn repository is very active, it can take a while to\nrun \"git svn fetch\" (which has to individually construct each revision\nby querying the svn server). It's much faster to run \"git fetch\" against\nanother git-svn repository to grab the exact same git revisions you'd get\nfrom \"git svn fetch\". But until now, git-svn was confused by this because\nit didn't know how to incrementally rebuild its map of revision IDs.\nThe only choice was to completely remove the map file and rebuild it\nfrom scratch, possibly a lengthy operation when there's a lot of history.\n\nWith this change, git-svn will try to do an incremental update of its\nrevision map if it sees that its tracking branch has svn revisions that\naren't in the map yet.\n\nSigned-off-by: Steven Grimm <koreth@midwinter.com>\n---\n\n\tAdded some documentation that I hope is comprehensive enough\n\tto be useful and non-misleading.\n\n Documentation/git-svn.txt |   72 +++++++++++++++++++++++++++++++++++++++++++-\n git-svn.perl              |   62 ++++++++++++++++++++++++++++++++++++--\n 2 files changed, 128 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex f4ba105..6bdbd51 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -511,8 +511,8 @@ inside git back upstream to SVN users.  Therefore it is advised that\n users keep history as linear as possible inside git to ease\n compatibility with SVN (see the CAVEATS section below).\n \n-CAVEATS\n--------\n+SHARING REVISIONS WITH OTHER GIT-SVN REPOSITORIES\n+-------------------------------------------------\n \n For the sake of simplicity and interoperating with a less-capable system\n (SVN), it is recommended that all git-svn users clone, fetch and dcommit\n@@ -521,6 +521,74 @@ operations between git repositories and branches.  The recommended\n method of exchanging code between git branches and users is\n git-format-patch and git-am, or just dcommiting to the SVN repository.\n \n+However, git-svn does have limited support for sharing the SVN history\n+between git repositories. For this to work, both git repositories need\n+to be using the same SVN repository via the same URL, such that the\n+commits in the git-svn tracking branches (e.g., the default \"git-svn\"\n+branch) have the same revision IDs in git.\n+\n+An easy way to test whether this is true is to use 'git svn find-rev'\n+in both repositories and pass it a recent SVN revision number that is\n+present in both repositories' history. For example, if both repositories\n+contain SVN revision 97446:\n+\n+------------------------------------------------------------------------\n+\tgit svn find-rev r97446\n+------------------------------------------------------------------------\n+\n+If the 40-digit hexadecimal value shown by that command is the same in\n+both repositories, their histories are compatible and the rest of this\n+section applies to them. There is currently no easy way to share history\n+between git repositories whose SVN tracking branches aren't identical.\n+\n+Assuming your two repositories match, you can use 'git fetch' in place\n+of 'git svn fetch' to fetch new SVN revisions. This is often\n+significantly faster than directly fetching from the svn server,\n+especially if large numbers of revisions are being fetched. Note that\n+this only applies to pulling changes *from* the SVN repository; you must\n+still use 'git svn dcommit' to push changes *to* SVN.\n+\n+Here's an example of how this works.\n+\n+------------------------------------------------------------------------\n+# Create the first git-svn clone; call it \"localmirror\"\n+\tgit svn clone svn+ssh://host/svn-repo localmirror\n+# Create an empty git-svn repository called \"work\" pointing to the same SVN repo\n+# (you could also use another 'git svn clone' if preferred)\n+\tmkdir work\n+\tcd work\n+\tgit svn init svn+ssh://host/svn-repo\n+# Set up a remote so 'git fetch' in work will fetch from localmirror\n+\tgit config remote.origin.url file://`pwd`/../localmirror\n+\tgit config remote.origin.fetch refs/remotes/git-svn:refs/remotes/git-svn\n+# Get a copy of the SVN history from localmirror\n+\tgit fetch\n+\tgit rebase git-svn\n+\n+# Some time passes...\n+\n+# Update localmirror with latest changes from SVN\n+\tcd ../localmirror\n+\tgit svn fetch\n+# And fetch those changes incrementally into work\n+\tcd ../work\n+\tgit fetch\n+# Incorporate the latest revisions into the current branch in work\n+\tgit rebase git-svn\n+# You can also fetch directly from SVN in work\n+\tgit svn rebase\n+# If you have local changes, you must dcommit directly to SVN\n+\tgit svn dcommit\n+------------------------------------------------------------------------\n+\n+In the above example, the \"localmirror\" repository might be set up to\n+run 'git svn fetch' periodically from a cron job, so that local developers\n+can always fetch recent SVN revisions without having to connect directly\n+to the SVN repository.\n+\n+CAVEATS\n+-------\n+\n Running 'git-merge' or 'git-pull' is NOT recommended on a branch you\n plan to dcommit from.  Subversion does not represent merges in any\n reasonable or useful fashion; so users using Subversion cannot see any\ndiff --git a/git-svn.perl b/git-svn.perl\nindex e47b1ea..87b104b 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -1382,6 +1382,7 @@ sub fetch_all {\n \t\t\t\t$base = $lr if ($lr < $base);\n \t\t\t}\n \t\t\tpush @gs, $gs;\n+\t\t\t$gs->sync_rev_map_with_commits;\n \t\t}\n \t}\n \n@@ -2114,6 +2115,44 @@ sub gc {\n \tcommand_noisy('gc', '--auto');\n };\n \n+# sync_rev_map_with_commits:\n+# If are commits on the tracking branch that aren't present in our revision\n+# map (e.g., because the user has done a git fetch from another git-svn repo\n+# rather than a git svn fetch), bring our revision map up to date. This is\n+# a no-op if the revision map is already up to date.\n+sub sync_rev_map_with_commits {\n+\tmy ($self) = @_;\n+\t# If we can't pull metadata out of log messages, there's nothing\n+\t# to import.\n+\treturn if $self->use_svm_props || $self->no_metadata;\n+\t# If there isn't a revision DB yet, we'll rebuild it from scratch\n+\t# elsewhere, so don't do anything here.\n+\treturn if ! -e $self->map_path || -z $self->map_path;\n+\t# Look at the most recent commit with a git-svn-id line.\n+\tmy ($log, $ctx) =\n+\t    command_output_pipe(qw/rev-list --pretty=raw --no-color /,\n+\t\t\t\t'--grep=^ *git-svn-id:',\n+\t\t\t\t'--max-count=1',\n+\t\t\t\t$self->refname, '--');\n+\tmy ($url, $rev, $uuid, $c);\n+\twhile (<$log>) {\n+\t\tif ( m{^commit ($::sha1)$} ) {\n+\t\t\t$c = $1;\n+\t\t\tnext;\n+\t\t}\n+\t\tnext unless s{^\\s*(git-svn-id:)}{$1};\n+\t\t($url, $rev, $uuid) = ::extract_metadata($_);\n+\t}\n+\tmy ($rev_commit) = $self->rev_map_get($rev, $uuid);\n+\tif (!$rev_commit) {\n+\t\t# The most recent commit in the branch isn't in our\n+\t\t# rev map. Pull in data from the revisions between the\n+\t\t# highest commit in our map and the head of the branch.\n+\t\tmy ($max_rev, $max_commit) = $self->rev_map_max(1);\n+\t\t$self->rebuild($max_commit);\n+\t}\n+}\n+\n sub do_git_commit {\n \tmy ($self, $log_entry) = @_;\n \tmy $lr = $self->last_rev;\n@@ -2489,6 +2528,7 @@ sub make_log_entry {\n sub fetch {\n \tmy ($self, $min_rev, $max_rev, @parents) = @_;\n \tmy ($last_rev, $last_commit) = $self->last_rev_commit;\n+\t$self->sync_rev_map_with_commits;\n \tmy ($base, $head) = $self->get_fetch_range($min_rev, $max_rev);\n \t$self->ra->gs_fetch_loop_common($base, $head, [$self]);\n }\n@@ -2535,10 +2575,17 @@ sub rebuild_from_rev_db {\n \tunlink $path or croak \"unlink: $!\";\n }\n \n+# rebuild:\n+# Reconstructs a revision map from the available metadata. If $min_git_rev\n+# is specified, this is an incremental rebuild that should stop when it hits\n+# the revision in question.\n+#\n+# Incremental rebuilding is only supported when commits contain git-svn\n+# metadata (the default) and not with use_svm_props or no_metadata.\n sub rebuild {\n-\tmy ($self) = @_;\n+\tmy ($self, $min_git_rev) = @_;\n \tmy $map_path = $self->map_path;\n-\treturn if (-e $map_path && ! -z $map_path);\n+\treturn if (!defined $min_git_rev && -e $map_path && ! -z $map_path);\n \treturn unless ::verify_ref($self->refname.'^0');\n \tif ($self->use_svm_props || $self->no_metadata) {\n \t\tmy $rev_db = $self->rev_db_path;\n@@ -2550,10 +2597,17 @@ sub rebuild {\n \t\t$self->unlink_rev_db_symlink;\n \t\treturn;\n \t}\n-\tprint \"Rebuilding $map_path ...\\n\";\n+\tmy $revs_to_scan;\n+\tif (defined $min_git_rev) {\n+\t\tprint \"Updating $map_path ...\\n\";\n+\t\t$revs_to_scan = $min_git_rev . \"..\" . $self->refname;\n+\t} else {\n+\t\tprint \"Rebuilding $map_path ...\\n\";\n+\t\t$revs_to_scan = $self->refname;\n+\t}\n \tmy ($log, $ctx) =\n \t    command_output_pipe(qw/rev-list --pretty=raw --no-color --reverse/,\n-\t                        $self->refname, '--');\n+\t                        $revs_to_scan, '--');\n \tmy $full_url = $self->full_url;\n \tremove_username($full_url);\n \tmy $svn_uuid = $self->ra_uuid;\n-- \n1.5.5.49.gf43e2\n"},{"id":"76351","messageId":"alpine.DEB.1.00.0805072332300.6948@swallowtail","threadId":"13426","inReplyTo":"20080508013956.GA24956@midwinter.com","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Asheesh Laroia","fromEmail":"asheesh@asheesh.org","sentAt":"2008-05-08T06:48:17Z","receivedAt":"2008-05-08T06:48:17Z","isPatch":true,"sender":{"key":"asheesh@asheesh.org","avatar":"https://avatars.githubusercontent.com/u/25457?v=4"},"body":"On Wed, 7 May 2008, Steven Grimm wrote:\n\n> In environments where a lot of people are sharing an svn repository using\n> git-svn, everyone has identical, but individually maintained, tracking\n> branches.\n\nTo further muddy the waters, let me talk about my setup, also one with a \n\"central git repository\" from which all developers clone, and also one \nbased on a Subversion tree.\n\nThe way I handle it is that, hidden somewhere, I have an account with a \ncron job that does this:\n\n$ git svn fetch\n$ git push origin refs/remotes/*:refs/heads/*\n$ git push origin refs/remotes/trunk:refs/heads/master\n\nThe first push synchronizes \"origin\" to have the same branches as this \ngit-svn copy of the git repository, and the second updates \"origin\" so \nthat it has a \"master\"; without that second step, \"git clone\" will error \nout when it get to its checkout phase.\n\nNote that in .git/config, the [remote \"origin\"] section has no \"fetch\" \nparameter.  If it did have one, a would end up creating the branch \norigin/master on the second push, and origin/origin/master on the third, \nand so on.\n\nAfter the push, \"origin\" ends up being a git repository that looks just \nlike the svn repository we're cloning.  When you \"git clone\" it, the \nremote has all the tags and branches of the upstream svn repository; and \nas the upstream svn repository updates its branches, the git branches get \nthose updates.\n\nI'm not saying this patch shouldn't be accepted; I have no comment on it. \nI just want to see what others think of my approach to this workflow.\n\n-- Asheesh.\n\n-- \nWhat happened last night can happen again.\n"},{"id":"76353","messageId":"CE2D2A30-CAAF-4903-95EA-B4577932B700@midwinter.com","threadId":"13426","inReplyTo":"alpine.DEB.1.00.0805072332300.6948@swallowtail","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2008-05-08T07:33:12Z","receivedAt":"2008-05-08T07:33:12Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"On May 7, 2008, at 11:48 PM, Asheesh Laroia wrote:\n> The way I handle it is that, hidden somewhere, I have an account  \n> with a cron job that does this:\n>\n> $ git svn fetch\n> $ git push origin refs/remotes/*:refs/heads/*\n> $ git push origin refs/remotes/trunk:refs/heads/master\n\nThat's a reasonable setup, and I think (without having tried it) that  \nit will be compatible with my patch -- assuming the clones of your  \norigin repository have appropriate svn-remote config entries, they  \nshould be able to mix and match fetching from your origin and the real  \nsvn repository, and dcommit stuff back to svn.\n\nThough I'd try that out with a toy svn repo first...\n\n-Steve\n"},{"id":"76355","messageId":"20080508073851.GA302@diana.vm.bytemark.co.uk","threadId":"13426","inReplyTo":"20080508022504.GA931@pe.Belkin","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-05-08T07:38:51Z","receivedAt":"2008-05-08T07:38:51Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2008-05-07 22:25:04 -0400, Chris Shoemaker wrote:\n\n> On Wed, May 07, 2008 at 07:08:50PM -0700, Steven Grimm wrote:\n>\n> > The assumption here is that you have exactly the same revision\n> > history in your tracking branches as the repo you're fetching\n> > from.\n>\n> In that case, it would be helpful to enumerate exactly how two\n> developers can ensure that they are creating the same revision\n> history. At the very least, svn URL scheme and svnauthors file have\n> to be the same.\n\nAlso, one mustn't use Subversion's ability to retroactively edit\ncommit messages. (Guess what we tend to do from time to time where I\nwork.)\n\nWhat'd really be needed to get all of the corner cases right, I think,\nis a single import point that everyone else pulls from. The patch that\nstarted this thread would help that scenario, since it makes it\npossible to pull from such a central import point, and then run the\ndcommit locally. (There's still the problem that dcommit will do a\nlocal import if necessary -- that would have to be fixed as well.\nMaybe simply teach git-svn to always try to pull from a given git\nrepository before hitting the real svn repo.)\n\n-- \nKarl Hasselström, kha@treskal.com\n      www.treskal.com/kalle\n"},{"id":"76356","messageId":"20080508074332.GB302@diana.vm.bytemark.co.uk","threadId":"13426","inReplyTo":"20080508073851.GA302@diana.vm.bytemark.co.uk","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-05-08T07:43:32Z","receivedAt":"2008-05-08T07:43:32Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2008-05-08 09:38:51 +0200, Karl Hasselström wrote:\n\n> (There's still the problem that dcommit will do a local import if\n> necessary -- that would have to be fixed as well. Maybe simply teach\n> git-svn to always try to pull from a given git repository before\n> hitting the real svn repo.)\n\nOr even _only_ pull from the git repo. That repo would have to have\nsome kind of hook to make sure that it's always up-to-date, then --\njust a cron job won't do -- but I'm sure that can be done.\n\n-- \nKarl Hasselström, kha@treskal.com\n      www.treskal.com/kalle\n"},{"id":"76357","messageId":"20080508074824.GA2197@pe.Belkin","threadId":"13426","inReplyTo":"alpine.DEB.1.00.0805072332300.6948@swallowtail","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Chris Shoemaker","fromEmail":"c.shoemaker@cox.net","sentAt":"2008-05-08T07:48:24Z","receivedAt":"2008-05-08T07:48:24Z","isPatch":true,"sender":{"key":"c.shoemaker@cox.net","avatar":null},"body":"On Wed, May 07, 2008 at 11:48:17PM -0700, Asheesh Laroia wrote:\n> On Wed, 7 May 2008, Steven Grimm wrote:\n>\n>> In environments where a lot of people are sharing an svn repository using\n>> git-svn, everyone has identical, but individually maintained, tracking\n>> branches.\n>\n> To further muddy the waters, let me talk about my setup, also one with a \n> \"central git repository\" from which all developers clone, and also one \n> based on a Subversion tree.\n>\n> The way I handle it is that, hidden somewhere, I have an account with a \n> cron job that does this:\n>\n> $ git svn fetch\n> $ git push origin refs/remotes/*:refs/heads/*\n> $ git push origin refs/remotes/trunk:refs/heads/master\n>\n> The first push synchronizes \"origin\" to have the same branches as this \n> git-svn copy of the git repository, and the second updates \"origin\" so that \n> it has a \"master\"; without that second step, \"git clone\" will error out \n> when it get to its checkout phase.\n>\n> Note that in .git/config, the [remote \"origin\"] section has no \"fetch\" \n> parameter.  If it did have one, a would end up creating the branch \n> origin/master on the second push, and origin/origin/master on the third, \n> and so on.\n>\n> After the push, \"origin\" ends up being a git repository that looks just \n> like the svn repository we're cloning.  When you \"git clone\" it, the remote \n> has all the tags and branches of the upstream svn repository; and as the \n> upstream svn repository updates its branches, the git branches get those \n> updates.\n>\n> I'm not saying this patch shouldn't be accepted; I have no comment on it. I \n> just want to see what others think of my approach to this workflow.\n\nThis workflow doesn't seem to provide a way for the developers who\nclone the \"origin\" above, to dcommit to svn.  Presumably, with the\nright initialization, Steve's patch would allow all those clones to\ndcommit to svn directly.\n\nI like your automated mirror setup, but IMO, it becomes a lot more\nuseful in conjunction with Steve's patch.\n\n-chris\n"},{"id":"76358","messageId":"604A38BB-E6A4-4303-BD7C-BF2968B6828D@midwinter.com","threadId":"13426","inReplyTo":"20080508074332.GB302@diana.vm.bytemark.co.uk","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2008-05-08T07:58:48Z","receivedAt":"2008-05-08T07:58:48Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"On May 8, 2008, at 12:43 AM, Karl Hasselström wrote:\n> Or even _only_ pull from the git repo. That repo would have to have\n> some kind of hook to make sure that it's always up-to-date, then --\n> just a cron job won't do -- but I'm sure that can be done.\n\nIf you control both the svn repo and the git repo, you can get close  \nto that with an svn commit trigger, but even then there'll be a race  \ncondition when you want to dcommit. You either have to be able to pull  \nfrom the svn repo when you want to dcommit, or you have to live with  \nthe possibility of a dcommit failing because you don't actually have  \nthe most recent rev locally.\n\nThis ties a bit into the patch I sent a few months back to allow  \nupdate hooks to change refs. I kind of ran out of spare time to  \niterate more on that back then, but the ultimate goal there was that  \nyou could interact only with the bridge repo, never directly with svn,  \nand the bridge repo would dcommit for you when you pushed to it.\n\nThe approach in this thread's patch is maybe not as conceptually  \nclean, but it's much simpler and (apparently) less controversial.\n\n-Steve"},{"id":"76360","messageId":"20080508081331.GC302@diana.vm.bytemark.co.uk","threadId":"13426","inReplyTo":"604A38BB-E6A4-4303-BD7C-BF2968B6828D@midwinter.com","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-05-08T08:13:31Z","receivedAt":"2008-05-08T08:13:31Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2008-05-08 00:58:48 -0700, Steven Grimm wrote:\n\n> On May 8, 2008, at 12:43 AM, Karl Hasselström wrote:\n>\n> > Or even _only_ pull from the git repo. That repo would have to\n> > have some kind of hook to make sure that it's always up-to-date,\n> > then -- just a cron job won't do -- but I'm sure that can be done.\n>\n> If you control both the svn repo and the git repo, you can get close\n> to that with an svn commit trigger, but even then there'll be a race\n> condition when you want to dcommit. You either have to be able to\n> pull from the svn repo when you want to dcommit, or you have to live\n> with the possibility of a dcommit failing because you don't actually\n> have the most recent rev locally.\n\nI don't see why this has to be the case. Surely, if the local git repo\ncan dcommit without races by importing new revisions from svn, the\nlocal git repo could dcommit without races by importing new revisions\nfrom svn via an intermediate git repo. This would require the\nintermediate repo to import new revisions when it gets a pull request,\nbut surely that should be doable with a pre-pull hook?\n\n> This ties a bit into the patch I sent a few months back to allow\n> update hooks to change refs. I kind of ran out of spare time to\n> iterate more on that back then, but the ultimate goal there was that\n> you could interact only with the bridge repo, never directly with\n> svn, and the bridge repo would dcommit for you when you pushed to\n> it.\n\nI guess that approach is what I'd really like to see, since that's the\nonly one that can guarantee that every git clone of the svn repository\nis identical.\n\nFurthermore, in this setup the clients wouldn't need to run git-svn at\nall. Only the bridge would need it.\n\n-- \nKarl Hasselström, kha@treskal.com\n      www.treskal.com/kalle\n"},{"id":"76361","messageId":"20080508082141.GB2197@pe.Belkin","threadId":"13426","inReplyTo":"alpine.DEB.1.00.0805072332300.6948@swallowtail","subject":"Re: [PATCH] Teach git-svn how to catch up with its tracking branches","fromName":"Chris Shoemaker","fromEmail":"c.shoemaker@cox.net","sentAt":"2008-05-08T08:21:41Z","receivedAt":"2008-05-08T08:21:41Z","isPatch":true,"sender":{"key":"c.shoemaker@cox.net","avatar":null},"body":"On Wed, May 07, 2008 at 11:48:17PM -0700, Asheesh Laroia wrote:\n> On Wed, 7 May 2008, Steven Grimm wrote:\n>\n>> In environments where a lot of people are sharing an svn repository using\n>> git-svn, everyone has identical, but individually maintained, tracking\n>> branches.\n>\n> To further muddy the waters, let me talk about my setup, also one with a \n> \"central git repository\" from which all developers clone, and also one \n> based on a Subversion tree.\n>\n> The way I handle it is that, hidden somewhere, I have an account with a \n> cron job that does this:\n>\n> $ git svn fetch\n> $ git push origin refs/remotes/*:refs/heads/*\n> $ git push origin refs/remotes/trunk:refs/heads/master\n>\n> The first push synchronizes \"origin\" to have the same branches as this \n> git-svn copy of the git repository, and the second updates \"origin\" so that \n> it has a \"master\"; without that second step, \"git clone\" will error out \n> when it get to its checkout phase.\n\nThis got me thinking about a potential design for a git-svnserver.\n[Warning: engineering hack ahead, proceeed with caution.]\n\nInstead of re-implementing any part of svn, just use a stock svn repo\n+ server.  From the svn post-commit hook, update a git-svn repo as\nabove.  From the git post-commit, do a git-svn rebase.  Of course, you\nneed a shared lock between the two pairs of pre/post commit hooks.\n\nThe problem of attribution in svn from git-svn is probably easier to\nsolve from within the context of a post-commit hook.  The problem of\nhaving to round-trip git commits through svn in a way that changes\ntheir ids remains.  Effectively, that means commits have to be\nconsidered \"unpublished\" (for the purpose of not basing other work\nupon them) until they are pushed to the git-half of the git+svn.\n\nStill, this scenario is a pretty gentle migration path from svn to git\n- one that allows regular git users to use only git-core, not git-svn,\nand still allows svn clients to work.  Maybe some git-alias magic\ncould hide the fact that a git push has to really become a push +\nfetch.\n\n\n-chris\n"},{"id":"76592","messageId":"20080511082727.GB23929@untitled","threadId":"13426","inReplyTo":"20080508041948.GA1095@midwinter.com","subject":"Re: [PATCH v2] Teach git-svn how to catch up with its tracking branches","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-05-11T08:27:27Z","receivedAt":"2008-05-11T08:27:27Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Steven Grimm <koreth@midwinter.com> wrote:\n> In environments where a lot of people are sharing an svn repository using\n> git-svn, everyone has identical, but individually maintained, tracking\n> branches. If the svn repository is very active, it can take a while to\n> run \"git svn fetch\" (which has to individually construct each revision\n> by querying the svn server). It's much faster to run \"git fetch\" against\n> another git-svn repository to grab the exact same git revisions you'd get\n> from \"git svn fetch\". But until now, git-svn was confused by this because\n> it didn't know how to incrementally rebuild its map of revision IDs.\n> The only choice was to completely remove the map file and rebuild it\n> from scratch, possibly a lengthy operation when there's a lot of history.\n> \n> With this change, git-svn will try to do an incremental update of its\n> revision map if it sees that its tracking branch has svn revisions that\n> aren't in the map yet.\n\nCool.  I agree with this is a useful change for people wanting to save\ntime and bandwidth although I've never been in a situation to need it\nmyself.\n\nHowever, I'm kind of uncomfortable with this being on by default, as it\nreally means they users have to trust the git repository they're\nfetching from to always be configured identically to what they're using\nand not change configurations midway through a project.  External things\nlike authors files would need to be synced, too, etc...\n\n> +sub sync_rev_map_with_commits {\n> +\tmy ($self) = @_;\n> +\t# If we can't pull metadata out of log messages, there's nothing\n> +\t# to import.\n> +\treturn if $self->use_svm_props || $self->no_metadata;\n> +\t# If there isn't a revision DB yet, we'll rebuild it from scratch\n> +\t# elsewhere, so don't do anything here.\n> +\treturn if ! -e $self->map_path || -z $self->map_path;\n> +\t# Look at the most recent commit with a git-svn-id line.\n> +\tmy ($log, $ctx) =\n> +\t    command_output_pipe(qw/rev-list --pretty=raw --no-color /,\n> +\t\t\t\t'--grep=^ *git-svn-id:',\n\nEven though rev-list outputs the commit message prefixed with spaces,\nthe --grep itself does not need ' *' to match the leading spaces.\nNo version of git-svn outputting a space before \"git-svn-id: \"\nin the commit itself.\n\nMore importantly I'd also prefer to actually grep for the URL+path of\nthe ref we're tracking, too.  This can catch mistakes if people somehow\nconfigured their remotes incorrectly.\n\n-- \nEric Wong\n"}]}