Show 216 quoted lines
>> I hate making more work for people but I would love a copy of your
>> notes.
>
> Okay, here we go! I've uploaded the applicable scripts to
> https://gist.github.com/f6902cb4e3534f07ba48
>
> If you (or anyone) finds I describe something here that isn't on github, let
> me know and I'll add to it. I did a cursory pass through the scripts to
> remove a lot of the specific-to-our-repo stuff, so I'm not even sure these
> scripts will run as is... But most errors should be pretty minor (typos in
> variable names, etc) the overall procedure is unchanged. (And please be
> gentle, these are not anything approaching production-ready)
>
> As always, these scripts come with ABSOLUTELY NO WARRANTEE, use at your own
> risk, your mileage may vary, etc.
>
> Converting to Git using svn-fe
> ------------------------------
> Most people who have tried using git-svn to convert a medium to large
> Subversion repository have found it's a slow process. When I asked the Git
> mailing list about this problem in June 2010, I was pointed to David Barr's
> svn-dump-fast-export tool:
> http://github.com/barrbrain/svn-dump-fast-export
> svn-fe (as the executable is called) converts an entire svn repository to git
> very quickly (our repository took about 20 minutes), but the entire svn file
> system is one branch. I developed the following process to reproduce the svn
> history in git.
>
> Initial Thoughts
> ----------------
> 1) Our SVN repository was approximately 20k commits, about 7k files in HEAD,
> a little less than 400 tags, and about 100-150 branches. It was organized
> /trunk/project rather than /project/trunk. Branches were
> /branches/branchName where the branchName directory was a copy of the entire
> trunk (so /branches/branchName/project is what a user would checkout). This
> does affect the scripts, but I think it should be relatively easy to modify
> (no guarantee though).
>
> 2) Our SVN repository originated from cvs2svn, so there are some artifacts
> from that conversion that affect this conversion.
>
> 3) I make very little use of Git.pm because while I was developing I ran into
> a bunch of problems with it (none of which I remember now). Instead I make
> use of perl's system call to send commands to Git (where possible I avoid
> invoking the shell, see perldoc -f exec). I don't want to imply Git.pm
> doesn't work, but at the time it didn't work for me (and I was more focused
> on making my scripts work than improving Git.pm. Sorry!).
>
> 4) The vast majority of our history was before SVN introduced merge-info, so
> I made no attempt to capture SVN merges in Git. Rather I kept all branch
> heads, but moved most of them to a "hidden" namespace (see hideFromGit.pl for
> details). This does mean for a couple merges post-conversion I've had to add
> temporary grafts to make the merge work, but I haven't bothered making those
> grafts permanent (hopefully this isn't a problem?)
>
> 5) I performed this entire process using a local mirror of our SVN repository
> in about 4 hours. It is mostly automated, but does require some human
> monitoring (maybe I'm just paranoid). Since svn-fe runs off a SVN dump file,
> creating the local mirror was a trivial additional step.
>
> 6) To keep what follows a *little* shorter, I'm going to assume you can read
> Perl to extract the details of what's going on. I'll try to keep the prose
> to a high level...
>
> Extracting SVN's History
> ------------------------
> First we want to understand SVN's branching/tagging history. Modify
> buildSVNTree.pl as necessary, then run
> perl buildSVNTree.pl > svnBranches.txt
>
> buildSVNTree.pl does the following steps:
> 1. Traverses the SVN history chronologically looking for copies.
> 2. Records the source path/rev and destination path/rev for (most) copies
> (see script for details)
> 3. Once all copies are collected, further filters copies based on:
> * source path is a directory
> * source and destination are not in trunk
> * source and destination are not in the same branch or tag
> * source path is not /vendor (an artifact of cvs2svn)
> 4. Checks that source path is "shortest" path from it's rev (protect against
> subdirectories that get added in the same commit)
> 5. Checks the source and destination paths match globs for expected paths
> (non-matching copies that make it this far are printed to STDERR)
> 6. Creates a Git branch name for destination (note that svn tags are closer to git branches than git tags)
> 7. Search history for the last commit that actually changed the source path
> 8. Find a parent path from the source path (mostly recurse up the SVN tree to a known branch)
> 9. Use the parent path to determine the parent git branch name
> 10. Record parent/child relationships
> 11. Dump output to STDOUT (which you should redirect to a file for later use)
>
> I did run into one place where two SVN branches had the same name but
> different SVN paths (it's complicated). In this case I just manually edited
> the git branch name in svnBranches.txt. As long as you do that before
> continuing, everything should be okay.
>
> There's also some logic in buildSVNTree to determine if a branch/tag is
> deleted in the SVN head. That information is used by hideFromGit.
>
> Create the Single Branch Git Repo
> ---------------------------------
> Use svn-fe for what it's designed:
> 1. svnadmin dump /path/to/svn/repo > svn-dump.txt
> 2. git init /path/to/initial/git/repo
> 3. cd /path/to/initial/git/repo
> 4. cat /path/to/svn-dump.txt | svn-fe svnRepoName | git fast-import
>
> svnRepoName in step 4 can be anything you want, but it has to be specified so
> that svn-fe appends the git-svn style "git-svn-id: svnRepoName@svnRevNum
> svnRepoUUID" line to each commit message. This line is required later to map
> SVN revs to Git commits.
>
> Create Git Branches and Tags
> ----------------------------
> Now comes the next script, filterBranch.pl. filterBranch will create Git
> branches and tags out of the single branch repo by creating a ton of clones
> and filtering each one. While it's doing this, it also changes the SVN user
> names to proper Git user IDs (name + email). fetchSVNNames.pl can be used to
> get all the svn users, then you can edit $authorScript in filterBranch to
> modify names appropriately ($authorScript is a git-filter-branch
> --env-filter, so it gets eval'ed by git). Per the git-filter-branch manpage,
> you'll want to create/use a RAM disk for temporary files (see $tempdir). And
> you'll need to set various paths like $parentRepo (this is the repo created
> in step 2 above), etc.
>
> Then the script should be (?) relatively automated:
> perl filterBranch.pl svnBranches.txt
>
> The fancy logic here is probably figuring out which Git refs go to which Git
> commit, but I'll leave that as an exercise to the reader... Ah, I should
> probably mention: svn-fe can produce "empty" commits, and filterBranch does
> nothing to remove them. By "empty" I mean there will be a commit object
> without any content changes. So creating a branch/tag in SVN creates a
> commit, but doesn't change content. That commit will be part of the new Git
> history. Similarly, filterBranch will create git tags from svn tags, but
> they point to one of these "empty" commits rather than the branch they are
> tagged from. It's not very git-ish, but it seems to work...
>
> filterBranch is probably the longest step of the process; there's a lot of
> filtering going on. It will be very verbose on STDOUT, so I recommend
> tee'ing to a file or a terminal with infinite scroll back. It also involves
> a lot of disk hits (somewhat reduced if $tempdir is a RAM disk), and
> potentially a lot of space (it will create a git repo for every branch/tag in
> your subversion history). For our repository this step took about 1.5-2
> hours IIRC.
>
> Create SVN/Git Revmaps
> ----------------------
> Next step is to create a map that goes from SVN rev to Git commit object.
> genRevmap.pl and genJointRevmap.pl will be helpful here:
> 1. cd $cleanDir (from filterBranch)
> 2. find . -type d -name "*.git" -exec genRevmap.pl '{}' svnRepoName destDir ';'
> 3. cd destDir
> 4. find . -name "*.revmap" -exec grep . '{}' + | genJointRevMap.pl > jointRevmap.revmap
>
> genRevmap will respect the directory hierarchy created by filterBranch, and
> destDir must have a similar structure (doesn't require the individual Git
> repos, but any directory that contains a git repo must exist in destDir).
> genJointRevMap takes individual revmaps and creates a big revmap for all the
> repositories. These scripts aren't doing any real magic, just parsing the
> Git log messages for commit ID and the git-svn-id line to get the SVN rev the
> commit corresponds to. Note that SVN rev to Git commit can be one to many!
> (genRevmap just lists the same rev twice if it has more than one git commit
> associated with it, genJointRevMap flags those revs specially and lists all
> commit IDs on a single line).
>
> Assembling the Final Git Repo
> -----------------------------
> Now we need to combine all the small git repos into one repo that represents
> the SVN history. Similar to filterBranch, you'll need to edit paths in
> repoFusion.pl to make sure it finds everything. Then simply:
> perl repoFusion.pl svnBranches.txt jointRevmap.revmap
>
> At a high level, repoFusion:
> 1. Clones the trunk repository, this will become the new master branch
> 2. Performs a git-fetch on every other repository created by filterBranch to
> retrieve the git branch/tags contained there
> 3. Creates grafts to match up git branches with their parents using the revmap
> 4. If manual grafts are required, it will pause so the user can edit the
> grafts file (search for '*', the message there might be a little cryptic, but
> using svn log and git log in combination, hopefully you can figure out what
> the correct SHA is to insert)
> 5. Runs filter-branch one more time to make the grafts permanent.
>
> This is a bit faster than filterBranch, but still takes on the order of an
> hour for our repository. It also produces a lot of stuff on STDOUT, but I
> think it's a little easier on the disk. At the end of the filter branch, I
> found it useful to scan the output for refs that weren't updated... That
> usually indicates a graft didn't get created correctly (although due to SVN
> conventions, it's unlikely the master ref will change) At this point it's
> also possible to get some branch/tag name clashes (I did), so those may
> require clean up.
>
> Hiding 'Deleted' Branches
> -------------------------
> hideFromGit.pl will use the svnBranches.txt file to move any git refs
> associated with deleted SVN paths to refs/hidden in the new repository. This
> keeps the objects associated with those refs from getting garbage collected,
> but hides them from most user commands. This is entirely a personal
> preference. (Just like the other scripts, you'll probably have to edit the
> paths in the script itself)
>
> 'Validating' the Conversion
> ---------------------------
> gitValidation.pl is a script I wrote to randomly select revs from SVN and try
> to compare the SVN diffs to the Git diffs. It uses git-patch-id to compute a
> SHA of the changes in each repository, and reports if something doesn't match
> up. It's not particularly polished, and does find "errors" in our Git repo,
> but after investigating all the discrepancies I'm pretty happy that nothing
> vital is wrong.
>
> Closing Thoughts
> ----------------
> Do I have any? This is quite the brain dump, so I'm sure I've been
> incomplete and probably somewhat confusing... I'm happy to answer questions
> as I can, but again, this is entirely based on my experience with our local
> repo. YMMV!Jonathan Nieder <jrnieder@gmail.com>