{"thread":{"id":"3840","subject":"[ANNOUNCE] git-svnconvert: YASI (Yet Another SVN importer)","startedAt":"2006-04-09T16:34:26Z","lastAt":"2006-04-09T22:06:55Z","messageCount":5,"participants":["Rutger Nijlunsing","Jakub Narebski","Johannes Schindelin","Randal L. Schwartz"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"18499","messageId":"20060409163426.GA22103@nospam.com","threadId":"3840","inReplyTo":null,"subject":"[ANNOUNCE] git-svnconvert: YASI (Yet Another SVN importer)","fromName":"Rutger Nijlunsing","fromEmail":"rutger@nospam.com","sentAt":"2006-04-09T16:34:26Z","receivedAt":"2006-04-09T16:34:26Z","isPatch":false,"sender":{"key":"rutger.nijlunsing@gmail.com","avatar":null},"body":"Hi all,\n\nSince I didn't succeed in importing a (private) SVN repo into git, I\nwrote a new converter to handle more cases like:\n\n  - use command line svn instead of some perl library to have less\n    dependancies and to support proxy + repo authentification.\n    Might even work on MacOSX ;)\n  - automatic merge detection by looking at from which revision a\n    revision gets its files\n  - visualisation of the branch tree with dotty to check what git-convert\n    would produce _before_ importing it.\n  - /trunk is moved to /branches/old, /branches/new_branch becomes /trunk\n    in next revision (ARGH!)\n  - To be able to continue after a ^C and be able to continue where\n    it stopped.\n  - have configurable repo layout to handle things like:\n    - files next to branches in /branches\n    - /branches/\n\nRegards,\nRutger.\n\n-- \nRutger Nijlunsing ---------------------------------- eludias ed dse.nl\nnever attribute to a conspiracy which can be explained by incompetence\n----------------------------------------------------------------------\n\n#!/usr/bin/env ruby\n\n# Convert a Subversion repository with all its branches\n# into git incrementally.\n#\n# The main difference with git-svnimport is\n# that it handles a badly-broken archive which I wanted to \n# convert which git-svnimport did not handle ;)\n#\n# But, of course, your milage may (will!) vary. svn only defines\n# snapshot-trees and interpretation of each leaf in the tree is left\n# as an exercise for the reader.\n#\n# Features:\n#  - import _all_ branches\n#  - uses svn command line, so:\n#    - supports HTTP proxy with authentification\n#    - supports repository authentification\n#    - supports incorrect (self-signed) SSL certificates ;)\n#  - can handle multiple branche changes in one revision\n#  - supports extra files in /branches next to real branches\n#  - gets merge information by looking at which revisions we're based on\n#    (so no need to parse commit messages)\n#    This (only) works when the merge of two branches results\n#    in files from both branches.\n#  - handles missing revisions\n#  - no extra packages needed, just ruby and subversion.\n#  - outputs GraphViz .dot files for visualisation of branches\n#  - fairly interrupt-save. Once in a while it saves it state\n#    to be able to continue next time where it left off.\n#  - works parallel: one process fetches, while another copies\n#    it into git.\n# Con:\n#  - needs a complete, checked-out SVN repository locally\n#    _for_ _each_ _branch_.\n#    This takes disk space. However, disk space is cheap ;)\n# TODO:\n#  - parse XML as XML instead of text. In case the exact XML formatting\n#    changes in the future...\n#  - add tagging support\n#  - handle file properties like execute and ignore\n#  - use 'svn switch' to checkout other branches fast\n#\n# Workings\n#  - Get a log of all commits\n#  - Create a branches graph from this log\n#  - Check out each revision separately and write a commit\n#    for each revision.\n#\n# (c)2006 R. Nijlunsing <git-svnconvert@tux.tmfweb.nl>\n# Released under the GNU Public License, version 2.\n\n\n$VERBOSE = true\t\t\t# Let Ruby warn more\n\nrequire 'set'\nrequire 'time'\nrequire 'find'\nrequire 'fileutils'\nrequire 'optparse'\nrequire 'cgi'\t\t\t# For unescapeHTML()\n\n#################### Configuration\n\n# Root directories of the branches and trunk(s).\n#   - one line per directory\n#   - '*' matches one filename component and is matched last\n#   - names are case sensitive\n#   - whitespaces at start and end are ignored\n$branch_dirs = %q{\n  /branches/*\n  /branches/cc_test/*\n  /branches/pre-dev/trunk\n  /branches/pre-dev/trunk_old\n  /branches/tasks/*\n  /trunk\n}\n\n# List of paths which might get matched by $branch_dirs,\n# but which are not the root of branches.\n#\n# Argh. People put files into the root of branches,\n# which ends up as a branch (so for example /branches/README\n# ends up as a branch). List here all files in the roots\n# to ignore them.\n$not_branch_dirs = %q{\n  /branches/README\n  /branches/pre-dev\n  /branches/cc_test\n  /branches/tasks\n}\n\n#################### End of configuration\n\ndef read_svn_authors(authors_filename)\n  users = {}\n  begin\n    IO.foreach(authors_filename) { |line|\n      if line =~ %r{^(\\S+?)\\s*=\\s*(.+?)\\s*<(.+)>\\s*$}\n\tuser, name, email = $1, $2, $3\n\tusers[user] = [name, email]\n      end\n    }\n  rescue Errno::ENOENT, Errno::EACCES\n    die(\"Could not read #{authors_filename}: #{$!}\")\n  end\n  if $verbose\n    puts \"Read #{users.size} authors from #{authors_filename}\"\n  end\n  return users\nend\n\ndef write_svn_authors(users, authors_filename)\n  begin\n    File.open(authors_filename, \"wb\") { |io|\n      users.keys.sort.each { |user|\n\tio.puts \"#{user} = #{users[user][0]} <#{users[user][1]}>\"\n      }\n    }\n    if $verbose\n      puts \"Wrote #{users.size} authors to #{authors_filename}\"\n    end\n  rescue Errno::EACCES\n    die(\"Could not write #{authors_filename}: #{$!}\")\n  end\nend\n\n# Given a string with on each line a root directory, generate a regular\n# expression matching one of those root directories.\ndef root_dirs_to_regexp(rootdirs, is_prefix)\n  Regexp.new(\n    \"^(?:\" +\t\t\t# Match at start without capturing\n    rootdirs.\n      split(\"\\n\").\n      find_all { |dir| dir.strip != \"\" }.\t# Delete empty lines\n      collect { |dir|\n        # make path absolute; use '/' as path separator;\n        (\"/\" + dir.strip).gsub(%r{[\\\\/]+}, \"/\")\n      }.sort { |dir1, dir2|\n        # Sort on size so '/branches/development/trunk' comes before\n        # '/branches/*' and will therefore be matched.\n        dir2.size <=> dir1.size\n      }.collect { |dir|\n        Regexp.escape(dir)\t# Escape everything\n      }.join(\"|\").gsub(\"\\\\*\", \"[^/]+\") + # Unescape '*' back\n    \")\" +\n    # If not a prefix, must match at end\n    # If prefix, must match at path delimiter (or at end)\n    (is_prefix ? \"(?=/|$)\" : \"$\")  \n  )\nend\n\n# Execute shell command; bail out at error\ndef safe_system(cmd)\n  puts cmd if $verbose\n  system(cmd)\n  if $? != 0\n    puts cmd if !$verbose\n    puts \"!!! Command returned non-zero exit code: #{$?}\"\n    puts \"!!! Working dir: #{Dir.pwd}\"\n    exit $?\n  end\nend\n\ndef safe_popen(cmd, mode = \"w+\", &callback)\n  puts \"|\" + cmd if $verbose\n  res = IO.popen(cmd, mode, &callback)\n  if $? != 0\n    puts cmd if !$verbose\n    puts \"!!! Command returned non-zero exit code: #{$?}\"\n    puts \"!!! Working dir: #{Dir.pwd}\"\n    exit $?\n  end\n  return res\nend\n\nmodule Shell\n  # Escape string string so that it is parsed to the string itself\n  # Compare to Regexp.escape .\n  def self.escape(string)\n    string !~ %r{[ \"\\\\]}i ? \n      string : '\"' + string.gsub(%r{([\"\\\\])}i, '\\\\\\\\\\1') + '\"'\n  end\nend\n\ndef svn_common_args(branch = \"\")\n  repo_url = $repo_url\n  repo_url += \"/\" + branch if branch != \"\"\n  res = \"--non-interactive\"\n  res += \" --username #{Shell.escape($username)}\" if $username\n  res += \" --password #{Shell.escape($password)}\" if $password\n  res += \" #{Shell.escape(repo_url)}\"\n  res\nend\n\ndef svn_get_current_revision\n  $stderr.print \"Retrieving current HEAD revision... \" if $verbose\n  # 'svn info' doesn't always work to get most recent revision.\n  # So parse output of 'svn log -r HEAD:HEAD'.\n  svn_info = `svn log --xml -r HEAD:HEAD #{svn_common_args}`\n  svn_info =~ %r{\\srevision=\\\"(\\d+)\\\"}m\n  $stderr.puts \"r#{$1}\" if $verbose\n  $1.to_i\nend\n\n# Prefix each line with \"!!!\" and only warn:\n#  - once for each unique warning\n#  - at most a fixed number of times per caller\n$warning_txt = Set.new\t\t# All dumped warnings texts\n$warning_callers = {}\t\t# Per caller: number of warnings\ndef warning(*txt)\n  txt = txt.flatten.collect { |t| t.split(\"\\n\") }.flatten\n  if !$warning_txt.include?(txt)\n    $warning_txt << txt\n    backtrace = caller[0]\n    $warning_callers[backtrace] ||= 0\n    times_warned = ($warning_callers[backtrace] += 1)\n    if times_warned <= 5\n      first = true\n      txt.each { |line|\n\t$stderr.puts((first ? \"!!!\" : \"   \") + \" #{line}\")\n\tfirst = false\n      }\n      if times_warned == 5\n\t$stderr.puts \n          \"    (more of this type of warnings will be suppressed)\"\n      end\n    end\n  end\nend\n\ndef die(*txt); warning(*txt); exit 1; end\n\nclass BranchColor\n  @@available_colors = [\n    \"green\", \"yellow\", \"orange\", \"cyan\", \"steelblue3\",\n    \"lightblue\", \"thistle\", \"red\", '\".7 .3 1.0\"',\n    \"navy\", \"violet\", \"crimson\", \"azure\", \"linen\", \"peru\",\n    \"tan\", \"darkgreen\", \"coral\"\n  ]\n  @@branch_color = {}\t\t# String branch -> String color\n\n  def self.color(branch)\n    # For a new branch, take a fixed color or a random one if out-of-colors.\n    @@branch_color[branch] ||= (\n      @@available_colors.shift ||\n      ('\"%.1f %.1f %.1f\"' % [rand + 0.1, rand + 0.1, rand + 0.1])\n    )\n  end\nend\n\n# Map a filename into a branch name part and element name part.\n# Branchname may contain '/''s.\n# Element name starts with a '/' or is empty.\n#\n# SVN does not contain branches, but only contains one large tree with\n# objects of (potentially different) versions. By only looking at a\n# subtree, a branch is emulated. However, we have to know where those\n# subtrees are rooted to be able to convert them to branches.\ndef path_to_branch(filename)\n  if filename =~ $branch_dirs\n    # We matched a branch-name.\n    branch, elem = $&, filename[$&.size..-1]\n    return nil if branch =~ $not_branch_dirs\n    return [branch[1..-1], elem] # Remove leading '/' from branch name\n  else\n    # Outside the branches (e.g. a tag). Skip this.\n    return nil\n  end\nend\n\n# Revision of a branch; part of a revision.\n# Forms a directed acyclic graph with other BranchRevisions\nclass BranchRevision\n  attr_accessor :must_add_implicit_dep\t# Boolean\n  attr_accessor :depends_on\t# Set of BranchRevision: parents\n  attr_accessor :dependers      # Set of BranchRevision: children\n  attr_reader :branch\t\t# String: branchname\n  attr_accessor :empty\t\t# Bool: true if no reason to keep this rev\n  attr_accessor :deleted\t# Bool: true if deleted at end\n  attr_accessor :commit_sha1\t# String. git's SHA1 commit hash.\n\n  # If true, implicitly adds an dependancy on the previous\n  # version of this branch. This the the default.\n  # However, when we detect that this branch consists of totally new files\n  # (for example, when copying /trunk to /branches/branch_name) we set\n  # this to false.\n  # @must_add_implicit_dep  # Boolean\n\n  def initialize(revision, branch)\n    @revision = revision\t# Revision\n    @branch = branch\n    @must_add_implicit_dep = true\n    @depends_on = Set.new\n    @dependers = Set.new\n    @empty = true\n    @deleted = false\n    @commit_sha1 = nil\n  end\n\n  # Compare on revision number\n  def <=>(other); self.nr <=> other.nr; end\n\n  def to_s; \"#{@branch}:#{nr}\"; end\n\n  def nr; @revision.nr; end\t# Fixnum: revision nr\n\n  # Returns true if this BranchRevision is a root revision since:\n  #   - it does not depend on another BranchRevision\n  #   - it contains changes _within_ the branch (== not empty)\n  def root?; @depends_on.empty? && !@empty; end\n\n  def add_depends_on(branch_rev)\n    @depends_on << branch_rev\n    branch_rev.dependers << self\n  end\n\n  def remove_depends_on(branch_rev)\n    @depends_on.delete(branch_rev)\n    branch_rev.dependers.delete(self)\n  end\n\n  def rev; @revision; end\n\n  def other_branch_depends_on\n    @depends_on.find_all { |b| b.branch != @branch }\n  end\n\n  # Returns all BranchRevisions we depend on in the same branch\n  def same_branch_depends_on\n    @depends_on.find_all { |b| b.branch == @branch }\n  end\n\n  # Returns branch revision on which we depend which is least number\n  # of revisions back on the same branch. Since we go back, this is\n  # the max. revision of the dependancies.\n  def closest_same_branch_depends_on\n    same_branch_depends_on.max\n  end\n\n  # Returns branch revision which depends on us which is least number\n  # of revisions forward on the same branch.\n  def closest_same_branch_depender\n    @dependers.find_all { |b| b.branch == @branch }.min\n  end\n\n  # A dependancy is unneeded when:\n  #   - the dependancy belongs to the same branch and the same revs\n  #     can be reached by removed this dependancy, or\n  #   - the dependancy belongs to another branch on which a rev in\n  #     the same branch above us is already depending\n  def remove_unneeded_depends_on\n    return if @depends_on.size <= 1 # Optimisation\n\n    # TODO: look farther back.\n\n    # Same branch dependancy remover:\n    # Follow the dependancy chain along the closest parent\n    # till we reach the farthest dep.\n    same_branch_deps = same_branch_depends_on()\n    closest_same_branch_dep = same_branch_deps.max\n    farthest_same_branch_nr = !same_branch_deps.empty? && same_branch_deps.min.nr\n    current_dep = closest_same_branch_dep\n    same_branch_deps_rec = Set.new\n    while current_dep && current_dep.nr >= farthest_same_branch_nr\n      same_branch_deps_rec.add(current_dep)\n      current_dep = current_dep.closest_same_branch_depends_on\n    end\n    same_branch_deps.each { |red_rev|\n      if red_rev != closest_same_branch_dep &&\n\t  same_branch_deps_rec.include?(red_rev)\n#\tputs \"r#{nr}: Dep. on same branch r#{red_rev.nr} redundant\" if $verbose\n\tremove_depends_on(red_rev)\n      end\n    }\n\n    # Via-other-branch redundant dependancy remover:\n    other_branch_depends_on.each { |other_dep|\n      if !((other_dep.dependers & same_branch_deps_rec).empty?)\n#\tputs \"r#{nr}: Dep. on other branch r#{other_dep.nr} redundant\" if $verbose\n\tremove_depends_on(other_dep)\n      end\n    }\n  end\n\n  # Signal the fact that the last action has been added to this revision.\n  # Now the dependancies can be calculated.\n  def last_action_added\n    if @must_add_implicit_dep\n      prev_branch_rev = Revision.find_revision_with_branch(nr - 1, @branch)\n      add_depends_on(prev_branch_rev) if prev_branch_rev\n    end\n    @depends_on.dup.each { |br|\n      # We might depend on empty branch revisions. Since empty branch will not\n      # be checked out, copy dependancies from those branches.\n      if br.empty\n\tbr.depends_on.each { |parent_depends_on|\n\t  add_depends_on(parent_depends_on)\n\t}\n\tremove_depends_on(br)\n      end\n    }\n    remove_unneeded_depends_on\n  end\n\n  # Internal dotty label.\n  def label; \"_#{@branch}_#{nr}\".delete(\"^0-9a-zA-Z_\"); end\n\n  def to_dotty\n    return \"\\t/* r#{nr} is empty */\\n\" if @empty\n    res = \"\\t#{label}[label=\\\"#{File.basename(@branch)} \" +\n      \"#{nr}\\\\n#{rev.author}\\\",color=#{BranchColor.color(@branch)}];\"\n    @depends_on.each { |d|\n      res += \" #{d.label} -> #{label}\"\n      res += \"[style=dashed]\" if @branch != d.branch # A merge or branch\n      res += \";\"\n    }\n    res + \"\\n\"\n  end\nend\n\n# One revision in SVN. Contains zero or more BranchRevisions.\nclass Revision\n  attr_reader :nr\t\t# Fixnum\n  attr_accessor :author\t\t# String\n  attr_accessor :msg\t\t# String\n  attr_accessor :time\t\t# String\n  attr_accessor :branches\t# Hash: String rootdir to BranchRevision\n\n  @@all_revs = []\t\t# Array: all revisions, indexed by nr\n\n  def self.get_all_revs; @@all_revs; end\n\n  def self.set_all_revs(new_all_revs); @@all_revs = new_all_revs; end\n\n  # Returns first revision number we're interested in.\n  def self.get_next_log_nr\n    @@all_revs[-1] ? @@all_revs[-1].nr + 1 : $start_revision\n  end\n\n  def self.[](nr); @@all_revs[nr]; end\n  # Iterate sorted over each revision\n  def self.each; @@all_revs.each { |rev| yield(rev) if rev }; end\n\n  # Search for most recent BranchRevision which changed given branch.\n  def self.find_revision_with_branch(start_nr, branch)\n    start_nr.downto($start_revision) { |nr|\n      rev = @@all_revs[nr]\n      return rev.branches[branch] if rev && rev.branches.has_key?(branch)\n    }\n    warning(\n      \"Could not find branch #{branch.inspect} from revision #{start_nr} back\"\n    ) if start_nr >= $start_revision\n    return nil\n  end\n\n  def initialize(nr)\n    @nr = nr\n    @branches = {}\n    # Store new revision in global revision array\n    @@all_revs[nr] = self\n  end\n\n  # Add an action to this revision\n  #   'copyto_path' is the destination being changed. In all cases,\n  #   this can be a file or directory.\n  #   action == :R  : Replace. copyfrom_* always filled in.\n  #   action == :M  : Modify\n  #   action == :D  : Delete\n  #   action == :A  : Create as new or copy from other revision\n  #                   If from other revision, copyfrom_* are filled in.\n  $branch_paths = Set.new\n  def add_action(action, copyto_path, copyfrom_path, copyfrom_rev)\n    branch, elem = *path_to_branch(copyto_path)\n    return if !branch\t\t# Action is outside a branch\n\n    if $verbose && !$branch_paths.include?(branch)\n      puts \"r#{nr}: New branch: #{branch}\"\n      $branch_paths.add(branch)\n    end\n\n    branch_rev = (@branches[branch] ||= BranchRevision.new(self, branch))\n\n    from_branch = from_elem = nil\n    if copyfrom_path\n      from_branch, from_elem = *path_to_branch(copyfrom_path)\n      if !from_branch\n\t# Not a branch. Must be a tag.\n\twarning(\n          \"Ignoring dependancy r#{@nr} on a non-branch: \" + \n          \"#{copyfrom_path.inspect}:#{copyfrom_rev}\"\n        )\n      else\n\tfrom_branch_rev = \n\t  Revision.find_revision_with_branch(copyfrom_rev, from_branch)\n\tbranch_rev.add_depends_on(from_branch_rev) if from_branch_rev\n      end\n    end\n\n    # We cannot checkout deleted branches, so record the fact that\n    # it is deleted at the end of the revision.\n    branch_rev.deleted = true if elem == \"\" && action == :D\n\n    # If something changes the root of the branch (deleted, copied\n    # from other branch, ...) we break the implicit dependancy chain.\n    # However, _modifying_ the root directory (adding files, removing files)\n    # does not remove the dependancy.\n    branch_rev.must_add_implicit_dep = false if elem == \"\" && action != :M\n\n    # Check whether this action results in a revision. For example,\n    # deleting the root of the branch or creating the root dir does not\n    # change anything _within_ the branch.\n    branch_rev.empty = false if elem != \"\" # Real files. Not empty.\n  end\n\n  # Signal the fact that the last action has been added to this revision.\n  # Now the dependancies can be calculated.\n  def last_action_added\n    @branches.each_value { |b| b.last_action_added }\n  end\n\n  def to_dotty\n    @branches.values.collect { |b| b.to_dotty }.join(\"\")\n  end\nend\n\n# Parse output of 'svn log --xml --verbose'\ndef parse_svn_log_xml(svnlog_filename)\n  $stderr.puts \"Branch analysis on svn log files...\" if $verbose\n  last_rev_nr = nil\n  rev = nil\t\t\t# Current Revision\n  File.open(svnlog_filename, \"rb\") { |io|\n    while line = io.gets\n      case line\n      when %r{^   revision=\"(\\d+)\"}\n\trev_nr = $1.to_i\n\t$stderr.print \"r#{rev_nr}\" + (\"\\010\" * 10)\n\t# Only create a new revision when not already loaded from\n\t# state file.\n\tif rev_nr > $start_revision && Revision[rev_nr] == nil\n\t  rev = Revision.new(rev_nr)\n\t  if last_rev_nr && (rev_nr != last_rev_nr + 1)\n\t    # Missing revisions are a sign of not getting the log of the\n\t    # whole archive, but only one of its branches (or subprojects).\n\t    # Which could be valid if we're after a subproject, but not\n\t    # if we want to import the whole repository.\n\t    warning(\"Missing reversion(s) #{last_rev_nr + 1}-#{rev_nr - 1}\")\n\t  end\n\t  last_rev_nr = rev_nr\n\tend\n      when %r{^\\<path}: copyfrom_path = nil; copyfrom_rev = nil\n      when %r{^   copyfrom-path=\"(.*)\"}: copyfrom_path = $1\n      when %r{^   copyfrom-rev=\"(\\d+)\"}: copyfrom_rev = $1.to_i\n      when %r{^   action=\"([A-Z])\"\\>(.*)\\</path\\>}\n\trev.add_action($1.to_sym, $2, copyfrom_path, copyfrom_rev) if rev\n      when %r{^\\</paths\\>}\n\t# End of this revision.\n\tnext if !rev\n\trev.last_action_added\n\troot_branches = rev.branches.values.find_all { |branch| branch.root? }\n\troot_branches.each { |bi|\n\t  puts \"r#{rev.nr} is root of #{bi.branch}\"\n\t}\n\tchanged_branches = rev.branches.values.find_all { |b| !b.empty }\n\tif changed_branches.size > 1\n\t  changed_branches_names = changed_branches.collect { |b| b.branch }\n\t  warning(\n            \"More than one branch changed in r#{rev.nr}..?\",\n            changed_branches_names\n          )\n\tend\n      when %r{^\\<author\\>(.*)\\</author\\>}\n\trev.author = $1 if rev\n      when %r{^\\<msg\\>(.*)}\n\tmsg = $1.rstrip\n\twhile !msg.gsub!(\"\\</msg\\>\", \"\")\n\t  msg += \"\\n\" + io.gets.rstrip\n\tend\n\tmsg = CGI::unescapeHTML(msg.strip + \"\\n\") # &amp; -> & etc.\n\trev.msg = msg if rev\n      when %r{^\\<date\\>(.*)\\</date\\>}\n\trev.time = Time.parse($1) if rev\n      end\n    end\n  }\nend\n\n$state_version = \"20060409\"\n\n# Load state of git-svnconvert. Returns true in case of succes.\ndef load_state\n  begin\n    state = File.open($state, \"rb\") { |io| Marshal.load(io) }\n    version = state.shift\n    if version != $state_version\n      warning(\n        \"Ignoring previous state: has different version (#{version}) \" +\n        \"than this git-svnconvert version (#{$state_version})\"\n      )\n    else\n      $repo_url, all_new_revs = *state\n      Revision.set_all_revs(all_new_revs)\n      return true\n    end\n  rescue Errno::ENOENT\n  end\n  return false\nend\n\ndef save_state\n  start_time = Time.now\n  File.open($state + \".new\", \"wb\") { |io|\n    io.write(Marshal.dump([$state_version, $repo_url, Revision.get_all_revs]))\n  }\n  # Rename atomically new state into _the_ state\n  File.rename($state + \".new\", $state)\n  $stderr.puts \"Saved new state in %.1fs\" % (Time.now - start_time) if $verbose\nend\n\n# Export current graph of revisions to a GraphViz dotty file\ndef export_to_dot(dot_file)\n  $stderr.puts \"Writing branch graph to \\\"#{dot_file}\\\"...\"\n  File.open(dot_file, \"wb\") { |io|\n    io.puts \"strict digraph svn {\"\n    io.puts \"\\tnode[shape=box,style=filled];\\n\\n\"\n    Revision.each { |rev|\n      next if !rev || (rev.nr < $start_revision)\n      io.print rev.to_dotty\n    }\n    io.puts \"}\"\n  }\nend\n\n# Runs an svn command, and run git-update-index on each file\n# echoed to stdout by svn.\ndef svn_cmd_with_update_index(svn_cmd)\n  safe_popen(svn_cmd, \"r\") { |svn_io|\n    safe_popen(\"git-update-index --add --remove --stdin\", \"w\") { |git_io|\n      git_io.sync = true\n      # Have a buffer of one line: svn echoes the file it is\n      # working on. When svn echoes the next line, we then know\n      # it finished the previous file, and can therefore be added\n      # to git now.\n      last_file = nil\n      while line = svn_io.gets\n\t$stderr.puts line if $verbose\n\tif line =~ %r{^[A-Z][A-Z ]{3} (.*)$}\n\t  git_io.puts last_file if last_file \n\t  last_file = $1\n\tend\n      end\n      git_io.puts last_file if last_file \n    }\n  }\nend\n\n# Unknown contents in directory; recreate index\ndef git_update_index\n  # Remove incomplete index\n  git_index_file = ENV['GIT_INDEX_FILE']\n  [git_index_file, git_index_file + \".lock\"].each { |f|\n    File.delete(f) if File.exist?(f)\n  }\n  # Add all files\n  to_prune = Set.new([\".git\", \".svn\"]) # Directories not to add\n  safe_popen(\"git-update-index --add --remove --stdin\") { |io|\n    elems = 0\n    Find.find(\".\") { |elem|\n      Find.prune if to_prune.include?(File.basename(elem))\n      if elem != \".\"\n\telem = elem[2..-1]  # Remove ./\n\tio.puts elem\n\telems += 1\n      end\n    }\n    puts \"Dir contained #{elems} elements\" if $verbose\n  }\nend\n\n######################################## Main\n\n# Default values for options\n$verbose = true\n$username = nil\t\t\t# Repository username\n$password = nil\t\t\t# Repository password\n$start_revision = 1\ndot_filename = nil\nsvn_authors_file = nil\t\t# Mapping from \n$run_dotty = false\n\n# Parse options\n$opts = OptionParser.new\n$opts.banner = %q{Convertor for a subversion archive into a git archive.\n\ngit-svnconvert [options] URL[@REV] [DIR]\n  ..to convert complete SVN archive at URL into directory DIR.\n  The URL must point to the root directory of the repo, so not f.e. to /trunk!\n  By default, DIR will be the basename of the URL.\n  Directory DIR will be created if it does not exist.\n  Otherwise, DIR must already be a git repository.\ngit-svnconvert [options]\n  ..to incrementally convert newly added revision and add them\n    to the git repo in the current directory.\n\nExamples:\n\ngit-svnconvert svn://svn.0pointer.net/fusedav .\n  ..checks out svn://svn.0pointer.net/fusedav into current dir.\ngit-svnconvert svn://svn.0pointer.net/fusedav@10\n  ..checks out starting at revision 10 into directory fusedav.\ngit-svnconvert\n  ..updates already imported svn archive in current dir.\n\n}\n$opts.on(\"Options (defaults between []'s):\")\n$opts.on(\"--verbose\", \"-v\", \"Toggle verbose mode [#{$verbose}]\") {\n  $verbose = !$verbose\n}\n$opts.on(\"--help\", \"-h\", \"This usage\") { puts $opts; exit 1 }\n$opts.on(\"--revision REV\", \"-s\", \"Start revision [#{$start_revision}]\") {\n  |$start_revision|\n}\n$opts.on(\"--dot FILENAME\", \"-d\", \"Export branch tree as .dot file.\") {\n  |dot_filename|\n}\n$opts.on(\"--dotty\", \"-D\", \"Export branch tree and run dotty on it\") {\n  $run_dotty = true\n}\n$opts.on(\"--authors FILENAME\", \"-A\", \"Filename of svnauthors file to add\") {\n  |svn_authors_file|\n}\n$opts.on(\"\\nOptions passed to 'svn':\");\n$opts.on(\"--username NAME\", \"-u\", \"Specify a username for SVN repo\") {\n  |$username|\n}\n$opts.on(\"--password PWD\", \"-p\", \"Specify a password for SVN repo\") {\n  |$password|\n}\n\nbegin\n  $opts.parse!(ARGV)\nrescue OptionParser::InvalidOption\n  die($!.to_s, $opts.to_s)\nend\n\n$repo_url = (ARGV.size > 0) ? ARGV.shift.dup : nil\n# Parse @REV part of URL@REV if given...\n$start_revision = $1.to_i if $repo_url && $repo_url.gsub!(%r{@(\\d+)$}, \"\")\ndest_dir = (ARGV.size > 0) ? ARGV.shift : nil\ndest_dir = File.basename($repo_url) if $repo_url && !dest_dir\ndest_dir ||= Dir.pwd\t# No URL and no destdir given: use current dir\ndest_dir = File.expand_path(dest_dir)\ngit_dir = ENV[\"GIT_DIR\"] || \"#{dest_dir}/.git\"\nsvnconvert_dir = \"#{git_dir}/svnconvert\"\n# Name of state file with all parsed log files and connection to git archive.\n$state = \"#{svnconvert_dir}/state\"\n\nif !File.exist?($state) && !$repo_url\n  puts $opts.to_s\n  puts \"\\n!!! Need an URL to checkout or a previously git-svnconvert'ed dir\"\n  exit 1\nend\n\n# Maping from SVN author name to full name and email\ndefault_svn_authors_file = \"#{git_dir}/svn-authors\"\nusers = File.readable?(default_svn_authors_file) ?\n  read_svn_authors(default_svn_authors_file) : {}\n# If authors file explicitly given, add\nif svn_authors_file\n  users_to_add = read_svn_authors(svn_authors_file)\n  if users_to_add.size > 0\n    users.merge!(users_to_add) \n    write_svn_authors(users, default_svn_authors_file)\n  end\nend\n\nFileUtils.mkdir_p(svnconvert_dir)\t# Create destination if not existing\nDir.chdir(dest_dir)\n\n$branch_dirs = root_dirs_to_regexp($branch_dirs, true)\n$not_branch_dirs = root_dirs_to_regexp($not_branch_dirs, false)\n\n# Load complete state, or failing that, same empty state with new repo URL\nload_state || save_state\nnext_revision_nr = Revision.get_next_log_nr\nif next_revision_nr < svn_get_current_revision\n  svnlog = \"#{svnconvert_dir}/svnlog.xml\"\n  puts \"Retrieving revision log from #{next_revision_nr} and upwards...\"\n  cmd = \"svn log #{svn_common_args} -r #{next_revision_nr}:HEAD --xml --verbose >#{svnlog}\"\n  safe_system(cmd)\n  parse_svn_log_xml(svnlog)\n  save_state\nelse\n  puts \"No new revision to get log of.\"\nend\n\n# The graph eases debugging, so export it always...\ndefault_dot_filename = \"#{svnconvert_dir}/branches.dot\"\nexport_to_dot(dot_filename || default_dot_filename)\nif $run_dotty\n  safe_system(\"dotty #{Shell.escape(dot_filename || default_dot_filename)}\")\nend\nexit 0 if dot_filename || $run_dotty\n\n# Directory of all branches checked out:\nsvn_co_dir = \"#{svnconvert_dir}/checkedout\" \nENV['GIT_DIR'] = git_dir\n\nhead_existed = File.exist?(\"#{git_dir}/HEAD\")\nsafe_system(\"git-init-db\")\nif !head_existed\n  $stderr.puts \"Creating HEAD pointing to trunk\"\n  File.open(\"#{git_dir}/HEAD\", \"wb\") { |io|\n    io.puts \"ref: refs/heads/trunk\"\n  }\nend\nRevision.each { |rev|\n  next if rev.nr < $start_revision\n\n  rev_commit_sha1s = []\n  branches = rev.branches.values\n  branches.each { |br|\n    branch_dir = \"#{svn_co_dir}/#{br.branch}\"\n    if br.commit_sha1\n      # Skip if already imported\n    elsif br.deleted\n      # Branch contains only an 'delete this branch' command.\n      $stderr.puts \"r#{rev.nr}: #{br.branch} deleting #{branch_dir}\"\n#      FileUtils.rm_rf(branch_dir)\n    elsif br.empty\n      $stderr.puts \"r#{rev.nr}: #{br.branch} is empty; skipping...\"\n    else\n      # Hide the git index inside .svn dir.\n      git_index_file = ENV['GIT_INDEX_FILE'] = \"#{branch_dir}/.svn/index.git\"\n      if !File.directory?(branch_dir)\n        # Start of new branch. Make (empty) directory to start new branch in\n        FileUtils.mkdir_p(branch_dir)\n\tFileUtils.mkdir_p(File.dirname(ENV['GIT_INDEX_FILE']))\n\tDir.chdir(branch_dir)\n        puts \"Checking out new branch #{br.branch}:#{rev.nr} in #{Dir.pwd}\"\n        cmd = \"svn checkout -r #{rev.nr} #{svn_common_args(br.branch)} .\"\n        svn_cmd_with_update_index(cmd)\n      else\n\t# Branched already checked out, update to new revision.\n\tDir.chdir(branch_dir)\n\tsvn_cmd = \"svn update -r #{rev.nr} .\"\n\tif File.exist?(git_index_file + \".lock\")\n\t  $stderr.puts \"Git index file already locked. Removing lock and recreating index.\"\n\t  safe_system(svn_cmd)\n\t  git_update_index\n\telse\n\t  puts \"Updating branch to new revision: #{br.branch}:#{rev.nr}\"\n\t  svn_cmd_with_update_index(svn_cmd)\n\tend\n      end\n\n      # git index is now up-to-date. Write the tree.\n      tree_sha1 = `git-write-tree`.chomp\n\n      # Now write this tree into an commit\n      author = rev.author\n      email = \"unknown\"\n      author, email = users[author] if users[author]\n      $stderr.puts \"r#{rev.nr}: Author: #{author} <#{email}>\" if $verbose\n      ENV['GIT_AUTHOR_NAME'] = ENV['GIT_COMMITTER_NAME'] = author\n      ENV['GIT_AUTHOR_EMAIL'] = ENV['GIT_COMMITTER_EMAIL'] = email\n      ENV['GIT_AUTHOR_DATE'] = ENV['GIT_COMMITTER_DATE'] =\n\trev.time.strftime(\"+0000 %Y-%m-%d %H:%M:%S\") \n      parents = br.depends_on.find_all { |p|\n\t# Filter out all non-converted parents\n\traise if !p.commit_sha1 && p.nr >= $start_revision\n\tp.commit_sha1 != nil\n      }.collect { |p| \"-p #{p.commit_sha1}\" }.join(\" \")\n      puts \"Committing #{tree_sha1}...\"\n      cmd = \"git-commit-tree #{tree_sha1} #{parents}\"\n      puts cmd if $verbose\n      safe_popen(cmd) { |io|\n        io.puts rev.msg.strip\n        io.close_write\n        br.commit_sha1 = io.read.chomp\n        rev_commit_sha1s << br.commit_sha1    \n      }\n      puts \"#{br.branch}:#{rev.nr} has id #{br.commit_sha1}\" if $verbose\n\n      Dir.chdir(dest_dir)\n      # Tag current branch + revision.\n#    safe_system(\"git-tag -f #{br.branch}/r#{rev.nr} #{br.commit_sha1}\")\n      # Update HEAD of this branch\n      safe_system(\"git-update-ref refs/heads/#{br.branch} #{br.commit_sha1}\")\n      puts\n    end\n  }\n  if rev_commit_sha1s.size == 1\n    # '-f' added to make this script restartable (state might not get saved)\n    safe_system(\"git-tag -f r#{rev.nr} #{rev_commit_sha1s}\")\n  end\n  save_state if !rev_commit_sha1s.empty?\n}\nputs \"Done.\"\n"},{"id":"18500","messageId":"e1bdjq$qf6$1@sea.gmane.org","threadId":"3840","inReplyTo":"20060409163426.GA22103@nospam.com","subject":"Re: [ANNOUNCE] git-svnconvert: YASI (Yet Another SVN importer)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-04-09T16:43:53Z","receivedAt":"2006-04-09T16:43:53Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Rutger Nijlunsing wrote:\n\n> Since I didn't succeed in importing a (private) SVN repo into git, I\n> wrote a new converter to handle more cases like:\n\nBoth git-svn[*1*] and git-svnimport failed? Have you tried Tailor tool:\n  http://www.darcs.net/DarcsWiki/Tailor\n\n>   - use command line svn instead of some perl library to have less\n>     dependancies and to support proxy + repo authentification.\n>     Might even work on MacOSX ;)\n\nInstead adding dependence on Ruby, eh?\n\n\nReferences\n----------\n[*1*] contrib/git-svn and http://git-svn.yhbt.net/\n-- \nJakub Narebski\nWarsaw, Poland\n"},{"id":"18516","messageId":"20060409211505.GA30567@nospam.com","threadId":"3840","inReplyTo":"e1bdjq$qf6$1@sea.gmane.org","subject":"Re: [ANNOUNCE] git-svnconvert: YASI (Yet Another SVN importer)","fromName":"Rutger Nijlunsing","fromEmail":"rutger@nospam.com","sentAt":"2006-04-09T21:15:05Z","receivedAt":"2006-04-09T21:15:05Z","isPatch":false,"sender":{"key":"rutger.nijlunsing@gmail.com","avatar":null},"body":"On Sun, Apr 09, 2006 at 06:43:53PM +0200, Jakub Narebski wrote:\n> Rutger Nijlunsing wrote:\n> \n> > Since I didn't succeed in importing a (private) SVN repo into git, I\n> > wrote a new converter to handle more cases like:\n> \n> Both git-svn[*1*] and git-svnimport failed? Have you tried Tailor tool:\n>   http://www.darcs.net/DarcsWiki/Tailor\n\ngit-svn and tailor can only track one branch (or trunk). As the\ngit-svn page states, it is for contributing to such a branch /\ntrunk. git-svnconvert is for converting a whole repository\nincrementally of which branches (IMHO) are important to keep and\nconvert.\n\ngit-svnimport does handle multiple branches, but could not cope with\nproxy + repo authentification, the weird repo layout I've had to cope\nwith (branches not only in /branches, several trunks) and some\nrevisions which contain non-sensical actions.\n\n> >   - use command line svn instead of some perl library to have less\n> >     dependancies and to support proxy + repo authentification.\n> >     Might even work on MacOSX ;)\n> \n> Instead adding dependence on Ruby, eh?\n\nTake some, lose some ;)\n\nSeriously, though, a dependancy on a mainstream language like\nPython/Perl/Ruby/.. isn't a problem since a package is available for\nall distributions. However, packages for mainstream languages are\nquite often out-of-date or are not supported at all. Seeing a program\nbeing dependant on a non-packaged module is enough for a truckload of\npeople to not even try it.\n\n-- \nRutger Nijlunsing ---------------------------------- eludias ed dse.nl\nnever attribute to a conspiracy which can be explained by incompetence\n----------------------------------------------------------------------\n"},{"id":"18517","messageId":"Pine.LNX.4.63.0604092325590.29434@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"3840","inReplyTo":"20060409211505.GA30567@nospam.com","subject":"Re: [ANNOUNCE] git-svnconvert: YASI (Yet Another SVN importer)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-04-09T21:30:23Z","receivedAt":"2006-04-09T21:30:23Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 9 Apr 2006, Rutger Nijlunsing wrote:\n\n> On Sun, Apr 09, 2006 at 06:43:53PM +0200, Jakub Narebski wrote:\n> > \n> > Instead adding dependence on Ruby, eh?\n> \n> Take some, lose some ;)\n> \n> Seriously, though, a dependancy on a mainstream language like\n> Python/Perl/Ruby/.. isn't a problem since a package is available for\n> all distributions. However, packages for mainstream languages are\n> quite often out-of-date or are not supported at all. Seeing a program\n> being dependant on a non-packaged module is enough for a truckload of\n> people to not even try it.\n\nI have _never_ seen a setup where Ruby was installed by default. Perl \nalways, Python often.\n\nFurthermore, my feeling is that we are in the beginning phase of migration \nfrom scripting languages (which are good for prototyping) towards plain C. \nSo adding yet another scripting language dependency is a little backwards.\n\nCiao,\nDscho\n"},{"id":"18519","messageId":"86mzeuflog.fsf@blue.stonehenge.com","threadId":"3840","inReplyTo":"Pine.LNX.4.63.0604092325590.29434@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [ANNOUNCE] git-svnconvert: YASI (Yet Another SVN importer)","fromName":"Randal L. Schwartz","fromEmail":"merlyn@stonehenge.com","sentAt":"2006-04-09T22:06:55Z","receivedAt":"2006-04-09T22:06:55Z","isPatch":false,"sender":{"key":"merlyn@stonehenge.com","avatar":"https://gravatar.com/avatar/dc528d210743ff0333e6213f9ee7b33b23f1b7bc1f3c5a8c2d819074ecd7ab19?d=mp&s=160"},"body":">>>>> \"Johannes\" == Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\nJohannes> I have _never_ seen a setup where Ruby was installed by\nJohannes> default. Perl always, Python often.\n\nOSX includes ruby by default.\n\nJohannes> Furthermore, my feeling is that we are in the beginning phase of\nJohannes> migration from scripting languages (which are good for prototyping)\nJohannes> towards plain C.  So adding yet another scripting language\nJohannes> dependency is a little backwards.\n\nYou seem a bit prejudiced here.  Are there performance problems in\nthe Perl and python parts of git?  If so, concentrate first on optimizing\nthe code where it matters.  Then, creating bindings to the \"git lib\"\nso that the heavy lifting can be done in C while still providing for\nthe basic algorithms to be written in a higher level language.\n\nIt would be a step *backwards* to recode all of git in C.\n\nNow, the *shell* parts, on the other hand, are screaming for a rewrite into\nPerl or Python.  fork-fork-fork and worrying about escaping special characters\nneedlessly burns a lot of cpu and programmer time.\n\n-- \nRandal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095\n<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>\nPerl/Unix/security consulting, Technical writing, Comedy, etc. etc.\nSee PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!\n"}]}