{"thread":{"id":"25374","subject":"[PATCH] contrib/svn-fe: Fast script to remap svn history","startedAt":"2010-10-07T06:06:01Z","lastAt":"2010-11-22T17:42:42Z","messageCount":7,"participants":["David Barr","Sverre Rabbelier","David Michael Barr","Jonathan Nieder","Stephen Bash"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"152849","messageId":"1286431561-24126-1-git-send-email-david.barr@cordelta.com","threadId":"25374","inReplyTo":null,"subject":"[PATCH] contrib/svn-fe: Fast script to remap svn history","fromName":"David Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-10-07T06:06:01Z","receivedAt":"2010-10-07T06:06:01Z","isPatch":true,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"This python script walks the commit sequence imported by svn-fe.\nFor each commit, it tries to identify the branch that was changed.\nCommits are rewritten to be rooted according to the standard layout.\nA basic heuristic of matching trees is used to find parents for the\nfirst commit in a branch and for tags.\n\nSigned-off-by: David Barr \n---\n contrib/svn-fe/svn-filter-root.py | 107 +++++++++++++++++++++++++++++++++++++\n fast-import.c | 9 +++\n 2 files changed, 116 insertions(+), 0 deletions(-)\n create mode 100755 contrib/svn-fe/svn-filter-root.py\n\ndiff --git a/contrib/svn-fe/svn-filter-root.py b/contrib/svn-fe/svn-filter-root.py\nnew file mode 100755\nindex 0000000..72d248f\n--- /dev/null\n+++ b/contrib/svn-fe/svn-filter-root.py\n@@ -0,0 +1,107 @@\n+#!/usr/bin/python\n+from subprocess import *\n+import re\n+import os\n+\n+subroot_re = re.compile(\"^trunk|^branches/[^/]*|^tags/[^/]*\") \n+\n+tree_re = re.compile(\"^tree ([0-9a-f]{40})\", flags=re.MULTILINE)\n+parent_re = re.compile(\"^parent ([0-9a-f]{40})\", flags=re.MULTILINE)\n+author_re = re.compile(\"^author (.*)$\", flags=re.MULTILINE)\n+committer_re = re.compile(\"^committer (.*)$\", flags=re.MULTILINE)\n+\n+git_svn_id_re = re.compile(\"^git-svn-id[^@]*\", flags=re.MULTILINE)\n+\n+ref_commit = {}\n+tree_commit = {}\n+count = 1\n+\n+# Open a cat-file process for subtree lookups\n+subtree_process = Popen([\"git\",\"cat-file\",\"--batch-check\"], stdin=PIPE, stdout=PIPE)\n+\n+# Iterate over commits from subversion imported with svn-fe\n+revlist = Popen([\"git\",\"rev-list\",\"--reverse\",\"--topo-order\",\"--default\",\"HEAD\"], stdout=PIPE)\n+cat_file = Popen([\"git\",\"cat-file\",\"--batch\"], stdin=revlist.stdout, stdout=PIPE)\n+object_header = cat_file.stdout.readline().strip().split(\" \");\n+while len(object_header) == 3:\n+ object_body = cat_file.stdout.read(int(object_header[2]))\n+ cat_file.stdout.read(1)\n+ git_commit = object_header[0]\n+ (commit_header, blank_line, commit_message) = object_body.partition(\"\\n\\n\")\n+ object_header = cat_file.stdout.readline().strip().split(\" \");\n+\n+ author = author_re.search(commit_header).group()\n+ committer = committer_re.search(commit_header).group()\n+\n+ # Diff against the empty tree if no parent\n+ match = parent_re.search(commit_header)\n+ if match:\n+ parent = match.group(1)\n+ else:\n+ parent = \"4b825dc642cb6eb9a060e54bf8d69288fbee4904\"\n+\n+ # Find a common path prefix in the changes for the revision\n+ subroot = \"\"\n+ changes = Popen([\"git\",\"diff\",\"--name-only\",parent,git_commit], stdout=PIPE)\n+ for path in changes.stdout:\n+ match = subroot_re.match(path)\n+ if match:\n+ subroot = match.group()\n+ changes.terminate()\n+ break\n+\n+ # Attempt to rewrite the commit on top of the matching branch\n+ if subroot == \"\":\n+ print \"progress Weird commit - no subroot.\"\n+ else:\n+ # Rewrite git-svn-id in the log to point to the subtree\n+ commit_message = git_svn_id_re.sub('\\g<0>/'+subroot, commit_message)\n+ subtree_process.stdin.write(git_commit+\":\"+subroot+\"\\n\")\n+ subtree_process.stdin.flush()\n+ subtree_line = subtree_process.stdout.readline()\n+ if re.match(\"^.*missing$\", subtree_line):\n+ print \"progress Weird commit - invalid subroot\"\n+ continue\n+ subtree = subtree_line[0:40]\n+ # Map the svn tag/branch name to a git-friendly one\n+\tref = \"refs/heads/\" + re.sub(\" \", \"%20\", subroot)\n+ # Choose a parent for the rewritten commit\n+ if ref in ref_commit:\n+ parent = ref_commit[ref]\n+ elif subtree in tree_commit:\n+ parent = tree_commit[subtree]\n+ else:\n+\t parent = \"\"\n+ # Update tags if necessary\n+ if re.match(\"^refs/heads/tags/\", ref):\n+ if parent == \"\":\n+ print \"progress Weird tag - no matching commit.\"\n+ else:\n+ tagname = ref[16:]\n+ print \"tag \"+tagname\n+ print \"from \"+parent\n+ print \"tagger \"+committer[10:]\n+ print \"data \"+str(len(commit_message))\n+ print commit_message\n+ else:\n+ # Default to trunk if the branch is new\n+ if parent == \"\" and \"refs/heads/trunk\" in ref_commit:\n+ parent = ref_commit[\"refs/heads/trunk\"]\n+ print \"commit \"+ref\n+ print \"mark :\"+str(count)\n+ print author\n+ print committer\n+ print \"data \"+str(len(commit_message))\n+ print commit_message\n+ if parent != \"\":\n+ print \"from \"+parent\n+ print \"M 040000 \"+subtree+\" \\\"\\\"\"\n+ commit = \":\"+str(count)\n+ # Advance the matching branch\n+ ref_commit[ref] = commit\n+ # Update latest commit by tree to drive parent matching\n+ tree_commit[subtree] = commit\n+ print \"progress \" + str(count)\n+ count = count + 1\n+\n+subtree_process.terminate()\ndiff --git a/fast-import.c b/fast-import.c\nindex 2317b0f..8f68a89 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1454,6 +1454,15 @@ static int tree_content_set(\n \t\tn = slash1 - p;\n \telse\n \t\tn = strlen(p);\n+\tif (!slash1 && !n) {\n+\t\tif (!S_ISDIR(mode))\n+\t\t\tdie(\"Root cannot be a non-directory\");\n+\t\thashcpy(root->versions[1].sha1, sha1);\n+\t\tif (root->tree)\n+\t\t\trelease_tree_content_recursive(root->tree);\n+\t\troot->tree = subtree;\n+\t\treturn 1;\n+\t}\n \tif (!n)\n \t\tdie(\"Empty path component found in input\");\n \tif (!slash1 && !S_ISDIR(mode) && subtree)\n-- \n1.7.3.4.g45608.dirty\n"},{"id":"152851","messageId":"AANLkTim_3m=YmyVL7eEoWDGwDwmuDJsYkQMgx8ciVPH0@mail.gmail.com","threadId":"25374","inReplyTo":"1286431561-24126-1-git-send-email-david.barr@cordelta.com","subject":"Re: [PATCH] contrib/svn-fe: Fast script to remap svn history","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-10-07T06:29:35Z","receivedAt":"2010-10-07T06:29:35Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Thu, Oct 7, 2010 at 08:06, David Barr wrote:\n> This python script walks the commit sequence imported by svn-fe.\n> For each commit, it tries to identify the branch that was changed.\n> Commits are rewritten to be rooted according to the standard layout.\n> A basic heuristic of matching trees is used to find parents for the\n> first commit in a branch and for tags.\n\nNice, how easy would it be to extend it to deal with other layouts?\n\n> diff --git a/fast-import.c b/fast-import.c\n> index 2317b0f..8f68a89 100644\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -1454,6 +1454,15 @@ static int tree_content_set(\n>                n = slash1 - p;\n>        else\n>                n = strlen(p);\n> +       if (!slash1 && !n) {\n> +               if (!S_ISDIR(mode))\n> +                       die(\"Root cannot be a non-directory\");\n> +               hashcpy(root->versions[1].sha1, sha1);\n> +               if (root->tree)\n> +                       release_tree_content_recursive(root->tree);\n> +               root->tree = subtree;\n> +               return 1;\n> +       }\n>        if (!n)\n>                die(\"Empty path component found in input\");\n>        if (!slash1 && !S_ISDIR(mode) && subtree)\n\nWhat is this hunk about?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"152853","messageId":"A609D4AA-04A8-4190-9CA9-3EF7F52BB93F@cordelta.com","threadId":"25374","inReplyTo":"AANLkTim_3m=YmyVL7eEoWDGwDwmuDJsYkQMgx8ciVPH0@mail.gmail.com","subject":"Re: [PATCH] contrib/svn-fe: Fast script to remap svn history","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-10-07T07:17:13Z","receivedAt":"2010-10-07T07:17:13Z","isPatch":true,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi,\n\n>> This python script walks the commit sequence imported by svn-fe.\n>> For each commit, it tries to identify the branch that was changed.\n>> Commits are rewritten to be rooted according to the standard layout.\n>> A basic heuristic of matching trees is used to find parents for the\n>> first commit in a branch and for tags.\n> \n> Nice, how easy would it be to extend it to deal with other layouts?\n\nI think its just a matter of adjusting the regular expression to match roots\nand the mapping from roots to refs.\n\n>> diff --git a/fast-import.c b/fast-import.c\n>> index 2317b0f..8f68a89 100644\n>> --- a/fast-import.c\n>> +++ b/fast-import.c\n>> @@ -1454,6 +1454,15 @@ static int tree_content_set(\n>> n = slash1 - p;\n>> else\n>> n = strlen(p);\n>> + if (!slash1 && !n) {\n>> + if (!S_ISDIR(mode))\n>> + die(\"Root cannot be a non-directory\");\n>> + hashcpy(root->versions[1].sha1, sha1);\n>> + if (root->tree)\n>> + release_tree_content_recursive(root->tree);\n>> + root->tree = subtree;\n>> + return 1;\n>> + }\n>> if (!n)\n>> die(\"Empty path component found in input\");\n>> if (!slash1 && !S_ISDIR(mode) && subtree)\n> \n> What is this hunk about?\n\nMy bad, that belongs in a separate commit. I'll break it out after review.\n\nThe subject would read: \"fast-import: Allow filemodify to set the root\".\n\n--\nDavid Barr\n"},{"id":"152859","messageId":"20101007082808.GC10523@burratino","threadId":"25374","inReplyTo":"AANLkTim_3m=YmyVL7eEoWDGwDwmuDJsYkQMgx8ciVPH0@mail.gmail.com","subject":"Re: [PATCH] contrib/svn-fe: Fast script to remap svn history","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-07T08:28:08Z","receivedAt":"2010-10-07T08:28:08Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Sverre Rabbelier wrote:\n> On Thu, Oct 7, 2010 at 08:06, David Barr wrote:\n\n>> --- a/fast-import.c\n>> +++ b/fast-import.c\n>> @@ -1454,6 +1454,15 @@ static int tree_content_set(\n>>                n = slash1 - p;\n>>        else\n>>                n = strlen(p);\n>> +       if (!slash1 && !n) {\n>> +               if (!S_ISDIR(mode))\n>> +                       die(\"Root cannot be a non-directory\");\n>> +               hashcpy(root->versions[1].sha1, sha1);\n>> +               if (root->tree)\n>> +                       release_tree_content_recursive(root->tree);\n>> +               root->tree = subtree;\n>> +               return 1;\n>> +       }\n>>        if (!n)\n>>                die(\"Empty path component found in input\");\n>>        if (!slash1 && !S_ISDIR(mode) && subtree)\n>\n> What is this hunk about?\n\nOoh, ack for this part (though I agree with you that it ought to be\nexplained in the log message).\n\nMost git commands do their writing to the object db via the index and\nloose objects. When you just have a pile of trees you want to convert\ninto commits, this is wasteful; for performance-critical operations\nlike filter-branch --subdirectory-filter, one might want a sort of\nhash-object --batch-to-pack to write a pack directly.\n\nFortunately we have fast-import (which is one of the only git commands\nthat will write to a pack directly) but there is not an advertised way\nto tell fast-import to use a given tree for its commits. So in\ncurrent git, one has the unpleasant choice of writing loose objects\nwithout parsing the trees or writing straight to pack but having to\nparse trees to do it.\n\nThis patch changes that, by allowing\n\n\tM 040000 \"\"\n\nas a filemodify line in a commit to reset to a particular tree without\nany need to unpack it. For example,\n\n\tM 040000 4b825dc642cb6eb9a060e54bf8d69288fbee4904 \"\"\n\nis a synonym for the deleteall command.\n"},{"id":"156281","messageId":"20101121051734.GA11856@burratino","threadId":"25374","inReplyTo":"1286431561-24126-1-git-send-email-david.barr@cordelta.com","subject":"Re: [PATCH] contrib/svn-fe: Fast script to remap svn history","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-21T05:17:34Z","receivedAt":"2010-11-21T05:17:34Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi David,\n\nDavid Barr wrote:\n\n> This python script walks the commit sequence imported by svn-fe.\n> For each commit, it tries to identify the branch that was changed.\n> Commits are rewritten to be rooted according to the standard layout.\n\nI like the idea and especially that the heuristics are simple.\n\nMaybe this could be made git-agnostic using the new ls-tree command\nyou are introducing in fast-import? Though it would need to get a\nrevision list from somewhere. Alternatively, do you think it would\nmake sense for something like this to be implemented as a filter or\nobserver of the fast-import stream as it is generated during an\nimport?\n\n> A basic heuristic of matching trees is used to find parents for the\n> first commit in a branch and for tags.\n\nMore precisely, the rule used is:\n\n> + # Find a common path prefix in the changes for the revision\n> + subroot = \"\"\n> + changes = Popen([\"git\",\"diff\",\"--name-only\",parent,git_commit], stdout=PIPE)\n> + for path in changes.stdout:\n> + match = subroot_re.match(path)\n> + if match:\n> + subroot = match.group()\n> + changes.terminate()\n> + break\n\nThe first change lying in one of\n\n\ttrunk\n\tbranch/*\n\ttags/*\n\ndetermines the branch. When a branch is renamed, this has a 50/50\nchance of choosing the right branch.\n\n> + # Choose a parent for the rewritten commit\n> + if ref in ref_commit:\n> + parent = ref_commit[ref]\n> + elif subtree in tree_commit:\n> + parent = tree_commit[subtree]\n> + else:\n> + parent = \"\"\n\nIf this is a live branch, the parent is the last commit from that\nbranch. Otherwise, we take the last commit whose resulting tree\nlooked like this one. Or...\n\n> + # Default to trunk if the branch is new\n> + if parent == \"\" and \"refs/heads/trunk\" in ref_commit:\n> + parent = ref_commit[\"refs/heads/trunk\"]\n\n... if all else fails, we take the tip commit on the trunk.\n\nFor comparison, here's the git-svn rule:\n\n> \t# look for a parent from another branch:\n> \tmy @b_path_components = split m#/#, $self->{path};\n\nAmong the paths above this commit's base directory [if this is\nbranches/foo, examine first branches/foo, then branches, then /]:\n\n> \twhile (@b_path_components) {\n> \t\t$i = $paths->{'/'.join('/', @b_path_components)};\n> \t\tlast if $i && defined $i->{copyfrom_path};\n> \t\tunshift(@a_path_components, pop(@b_path_components));\n> \t}\n> \treturn undef unless defined $i && defined $i->{copyfrom_path};\n\nFind the first one with copyfrom information (i.e., that was\nrenamed or copied from another rev in this revision).\n\n> \tmy $branch_from = $i->{copyfrom_path};\n> \tif (@a_path_components) {\n> \t\tprint STDERR \"branch_from: $branch_from => \";\n> \t\t$branch_from .= '/'.join('/', @a_path_components);\n> \t\tprint STDERR $branch_from, \"\\n\";\n> \t}\n\nBuild back up the URL (so if branches was renamed to Branches but\nbranches/foo had no copyfrom information, we look for Branches/foo).\n\n[...]\n> \tmy $gs = $self->other_gs($new_url, $url,\n> \t\t $branch_from, $r, $self->{ref_id});\n> \tmy ($r0, $parent) = $gs->find_rev_before($r, 1);\n\nFind the last revision that changed that path and record it.\n\nMaybe we could benefit from including the copyfrom information in the\nfast-import stream output by svn-fe somehow? The simplest way to do\nthis would be some specially formatted comments. An alternative (in\nthe spirit of Sam's earlier suggestions) might be to represent it in\nthe tree svn-fe creates, for example by introducing dummy\n\n\tfoo.copiedfrom\n\nsymlinks.\n\nThanks, that was interesting.\nJonathan\n"},{"id":"156317","messageId":"15144262.32618.1290434477881.JavaMail.root@mail.hq.genarts.com","threadId":"25374","inReplyTo":"20101121051734.GA11856@burratino","subject":"Re: [PATCH] contrib/svn-fe: Fast script to remap svn history","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2010-11-22T14:01:17Z","receivedAt":"2010-11-22T14:01:17Z","isPatch":true,"sender":{"key":"bash@genarts.com","avatar":null},"body":"----- Original Message -----\n> From: \"Jonathan Nieder\" \n> Sent: Sunday, November 21, 2010 12:17:34 AM\n> Subject: Re: [PATCH] contrib/svn-fe: Fast script to remap svn history\n> \n> Maybe we could benefit from including the copyfrom information in the\n> fast-import stream output by svn-fe somehow?\n\nThis has been discussed (and IMO it is essentially required to achieve high accuracy in the mapping):\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/158940/focus=159331\n\nThanks,\nStephen\n"},{"id":"156329","messageId":"20101122174242.GD2084@burratino","threadId":"25374","inReplyTo":"15144262.32618.1290434477881.JavaMail.root@mail.hq.genarts.com","subject":"Re: [PATCH] contrib/svn-fe: Fast script to remap svn history","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-22T17:42:42Z","receivedAt":"2010-11-22T17:42:42Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Stephen Bash wrote:\n\n> This has been discussed (and IMO it is essentially required to achieve high accuracy in the mapping):\n> \n> http://thread.gmane.org/gmane.comp.version-control.git/158940/focus=159331\n\nI think the suggestion of that thread was (tweaked a little)\nsomething like this:\n\n - List of directories with copyfrom information.\n Prune them so no listed directory is an ancestor of another.\n The result would usually be a single directory name.\n\n - Record that directory's (or those directories') copyfrom\n information in the log message.\n\nIn general, I don't like limiting the information accessible to branch\nmappers this way. Maybe a branch mapper would like to look at the\ncopyfrom information for files instead of directories. But this does\nhave the advantages of being simple and of not littering imported\ntrees with spurious files.\n\nIt also leaves open the question of how we would record unhandled node\nproperties (like svn:ignore and svn:eol) and empty directories, if at\nall.\n\nProbably in the end we will have to give up and provide multiple\noptions to choose between. :)\n\nJonathan\n"}]}