{"thread":{"id":"6774","subject":"darcs2git.py - convert darcs repository using gfi","startedAt":"2007-02-11T23:56:34Z","lastAt":"2007-02-19T08:25:26Z","messageCount":5,"participants":["Han-Wen Nienhuys","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"34220","messageId":"eqoaf7$loq$1@sea.gmane.org","threadId":"6774","inReplyTo":null,"subject":"darcs2git.py - convert darcs repository using gfi","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2007-02-11T23:56:34Z","receivedAt":"2007-02-11T23:56:34Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"The python script attached is a try at providing a sane\nconversion from Darcs to GIT.  It tries to map darcs conflict\nresolutions onto git branch merges.\n\nRegarding GFI, it's a breeze to work with; my compliments to its\nauthor. My only gripe is the need to specify a branch for each commit.\nDarcs uses changeset based storage. It doesn't really have branches,\nbut it does record divergent changes and merges of resulting\nconflicts.  Hence, it's not clear which refs/head/BRANCH should be\nused when creating a commit object.\n\nI found it easiest to write each commit to a\n\n  refs/head/darcs-tmp-COUNT\n\nbranch, use the reset command to specify at the end which commits are\ntops of branches, and delete the temporary branches.\n\nSo, my feature request: please make the \"commit\" command always accept\na \"from\" command, and make the \"refs\" argument optional.  This will\ncleanup my converter, and separate out two logical functions of the\ngfi \"commit\" command: creating a commit object, and advancing the head\nref.\n\n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n\n\nimport os  \nimport sys\nimport time\nimport xml.dom.minidom\nimport re\nimport gdbm as dbmodule\nimport gzip\nimport optparse\n\n################################################################\n# globals\n\nsilent=False\nmail_to_name_dict = {}\npending_patches = {}\nused_tags = {}\n\n################################################################\n# utils\n\nclass PullConflict (Exception):\n    pass\nclass CommandFailed (Exception):\n    pass\n\ndef progress (s):\n    sys.stderr.write (s + '\\n')\n    \ndef get_cli_options ():\n    p = optparse.OptionParser ()\n\n    p.usage='''darcs2git [OPTIONS] DARCS-REPO'''\n    p.description='''Convert darcs repo to git.\n\nThis tool is a one shot conversion utility for Darcs repositories.  It\nrequires Git version that has git-fast-import.  It does not support\nincremental updating.\n\nThis tool will import the patches in chronological order, and only creates\nmerges when a resolved conflict is detected.\n\nTODO:\n\n- correct time zone handling\n- \n\n'''\n\n    def update_map (option, opt, value, parser):\n        for l in open (value).readlines ():\n            (mail, name) = tuple (l.strip ().split ('='))\n            mail_to_name_dict[mail] = name\n\n    p.add_option ('-a', '--authors', action='callback',\n                  callback=update_map,\n                  type='string',\n                  nargs=1,\n                  help='read a text file, containing EMAIL=NAME lines')\n\n    p.add_option ('-d', '--destination', action='store',\n                  type='string',\n                  default='',\n                  dest='target_git_repo',\n                  help='where to put the resulting Git repo.')\n\n    p.add_option ('--verbose', action='store_true',\n                  dest='verbose',\n                  default=False, \n                  help='show commands as they are invoked')\n    \n    options, args = p.parse_args ()\n    if not args:\n        p.print_help ()\n        sys.exit (2)\n        \n    global silent\n    silent = not options.verbose\n\n    if not options.target_git_repo:\n        p = args[0]\n        p = os.path.abspath (p)\n        options.target_git_repo = os.path.basename (p).replace ('.darcs', '')\n        options.target_git_repo += '.git' \n       \n        \n    return (options, args)\n\ndef read_pipe (cmd, ignore_errors=False):\n    if not silent:\n        progress ('pipe %s' % cmd)\n    pipe = os.popen (cmd)\n\n    val = pipe.read ()\n    if pipe.close () and not ignore_errors:\n        raise CommandFailed (\"Pipe failed: %s\" % cmd)\n    \n    return val\n\ndef system (c, ignore_error=0):\n    if not silent:\n        progress ( c)\n    if os.system (c) and not ignore_error:\n        raise CommandFailed (\"Command failed: %s\" % c)\n\ndef darcs_date_to_git (x):\n    t = time.strptime (x, '%Y%m%d%H%M%S')\n    return '%d' % int (time.mktime (t))\n\ndef darcs_timezone (x) :\n    time.strptime (x, '%a %b %d %H:%M:%S %Z %Y')\n\n    # todo\n    return \"+0100\"\n\n################################################################\n# darcs\n\nclass DarcsConversionRepo:\n    def __init__ (self, dir, patches):\n        self.dir = dir\n        self.patches = patches\n        \n    def clean (self):\n        system ('rm -rf %s' % self.dir)\n        \n    def pull (self, patch):\n        id = patch.attributes['hash']\n        source_repo = patch.dir\n        dir = self.dir\n        system ('cd %(dir)s && darcs pull --quiet --all --match \"hash %(id)s\" %(source_repo)s ' % locals ())\n\n    def go_from_to (self, from_patch, to_patch):\n\n        \"\"\"Move the repo to FROM_PATCH, then go to TO_PATCH. Raise\n        PullConflict if conflict is detected\n\n        This uses the fishy technique of writing the inventory and\n        constructing the pristine tree with 'darcs repair'\n\n        It might be quicker and/or more correct to wind/rewind the\n        repo with pull and unpull.  \"\"\"\n\n        \n        dir = os.path.abspath (self.dir)\n        system ('rm -rf %(dir)s && mkdir %(dir)s && darcs init --repo  %(dir)s'\n                % locals ())\n    \n        source = to_patch.dir\n        if from_patch:\n            iv = open (dir + '/_darcs/inventory', 'w')\n            for p in self.patches[:from_patch.number+1]:\n                os.link (p.filename (), dir + '/_darcs/patches/' + os.path.basename (p.filename ()))\n                iv.write (p.header ())\n                \n            iv.close ()\n\n            progress ('Go to patch %d' % from_patch.number)\n            system ('cd %(dir)s && darcs repair --quiet' % locals ())\n            system ('rsync -a  %(dir)s/_darcs/pristine/ %(dir)s/' % locals ())\n        try:\n            self.pull (to_patch)\n            success = 'No conflicts to resolve' in read_pipe ('cd %(dir)s && echo y|darcs resolve' % locals  ())\n        except CommandFailed:\n            raise PullConflict ()\n    \n\n        if not success:\n            raise PullConflict ()\n\n    def has_patch (self, p):\n        id = p.attributes['hash']\n        f = self.dir + '/_darcs/patches/' + id\n        return os.path.exists (f)\n\n    def pristine_tree (self):\n        return self.dir + '/_darcs/pristine'\n    \nclass DarcsPatch:\n    def __init__ (self, xml, dir):\n        self.xml = xml\n        self.dir = dir\n        self.number = -1\n        self.attributes = {}\n        for (nm, value) in xml.attributes.items():\n            self.attributes[nm] = value\n\n        # fixme: ugh attributes vs. methods.\n        self.extract_author ()\n        self.extract_message ()\n        self.extract_time ()\n\n    def filename (self):\n        return self.dir + '/_darcs/patches/' + self.attributes['hash']\n\n    def contents (self):\n        f = gzip.open (self.filename ())\n        return f.read ()\n\n    def header (self):\n        lines = self.contents ().split ('\\n')\n\n        name = lines[0]\n        committer = lines[1] + '\\n'\n        committer = re.sub ('] {\\n$', ']\\n', committer)\n        committer = re.sub ('] *\\n$', ']\\n', committer)\n        comment = ''\n        if not committer.endswith (']\\n'):\n            for l in lines[2:]:\n                if l[0] == ']':\n                    comment += ']\\n'\n                    break\n                comment += l + '\\n'\n\n        header = name  + '\\n' + committer \n        if comment:\n            header += comment\n   \n        return header\n\n    def extract_author (self):\n        mail = self.attributes['author']\n        name = ''\n        m = re.search (\"^(.*) <(.*)>$\", mail)\n\n        if m:\n            name = m.group (1)\n            mail = m.group (2)\n        else:\n            try:\n                name = mail_to_name_dict[mail]\n            except KeyError:\n                name = mail.split ('@')[0]\n\n        self.author_name = name\n        self.author_mail = mail\n\n    def extract_time (self):\n        self.date = darcs_date_to_git (self.attributes['date']) + ' ' + darcs_timezone (self.attributes['local_date'])\n\n    def name (self):\n        patch_name = '(no comment)'\n        try:\n            name_elt = self.xml.getElementsByTagName ('name')[0]\n            patch_name = name_elt.childNodes[0].data\n        except IndexError:\n            pass\n        return patch_name\n    \n    def extract_message (self):\n        patch_name = self.name ()\n        comment_elts = self.xml.getElementsByTagName ('comment')\n        comment = ''\n        if comment_elts:\n            comment = comment_elts[0].childNodes[0].data\n\n        if self.attributes['inverted'] == 'True':\n            patch_name = 'UNDO: ' + patch_name\n\n        self.message = '%s\\n\\n%s' % (patch_name, comment)\n\n    def tag_name (self):\n        patch_name = self.name ()\n        if patch_name.startswith (\"TAG \"):\n           tag = patch_name[4:]\n           tag = re.sub (r'\\s', '_', tag).strip ()\n           tag = re.sub (r':', '_', tag).strip ()\n           return tag\n        return ''\n\ndef get_darcs_patches (darcs_repo):\n    progress ('reading patches.')\n    \n    xml_string = read_pipe ('darcs changes --xml --reverse --repo ' + darcs_repo)\n\n    dom = xml.dom.minidom.parseString(xml_string)\n    xmls = dom.documentElement.getElementsByTagName('patch')\n\n    patches = [DarcsPatch (x, darcs_repo) for x in xmls]\n\n    n = 0\n    for p in patches:\n        p.number = n\n        n += 1\n\n    return patches\n\n################################################################\n# GIT export\n\ndef export_tree (tree, gfi):\n    tree = os.path.normpath (tree)\n    gfi.write ('deleteall\\n')\n    for (root, dirs, files) in os.walk (tree):\n        for f in files:\n            rf = os.path.normpath (os.path.join (root, f))\n            s = open (rf).read ()\n            rf = rf.replace (tree + '/', '')\n            \n            gfi.write ('M 644 inline %s\\n' % rf)\n            gfi.write ('data %d\\n%s\\n' % (len (s), s))\n    gfi.write ('\\n')\n\n    \ndef export_commit (repo, patch, last_patch, gfi):\n    gfi.write ('commit refs/heads/darcstmp%d\\n' % patch.number)\n    gfi.write ('mark :%d\\n' % (patch.number + 1))\n    gfi.write ('committer %s <%s> %s\\n' % (patch.author_name,\n                                           patch.author_mail,\n                                           patch.date))\n    gfi.write ('data %d\\n%s\\n' % (len (patch.message), patch.message))\n    if last_patch:\n        gfi.write ('from :%d\\n' % (last_patch.number + 1))\n        if pending_patches.has_key (last_patch.number):\n            del pending_patches[last_patch.number]\n        \n    for (n, p) in pending_patches.items ():\n        if repo.has_patch (p):\n            gfi.write ('merge :%d\\n' % (n + 1))\n            del pending_patches[n]\n        \n    pending_patches[patch.number] = patch\n    export_tree (repo.pristine_tree (), gfi)\n\ndef export_pending (gfi):\n    if len (pending_patches.items ()) == 1:\n        gfi.write ('reset refs/heads/master\\n')\n        gfi.write ('from :%d\\n\\n' % (pending_patches.values()[0].number+1))\n        return\n    \n    \n    for (n, p) in pending_patches.items ():\n        gfi.write ('reset refs/heads/master%d\\n' % n)\n        gfi.write ('from :%d\\n\\n' % (n+1))\n\n    patches = pending_patches.values()\n    patch = patches[0]\n    gfi.write ('commit refs/heads/master\\n')\n    gfi.write ('committer %s <%s> %s\\n' % (patch.author_name,\n                                           patch.author_mail,\n                                           patch.date))\n    msg = 'tie together'\n    gfi.write ('data %d\\n%s\\n' % (len(msg), msg))\n    gfi.write ('from :%d\\n' % (patch.number + 1))\n    for p in patches[1:]:\n        gfi.write ('merge :%d\\n' % (p.number + 1))\n    gfi.write ('\\n')\n        \n    \ndef export_tag (patch, gfi):\n    gfi.write ('tag %s\\n' % patch.tag_name ())\n    gfi.write ('from :%d\\n' % (patch.number + 1))\n    gfi.write ('tagger %s <%s> %s\\n' % (patch.author_name,\n                                    patch.author_mail,\n                                    patch.date))\n    gfi.write ('data %d\\n%s\\n' % (len (patch.message),\n                                  patch.message))\n    \n################################################################\n# main.\ndef main ():\n\n    (options, args) = get_cli_options ()\n    \n    darcs_repo = os.path.abspath (args[0])\n    git_repo = os.path.abspath (options.target_git_repo)\n    \n    system ('mkdir %(git_repo)s && cd %(git_repo)s && git --bare init' % locals ())\n    os.environ['GIT_DIR'] = git_repo\n\n    gfi = os.popen ('git-fast-import', 'w')    #\n    \n    patches = get_darcs_patches (darcs_repo)\n\n    conv_repo = DarcsConversionRepo (\"darcs2git.tmpdarcs\", patches)\n    for p in patches:\n        parent = p.number - 1\n\n        last = None\n        while 1:\n            if parent >= 0:\n                last = patches[parent]\n\n            try:\n                conv_repo.go_from_to (last, p)\n                break\n            except PullConflict:\n                ## simplistic, may not be enough.\n                progress ('conflict, going one back')\n                parent -= 1\n\n                if parent < 0:\n                    raise Exception('urg')\n\n        progress ('Export %d -> %d (total %d)' % (parent,\n                                                 p.number, len (patches)))\n        export_commit (conv_repo, p, last, gfi)\n        if p.tag_name ():\n            export_tag (p, gfi)\n    export_pending (gfi)\n    gfi.close ()\n\n    system ('rm %(git_repo)s/refs/heads/darcstmp*' % locals ())\n    conv_repo.clean ()\n    \nmain ()\n"},{"id":"34228","messageId":"20070212011401.GK31488@spearce.org","threadId":"6774","inReplyTo":"eqoaf7$loq$1@sea.gmane.org","subject":"Re: darcs2git.py - convert darcs repository using gfi","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T01:14:01Z","receivedAt":"2007-02-12T01:14:01Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Han-Wen Nienhuys <hanwen@xs4all.nl> wrote:\n> The python script attached is a try at providing a sane\n> conversion from Darcs to GIT.  It tries to map darcs conflict\n> resolutions onto git branch merges.\n\nImpressive.\n \n> Regarding GFI, it's a breeze to work with; my compliments to its\n> author.\n\nHey, thanks!  ;-)\n\n> My only gripe is the need to specify a branch for each commit.\n> Darcs uses changeset based storage. It doesn't really have branches,\n> but it does record divergent changes and merges of resulting\n> conflicts.  Hence, it's not clear which refs/head/BRANCH should be\n> used when creating a commit object.\n\nJust make something up.  Or don't use refs/heads, instead use your\nown directory e.g. refs/patches/.  Then you can delete the entire\ndirectory when you are done importing.\n\n> So, my feature request: please make the \"commit\" command always accept\n> a \"from\" command\n\nThis restriction was a safety valve.  fast-import itself would be\nOK if I permitted a from all of the time.  A bug in cvs2svn caused\nmultiple froms to be emitted for the same branch, and that wasn't\ncorrect, so fast-import crashed on it rather than silently accepting\nthe data corruption.\n\nIts actually one of those things that is nice to remove, as its 3\nlines of code that just need to be deleted.  ;-)\n\n> and make the \"refs\" argument optional.\n\nThis is harder than it sounds.  fast-import internally is built\naround the assumption of a branch, which has a name, and which lives\nin the branch LRU.  With the \"from\" command restriction lifted\nyou can just import every single commit onto the same hardcoded\nbranch name (e.g. DARCS_HEAD) then delete it when you are done\n(e.g. rm .git/DARCS_HEAD).  That's basically the same thing as an\noptional ref argument.\n\n-- \nShawn.\n"},{"id":"34476","messageId":"45D23EC6.9000806@xs4all.nl","threadId":"6774","inReplyTo":"20070212011401.GK31488@spearce.org","subject":"Re: darcs2git.py - convert darcs repository using gfi","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2007-02-13T22:42:14Z","receivedAt":"2007-02-13T22:42:14Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Shawn O. Pearce escreveu:\n>> The python script attached is a try at providing a sane\n>> conversion from Darcs to GIT.  It tries to map darcs conflict\n>> resolutions onto git branch merges.\n> \n> Impressive.\n\nIf anyone's interested I've hacked at it some more, \nand the result is at \n\n  http://repo.or.cz/w/darcs2git.git\n\nI've further improved the script so it generates nicer \nbifurcations, and does the conversion more incrementally as \npossible, and can be considered fast (as far as any darcs project\ncan be called fast) \n\nI've successfully converted a project containing\n3000 patches with it.  \n\n> This is harder than it sounds.  fast-import internally is built\n> around the assumption of a branch, which has a name, and which lives\n> in the branch LRU.  With the \"from\" command restriction lifted\n> you can just import every single commit onto the same hardcoded\n> branch name (e.g. DARCS_HEAD) then delete it when you are done\n> (e.g. rm .git/DARCS_HEAD).  That's basically the same thing as an\n\nwill this be in git v1.5 ?\n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"34955","messageId":"45D84A6B.3060402@xs4all.nl","threadId":"6774","inReplyTo":"20070212011401.GK31488@spearce.org","subject":"git-fast-export ?","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2007-02-18T12:45:31Z","receivedAt":"2007-02-18T12:45:31Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Shawn O. Pearce escreveu:\n>> Regarding GFI, it's a breeze to work with; my compliments to its\n>> author.\n> \n> Hey, thanks!  ;-)\n\nBTW, I just had a brainwave\n\nWouldn't it make sense to build a git-fast-export, which \ncan be used to interrogate a git-repository: a GIT tool \ncould run a single git command, \n\n  git-fast-export\n\nand communicate with that process in a similar fashion, eg.\n\n  object-type SP 0ac9f9cb54e7cf400453b85a3ae5f63813f5fdae LF\n\n=>\n\n  show-raw SP 0ac9f9cb54e7cf400453b85a3ae5f63813f5fdae LF\n=>\n  data SP 235 LF \ntree 76c9d63f83530851d911f6ead36e3899929e0cda\nparent e3559e3c52cf006a6b3b03ec083ed658ba1941ee\nauthor Han-Wen Nienhuys <hanwen@lilypond.org> 1171799942 +0100\ncommitter Han-Wen Nienhuys <hanwen@lilypond.org> 1171799942 +0100\n\nignorance. LF\n\nthis would make interfacing with Git from scripts more ergonomical, \nless dependent on changes in the UI of porcelains, and in some cases\nmore efficient.\n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"34984","messageId":"20070219082525.GD30030@spearce.org","threadId":"6774","inReplyTo":"45D84A6B.3060402@xs4all.nl","subject":"Re: git-fast-export ?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-19T08:25:26Z","receivedAt":"2007-02-19T08:25:26Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Han-Wen Nienhuys <hanwen@xs4all.nl> wrote:\n> Wouldn't it make sense to build a git-fast-export, which \n> can be used to interrogate a git-repository: a GIT tool \n> could run a single git command, \n> \n> this would make interfacing with Git from scripts more ergonomical, \n> less dependent on changes in the UI of porcelains, and in some cases\n> more efficient.\n\nMaybe.\n\nBut without knowing what the UI program wants, its hard to say what\nshould be implemented there.  I'm not going to create something on\na hunch that it will be useful someday - that's just not a practical\nuse of my time.\n\nWorse, most scripting level languages have a hard time working\nwith a bidirectional pipe to a process.  What you want here is\nstdin and stdout pipes, so you can send a command and then receive\nthe response.  This can be a challenge in something like Tcl,\nmaybe not fully portable in Perl, etc.\n\nEven worse, some parts of Git are not reentrant.  They are currently\nbuilt to run once and have the UNIX process terminate quickly\nafterwards.  Keeping it running to answer more queries from the\nUI may cause the Git process to leak memory over a longer term,\ncause it to crash after a couple of successive repack/prune/gc, etc.\n\nThere are a number of interesting operations within Git that a UI\nwould want to query, but that may not be a good idea to expose from\na within a long running UNIX process, for those reasons.  fast-import\ndoesn't do these, so its reasonable to keep up for extended periods,\nbut even fast-import assumes it will terminate at some point as it\nhangs onto its object table for the entire lifespan of the process.\n\n-- \nShawn.\n"}]}