{"thread":{"id":"33392","subject":"[RFC/PATH 2/4] remote-hg: improve node traversing","startedAt":"2013-04-05T11:36:15Z","lastAt":"2013-04-05T22:29:26Z","messageCount":7,"participants":["Felipe Contreras","Jed Brown"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"213268","messageId":"1365161779-32170-1-git-send-email-felipe.contreras@gmail.com","threadId":"33392","inReplyTo":null,"subject":"[RFC/PATH 0/4] remoe-hg: switch to SHA-1 ids","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-04-05T11:36:15Z","receivedAt":"2013-04-05T11:36:15Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Hi,\n\nAs some people suggested this is necessary for some use-cases, because revision\nid numbers can change and point to different revisions.\n\nSeems to work fine, but I wouldn't merge it just yet.\n\nFelipe Contreras (4):\n remote-hg: shuffle some code\n remote-hg: improve node traversing\n remote-hg: add version checks to the marks\n remote-hg: switch from revisions to SHA-1 noteids\n\n contrib/remote-helpers/git-remote-hg | 77 ++++++++++++++++++++++--------------\n 1 file changed, 48 insertions(+), 29 deletions(-)\n\n-- \n1.8.2\n"},{"id":"213267","messageId":"1365161779-32170-2-git-send-email-felipe.contreras@gmail.com","threadId":"33392","inReplyTo":"1365161779-32170-1-git-send-email-felipe.contreras@gmail.com","subject":"[RFC/PATH 1/4] remote-hg: shuffle some code","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-04-05T11:36:16Z","receivedAt":"2013-04-05T11:36:16Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"In preparation to shift to SHA-1's.\n\nSigned-off-by: Felipe Contreras \n---\n contrib/remote-helpers/git-remote-hg | 27 +++++++++++++++------------\n 1 file changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg\nindex d82eb2d..a8591a2 100755\n--- a/contrib/remote-helpers/git-remote-hg\n+++ b/contrib/remote-helpers/git-remote-hg\n@@ -63,6 +63,9 @@ def hgmode(mode):\n def hghex(node):\n return hg.node.hex(node)\n \n+def hgbin(node):\n+ return hg.node.bin(node)\n+\n def get_config(config):\n cmd = ['git', 'config', '--get', config]\n process = subprocess.Popen(cmd, stdout=subprocess.PIPE)\n@@ -207,7 +210,7 @@ def get_filechanges(repo, ctx, parent):\n removed = set()\n \n cur = ctx.manifest()\n- prev = repo[parent].manifest().copy()\n+ prev = parent.manifest().copy()\n \n for fn in cur:\n if fn in prev:\n@@ -326,7 +329,7 @@ def export_ref(repo, name, kind, head):\n else:\n committer = author\n \n- parents = [p for p in repo.changelog.parentrevs(rev) if p >= 0]\n+ parents = [repo[p] for p in repo.changelog.parentrevs(rev) if p >= 0]\n \n if len(parents) == 0:\n modified = c.manifest().keys()\n@@ -372,9 +375,9 @@ def export_ref(repo, name, kind, head):\n print desc\n \n if len(parents) > 0:\n- print \"from :%s\" % (rev_to_mark(parents[0]))\n+ print \"from :%s\" % (rev_to_mark(parents[0].rev()))\n if len(parents) > 1:\n- print \"merge :%s\" % (rev_to_mark(parents[1]))\n+ print \"merge :%s\" % (rev_to_mark(parents[1].rev()))\n \n for f in modified:\n export_file(c.filectx(f))\n@@ -389,10 +392,10 @@ def export_ref(repo, name, kind, head):\n \n # make sure the ref is updated\n print \"reset %s/%s\" % (prefix, ename)\n- print \"from :%u\" % rev_to_mark(rev)\n+ print \"from :%u\" % rev_to_mark(head.rev())\n print\n \n- marks.set_tip(ename, rev)\n+ marks.set_tip(ename, head.rev())\n \n def export_tag(repo, tag):\n export_ref(repo, tag, 'tags', repo[tag])\n@@ -651,7 +654,7 @@ def parse_commit(parser):\n tmp = encoding.encoding\n encoding.encoding = 'utf-8'\n \n- node = repo.commitctx(ctx)\n+ node = hghex(repo.commitctx(ctx))\n \n encoding.encoding = tmp\n \n@@ -675,7 +678,7 @@ def parse_reset(parser):\n parser.next()\n \n node = parser.repo.changelog.node(mark_to_rev(from_mark))\n- parsed_refs[ref] = node\n+ parsed_refs[ref] = hghex(node)\n \n def parse_tag(parser):\n name = parser[1]\n@@ -720,10 +723,10 @@ def do_export(parser):\n elif ref.startswith('refs/tags/'):\n tag = ref[len('refs/tags/'):]\n if mode == 'git':\n- msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));\n- parser.repo.tag([tag], node, msg, False, None, {})\n+ msg = 'Added tag %s for changeset %s' % (tag, node[:6]);\n+ parser.repo.tag([tag], hgbin(node), msg, False, None, {})\n else:\n- parser.repo.tag([tag], node, None, True, None, {})\n+ parser.repo.tag([tag], hgbin(node), None, True, None, {})\n print \"ok %s\" % ref\n else:\n # transport-helper/fast-export bugs\n@@ -735,7 +738,7 @@ def do_export(parser):\n # handle bookmarks\n for bmark, node in p_bmarks:\n ref = 'refs/heads' + bmark\n- new = hghex(node)\n+ new = node\n \n if bmark in bmarks:\n old = bmarks[bmark].hex()\n-- \n1.8.2\n"},{"id":"213265","messageId":"1365161779-32170-3-git-send-email-felipe.contreras@gmail.com","threadId":"33392","inReplyTo":"1365161779-32170-1-git-send-email-felipe.contreras@gmail.com","subject":"[RFC/PATH 2/4] remote-hg: improve node traversing","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-04-05T11:36:17Z","receivedAt":"2013-04-05T11:36:17Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"We won't be able to count the unmarked commits, but we are not going to\nbe able to do that anyway when we switch to SHA-1 ids.\n\nSigned-off-by: Felipe Contreras \n---\n contrib/remote-helpers/git-remote-hg | 10 +++++++---\n 1 file changed, 7 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg\nindex a8591a2..02fda2d 100755\n--- a/contrib/remote-helpers/git-remote-hg\n+++ b/contrib/remote-helpers/git-remote-hg\n@@ -314,12 +314,16 @@ def export_ref(repo, name, kind, head):\n revs = xrange(tip, head.rev() + 1)\n count = 0\n \n- revs = [rev for rev in revs if not marks.is_marked(rev)]\n-\n for rev in revs:\n \n c = repo[rev]\n- (manifest, user, (time, tz), files, desc, extra) = repo.changelog.read(c.node())\n+ node = c.node()\n+\n+ if marks.is_marked(c.hex()):\n+ count += 1\n+ continue\n+\n+ (manifest, user, (time, tz), files, desc, extra) = repo.changelog.read(node)\n rev_branch = extra['branch']\n \n author = \"%s %d %s\" % (fixup_user(user), time, gittz(tz))\n-- \n1.8.2\n"},{"id":"213271","messageId":"1365161779-32170-4-git-send-email-felipe.contreras@gmail.com","threadId":"33392","inReplyTo":"1365161779-32170-1-git-send-email-felipe.contreras@gmail.com","subject":"[RFC/PATH 3/4] remote-hg: add version checks to the marks","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-04-05T11:36:18Z","receivedAt":"2013-04-05T11:36:18Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Signed-off-by: Felipe Contreras \n---\n contrib/remote-helpers/git-remote-hg | 16 +++++++++++++---\n 1 file changed, 13 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg\nindex 02fda2d..9e124e1 100755\n--- a/contrib/remote-helpers/git-remote-hg\n+++ b/contrib/remote-helpers/git-remote-hg\n@@ -43,6 +43,8 @@ AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$')\n AUTHOR_HG_RE = re.compile('^(.*?) ?<(.*?)(?:>(.+)?)?$')\n RAW_AUTHOR_RE = re.compile('^(\\w+) (?:(.+)? )?<(.*)> (\\d+) ([+-]\\d+)')\n \n+VERSION = 1\n+\n def die(msg, *args):\n sys.stderr.write('ERROR: %s\\n' % (msg % args))\n sys.exit(1)\n@@ -76,12 +78,19 @@ class Marks:\n \n def __init__(self, path):\n self.path = path\n+ self.clear()\n+ self.load()\n+\n+ if self.version < VERSION:\n+ self.clear()\n+ self.version = VERSION\n+\n+ def clear(self):\n self.tips = {}\n self.marks = {}\n self.rev_marks = {}\n self.last_mark = 0\n-\n- self.load()\n+ self.version = 0\n \n def load(self):\n if not os.path.exists(self.path):\n@@ -92,12 +101,13 @@ class Marks:\n self.tips = tmp['tips']\n self.marks = tmp['marks']\n self.last_mark = tmp['last-mark']\n+ self.version = tmp.get('version', 1)\n \n for rev, mark in self.marks.iteritems():\n self.rev_marks[mark] = int(rev)\n \n def dict(self):\n- return { 'tips': self.tips, 'marks': self.marks, 'last-mark' : self.last_mark }\n+ return { 'tips': self.tips, 'marks': self.marks, 'last-mark' : self.last_mark, 'version' : self.version }\n \n def store(self):\n json.dump(self.dict(), open(self.path, 'w'))\n-- \n1.8.2\n"},{"id":"213269","messageId":"1365161779-32170-5-git-send-email-felipe.contreras@gmail.com","threadId":"33392","inReplyTo":"1365161779-32170-1-git-send-email-felipe.contreras@gmail.com","subject":"[RFC/PATH 4/4] remote-hg: switch from revisions to SHA-1 noteids","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-04-05T11:36:19Z","receivedAt":"2013-04-05T11:36:19Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Otherwise we won't know if revisions are replaced.\n\nSigned-off-by: Felipe Contreras \n---\n contrib/remote-helpers/git-remote-hg | 40 +++++++++++++++++++-----------------\n 1 file changed, 21 insertions(+), 19 deletions(-)\n\ndiff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg\nindex 9e124e1..162dabc 100755\n--- a/contrib/remote-helpers/git-remote-hg\n+++ b/contrib/remote-helpers/git-remote-hg\n@@ -43,7 +43,7 @@ AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$')\n AUTHOR_HG_RE = re.compile('^(.*?) ?<(.*?)(?:>(.+)?)?$')\n RAW_AUTHOR_RE = re.compile('^(\\w+) (?:(.+)? )?<(.*)> (\\d+) ([+-]\\d+)')\n \n-VERSION = 1\n+VERSION = 2\n \n def die(msg, *args):\n sys.stderr.write('ERROR: %s\\n' % (msg % args))\n@@ -104,7 +104,7 @@ class Marks:\n self.version = tmp.get('version', 1)\n \n for rev, mark in self.marks.iteritems():\n- self.rev_marks[mark] = int(rev)\n+ self.rev_marks[mark] = rev\n \n def dict(self):\n return { 'tips': self.tips, 'marks': self.marks, 'last-mark' : self.last_mark, 'version' : self.version }\n@@ -116,23 +116,23 @@ class Marks:\n return str(self.dict())\n \n def from_rev(self, rev):\n- return self.marks[str(rev)]\n+ return self.marks[rev]\n \n def to_rev(self, mark):\n return self.rev_marks[mark]\n \n def get_mark(self, rev):\n self.last_mark += 1\n- self.marks[str(rev)] = self.last_mark\n+ self.marks[rev] = self.last_mark\n return self.last_mark\n \n def new_mark(self, rev, mark):\n- self.marks[str(rev)] = mark\n+ self.marks[rev] = mark\n self.rev_marks[mark] = rev\n self.last_mark = mark\n \n def is_marked(self, rev):\n- return self.marks.has_key(str(rev))\n+ return self.marks.has_key(rev)\n \n def get_tip(self, branch):\n return self.tips.get(branch, 0)\n@@ -305,7 +305,7 @@ def get_repo(url, alias):\n \n def rev_to_mark(rev):\n global marks\n- return marks.from_rev(rev)\n+ return marks.from_rev(rev.hex())\n \n def mark_to_rev(mark):\n global marks\n@@ -316,6 +316,10 @@ def export_ref(repo, name, kind, head):\n \n ename = '%s/%s' % (kind, name)\n tip = marks.get_tip(ename)\n+ if tip in repo:\n+ tip = repo[tip].rev()\n+ else:\n+ tip = 0\n \n # mercurial takes too much time checking this\n if tip and tip == head.rev():\n@@ -382,16 +386,16 @@ def export_ref(repo, name, kind, head):\n print 'reset %s/%s' % (prefix, ename)\n \n print \"commit %s/%s\" % (prefix, ename)\n- print \"mark :%d\" % (marks.get_mark(rev))\n+ print \"mark :%d\" % (marks.get_mark(c.hex()))\n print \"author %s\" % (author)\n print \"committer %s\" % (committer)\n print \"data %d\" % (len(desc))\n print desc\n \n if len(parents) > 0:\n- print \"from :%s\" % (rev_to_mark(parents[0].rev()))\n+ print \"from :%s\" % (rev_to_mark(parents[0]))\n if len(parents) > 1:\n- print \"merge :%s\" % (rev_to_mark(parents[1].rev()))\n+ print \"merge :%s\" % (rev_to_mark(parents[1]))\n \n for f in modified:\n export_file(c.filectx(f))\n@@ -406,10 +410,10 @@ def export_ref(repo, name, kind, head):\n \n # make sure the ref is updated\n print \"reset %s/%s\" % (prefix, ename)\n- print \"from :%u\" % rev_to_mark(head.rev())\n+ print \"from :%u\" % rev_to_mark(head)\n print\n \n- marks.set_tip(ename, head.rev())\n+ marks.set_tip(ename, head.hex())\n \n def export_tag(repo, tag):\n export_ref(repo, tag, 'tags', repo[tag])\n@@ -626,12 +630,12 @@ def parse_commit(parser):\n extra['committer'] = \"%s %u %u\" % committer\n \n if from_mark:\n- p1 = repo.changelog.node(mark_to_rev(from_mark))\n+ p1 = mark_to_rev(from_mark)\n else:\n p1 = '\\0' * 20\n \n if merge_mark:\n- p2 = repo.changelog.node(mark_to_rev(merge_mark))\n+ p2 = mark_to_rev(merge_mark)\n else:\n p2 = '\\0' * 20\n \n@@ -672,10 +676,8 @@ def parse_commit(parser):\n \n encoding.encoding = tmp\n \n- rev = repo[node].rev()\n-\n parsed_refs[ref] = node\n- marks.new_mark(rev, commit_mark)\n+ marks.new_mark(node, commit_mark)\n \n def parse_reset(parser):\n global parsed_refs\n@@ -691,8 +693,8 @@ def parse_reset(parser):\n from_mark = parser.get_mark()\n parser.next()\n \n- node = parser.repo.changelog.node(mark_to_rev(from_mark))\n- parsed_refs[ref] = hghex(node)\n+ rev = mark_to_rev(from_mark)\n+ parsed_refs[ref] = rev\n \n def parse_tag(parser):\n name = parser[1]\n-- \n1.8.2\n"},{"id":"213270","messageId":"8738v5qh5v.fsf@59A2.org","threadId":"33392","inReplyTo":"1365161779-32170-4-git-send-email-felipe.contreras@gmail.com","subject":"Re: [RFC/PATH 3/4] remote-hg: add version checks to the marks","fromName":"Jed Brown","fromEmail":"jed@59a2.org","sentAt":"2013-04-05T12:46:20Z","receivedAt":"2013-04-05T12:46:20Z","isPatch":false,"sender":{"key":"jed@59a2.org","avatar":"https://gravatar.com/avatar/1391d04d82555f9058a9fdf5eead233e909a48e40480db31fc554e7afeb301da?d=mp&s=160"},"body":"Felipe Contreras writes:\n\n> @@ -76,12 +78,19 @@ class Marks:\n> \n> def __init__(self, path):\n> self.path = path\n> + self.clear()\n> + self.load()\n> +\n> + if self.version < VERSION:\n> + self.clear()\n\nIt's friendlier to just upgrade the marks in-place. This takes less than\none second to run on repositories where full re-import would take half\nan hour:\n\n def upgrade_marks(self, hgrepo):\n if self.marks_version == 1: # Convert from integer reversions to hgsha1\n warn(\"Upgrading marks-hg from hg sequence number to SHA1\")\n self.marks_to_revisions = dict(\n (mark, hghex(hgrepo.changelog.node(int(rev))))\n for mark, rev in self.marks_to_revisions.iteritems())\n self.revisions_to_marks = dict(\n (hghex(hgrepo.changelog.node(int(rev))), mark)\n for rev, mark in self.revisions_to_marks.iteritems())\n self.marks_version = 2\n warn(\"Upgrade complete\")\n\nhttps://github.com/buchuki/gitifyhg/commit/23a6709efd14f3e058e3a846624b7677d1e8b497#L0R195\n"},{"id":"213338","messageId":"CAMP44s3o2ewvJiyvOBqEM5ztByEzdRCLHXpVYHe3=okjfr6L8A@mail.gmail.com","threadId":"33392","inReplyTo":"8738v5qh5v.fsf@59A2.org","subject":"Re: [RFC/PATH 3/4] remote-hg: add version checks to the marks","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-04-05T22:29:26Z","receivedAt":"2013-04-05T22:29:26Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Fri, Apr 5, 2013 at 6:46 AM, Jed Brown wrote:\n> Felipe Contreras writes:\n>\n>> @@ -76,12 +78,19 @@ class Marks:\n>>\n>> def __init__(self, path):\n>> self.path = path\n>> + self.clear()\n>> + self.load()\n>> +\n>> + if self.version < VERSION:\n>> + self.clear()\n>\n> It's friendlier to just upgrade the marks in-place. This takes less than\n> one second to run on repositories where full re-import would take half\n> an hour:\n\nYeah, but that's riskier, and only works for this particular version.\nBesides, it only happens one time per repository at best. Also, it\nwould mess up the current organization of the code. I'm not sure it's\nworth worrying about it at this point, and we could always add a patch\non top.\n\n-- \nFelipe Contreras\n"}]}