# [PATCH 00/16] remote-hg: second round of improvements 24 messages from 2013-04-22 to 2013-04-22. Participants: Felipe Contreras, Junio C Hamano, Dusty Phillips. Thread: https://gitlist.dev/t/33575 ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 00/16] remote-hg: second round of improvements Message-ID: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-1-git-send-email-felipe.contreras%40gmail.com ``` Hi, Now that the safter first round is merged to master, it's time for the second round which is a little tricker. Most of the patches should be safe, but the later ones might be not. All the tests pass, even gitifyhg ones, so everything seems to be fine, but we shall see. Among the important changes are: * Proper support for tags (they are global, and we can push them) * Improved e-mail sanitaztion (a bit similar to gitifyhg, but better) * Support for hg schemes (like bb://felipec/repo) * Support for tags, bookmarks and branches with spaces * Performance improvements Good stuff we should probably merge to master if there are no issues. Cheers. Dusty Phillips (1): remote-helpers: avoid has_key Felipe Contreras (15): remote-hg: safer bookmark pushing remote-hg: use python urlparse remote-hg: properly mark branches up-to-date remote-hg: add branch_tip() helper remote-hg: add support for tag objects remote-hg: custom method to write tags remote-hg: write tags in the appropriate branch remote-hg: add custom local tag write code remote-hg: improve email sanitation remote-hg: add support for schemes extension remote-hg: don't update bookmarks unnecessarily remote-hg: allow refs with spaces remote-hg: small performance improvement remote-hg: use marks instead of inlined files remote-hg: strip extra newline contrib/remote-helpers/git-remote-bzr | 2 +- contrib/remote-helpers/git-remote-hg | 167 +++++++++++++++++++++++++++------- contrib/remote-helpers/test-hg.sh | 8 +- 3 files changed, 141 insertions(+), 36 deletions(-) -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 01/16] remote-helpers: avoid has_key Message-ID: <1366667724-567-2-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-2-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` From: Dusty Phillips It is deprecated. [fc: do the same in remote-bzr] Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 2 +- contrib/remote-helpers/git-remote-hg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index aa7bc97..cc6609b 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -94,7 +94,7 @@ class Marks: return self.last_mark def is_marked(self, rev): - return self.marks.has_key(rev) + return str(rev) in self.marks def new_mark(self, rev, mark): self.marks[rev] = mark diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index 5481331..2cd1996 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -129,7 +129,7 @@ class Marks: self.last_mark = mark def is_marked(self, rev): - return self.marks.has_key(str(rev)) + return str(rev) in self.marks def get_tip(self, branch): return self.tips.get(branch, 0) -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 02/16] remote-hg: safer bookmark pushing Message-ID: <1366667724-567-3-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-3-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` It is possible that the remote has changed the bookmarks, so let's fetch them before we make any assumptions, just the way mercurial does. Probably doesn't make a difference, but better be safe than sorry. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index 2cd1996..dcf6c98 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -782,6 +782,8 @@ def do_export(parser): continue if peer: + rb = peer.listkeys('bookmarks') + old = rb.get(bmark, '') if not peer.pushkey('bookmarks', bmark, old, new): print "error %s" % ref continue -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 03/16] remote-hg: use python urlparse Message-ID: <1366667724-567-4-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-4-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` It's simpler, and we don't need to depend on certain Mercurial versions. Also, now we don't update the URL if 'file://' is not present. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index dcf6c98..b6589a3 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -22,6 +22,7 @@ import shutil import subprocess import urllib import atexit +import urlparse # # If you want to switch to hg-git compatibility mode: @@ -793,11 +794,11 @@ def do_export(parser): print def fix_path(alias, repo, orig_url): - repo_url = util.url(repo.url()) - url = util.url(orig_url) - if str(url) == str(repo_url): + url = urlparse.urlparse(orig_url, 'file') + if url.scheme != 'file' or os.path.isabs(url.path): return - cmd = ['git', 'config', 'remote.%s.url' % alias, "hg::%s" % repo_url] + abs_url = urlparse.urljoin("%s/" % os.getcwd(), orig_url) + cmd = ['git', 'config', 'remote.%s.url' % alias, "hg::%s" % abs_url] subprocess.call(cmd) def main(args): -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 04/16] remote-hg: properly mark branches up-to-date Message-ID: <1366667724-567-5-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-5-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index b6589a3..a4db5b0 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -742,6 +742,10 @@ def do_export(parser): for ref, node in parsed_refs.iteritems(): if ref.startswith('refs/heads/branches'): + branch = ref[len('refs/heads/branches/'):] + if branch in branches and node in branches[branch]: + # up to date + continue print "ok %s" % ref elif ref.startswith('refs/heads/'): bmark = ref[len('refs/heads/'):] -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 05/16] remote-hg: add branch_tip() helper Message-ID: <1366667724-567-6-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-6-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` Idea from gitifyhg, the backwards compatibility is how Mercurial used to do it. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index a4db5b0..bd93f82 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -457,6 +457,13 @@ def do_capabilities(parser): print +def branch_tip(repo, branch): + # older versions of mercurial don't have this + if hasattr(repo, 'branchtip'): + return repo.branchtip(branch) + else: + return repo.branchtags()[branch] + def get_branch_tip(repo, branch): global branches @@ -467,9 +474,7 @@ def get_branch_tip(repo, branch): # verify there's only one head if (len(heads) > 1): warn("Branch '%s' has more than one head, consider merging" % branch) - # older versions of mercurial don't have this - if hasattr(repo, "branchtip"): - return repo.branchtip(branch) + return branch_tip(repo, branch) return heads[0] -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 06/16] remote-hg: add support for tag objects Message-ID: <1366667724-567-7-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-7-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index bd93f82..4a1c637 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -722,7 +722,7 @@ def parse_tag(parser): data = parser.get_data() parser.next() - # nothing to do + parsed_tags[name] = (tagger, data) def do_export(parser): global parsed_refs, bmarks, peer @@ -758,9 +758,11 @@ def do_export(parser): continue elif ref.startswith('refs/tags/'): tag = ref[len('refs/tags/'):] + author, msg = parsed_tags.get(tag, (None, None)) if mode == 'git': - msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6])); - parser.repo.tag([tag], node, msg, False, None, {}) + if not msg: + msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6])); + parser.repo.tag([tag], node, msg, False, author[0], {}) else: parser.repo.tag([tag], node, None, True, None, {}) print "ok %s" % ref @@ -815,6 +817,7 @@ def main(args): global marks, blob_marks, parsed_refs global peer, mode, bad_mail, bad_name global track_branches, force_push, is_tmp + global parsed_tags alias = args[1] url = args[2] @@ -857,6 +860,7 @@ def main(args): blob_marks = {} parsed_refs = {} marks = None + parsed_tags = {} repo = get_repo(url, alias) prefix = 'refs/hg/%s' % alias -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 07/16] remote-hg: custom method to write tags Message-ID: <1366667724-567-8-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-8-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` The one from mercurial is meant for users, on top of the latest tip. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index 4a1c637..f5e4ba7 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -724,6 +724,37 @@ def parse_tag(parser): parsed_tags[name] = (tagger, data) +def write_tag(repo, tag, node, msg, author): + tip = repo['tip'] + + def getfilectx(repo, memctx, f): + try: + fctx = tip.filectx(f) + data = fctx.data() + except error.ManifestLookupError: + data = "" + content = data + "%s %s\n" % (hghex(node), tag) + return context.memfilectx(f, content, False, False, None) + + p1 = tip.hex() + p2 = '\0' * 20 + if not author: + author = (None, 0, 0) + user, date, tz = author + + ctx = context.memctx(repo, (p1, p2), msg, + ['.hgtags'], getfilectx, + user, (date, tz), {}) + + tmp = encoding.encoding + encoding.encoding = 'utf-8' + + tagnode = repo.commitctx(ctx) + + encoding.encoding = tmp + + return tagnode + def do_export(parser): global parsed_refs, bmarks, peer @@ -762,7 +793,7 @@ def do_export(parser): if mode == 'git': if not msg: msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6])); - parser.repo.tag([tag], node, msg, False, author[0], {}) + write_tag(parser.repo, tag, node, msg, author) else: parser.repo.tag([tag], node, None, True, None, {}) print "ok %s" % ref -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 08/16] remote-hg: write tags in the appropriate branch Message-ID: <1366667724-567-9-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-9-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index f5e4ba7..f685990 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -725,7 +725,9 @@ def parse_tag(parser): parsed_tags[name] = (tagger, data) def write_tag(repo, tag, node, msg, author): - tip = repo['tip'] + branch = repo[node].branch() + tip = branch_tip(repo, branch) + tip = repo[tip] def getfilectx(repo, memctx, f): try: @@ -744,7 +746,7 @@ def write_tag(repo, tag, node, msg, author): ctx = context.memctx(repo, (p1, p2), msg, ['.hgtags'], getfilectx, - user, (date, tz), {}) + user, (date, tz), {'branch' : branch}) tmp = encoding.encoding encoding.encoding = 'utf-8' -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 09/16] remote-hg: add custom local tag write code Message-ID: <1366667724-567-10-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-10-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` There's no point in calling the tag method for such simple action. Not that we care much about the hg-git compat mode, it's mostly just for comparison testing purposes. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index f685990..3c6eeb7 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -797,7 +797,9 @@ def do_export(parser): msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6])); write_tag(parser.repo, tag, node, msg, author) else: - parser.repo.tag([tag], node, None, True, None, {}) + fp = parser.repo.opener('localtags', 'a') + fp.write('%s %s\n' % (hghex(node), tag)) + fp.close() print "ok %s" % ref else: # transport-helper/fast-export bugs -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 10/16] remote-hg: improve email sanitation Message-ID: <1366667724-567-11-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-11-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 10 ++++++++-- contrib/remote-helpers/test-hg.sh | 8 ++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index 3c6eeb7..0084709 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -51,6 +51,7 @@ import urlparse NAME_RE = re.compile('^([^<>]+)') AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$') +EMAIL_RE = re.compile('^([^<>]+[^ \\\t<>])?\\b(?:[ \\t<>]*?)\\b([^ \\t<>]+@[^ \\t<>]+)') AUTHOR_HG_RE = re.compile('^(.*?) ?<(.*?)(?:>(.+)?)?$') RAW_AUTHOR_RE = re.compile('^(\w+) (?:(.+)? )?<(.*)> (\d+) ([+-]\d+)') @@ -245,9 +246,14 @@ def fixup_user_git(user): name = m.group(1) mail = m.group(2).strip() else: - m = NAME_RE.match(user) + m = EMAIL_RE.match(user) if m: - name = m.group(1).strip() + name = m.group(1) + mail = m.group(2) + else: + m = NAME_RE.match(user) + if m: + name = m.group(1).strip() return (name, mail) def fixup_user_hg(user): diff --git a/contrib/remote-helpers/test-hg.sh b/contrib/remote-helpers/test-hg.sh index d5b8730..8de2aa7 100755 --- a/contrib/remote-helpers/test-hg.sh +++ b/contrib/remote-helpers/test-hg.sh @@ -137,15 +137,15 @@ test_expect_success 'authors' ' author_test alpha "" "H G Wells " && author_test beta "test" "test " && - author_test beta "test (comment)" "test " && + author_test beta "test (comment)" "test " && author_test gamma "" "Unknown " && author_test delta "name" "name " && - author_test epsilon "name " && + author_test epsilon "name " && author_test zeta " test " "test " && author_test eta "test < test@example.com >" "test " && - author_test theta "test >test@example.com>" "test " && + author_test theta "test >test@example.com>" "test " && author_test iota "test < test example com>" "test " && - author_test kappa "test@example.com" "test@example.com " + author_test kappa "test@example.com" "Unknown " ) && git clone "hg::$PWD/hgrepo" gitrepo && -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 11/16] remote-hg: add support for schemes extension Message-ID: <1366667724-567-12-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-12-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` So that we can use shortened URLs, for example 'bb:://felipec/repo' (Bitbucket). Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index 0084709..4f6c7b7 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -12,7 +12,7 @@ # For remote repositories a local clone is stored in # "$GIT_DIR/hg/origin/clone/.hg/". -from mercurial import hg, ui, bookmarks, context, util, encoding, node, error +from mercurial import hg, ui, bookmarks, context, util, encoding, node, error, extensions import re import sys @@ -305,6 +305,12 @@ def get_repo(url, alias): except subprocess.CalledProcessError: pass + try: + mod = extensions.load(myui, 'hgext.schemes', None) + mod.extsetup(myui) + except ImportError: + pass + if hg.islocal(url): repo = hg.repository(myui, url) else: -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 12/16] remote-hg: don't update bookmarks unnecessarily Message-ID: <1366667724-567-13-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-13-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 3 +++ 1 file changed, 3 insertions(+) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index 4f6c7b7..dbb4091 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -830,6 +830,9 @@ def do_export(parser): else: old = '' + if old == new: + continue + if bmark == 'master' and 'master' not in parser.repo._bookmarks: # fake bookmark pass -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 13/16] remote-hg: allow refs with spaces Message-ID: <1366667724-567-14-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-14-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` Mercurial supports them, Git doesn't. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index dbb4091..6f4afd7 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -75,6 +75,12 @@ def hgmode(mode): def hghex(node): return hg.node.hex(node) +def hgref(ref): + return ref.replace('___', ' ') + +def gitref(ref): + return ref.replace(' ', '___') + def get_config(config): cmd = ['git', 'config', '--get', config] process = subprocess.Popen(cmd, stdout=subprocess.PIPE) @@ -437,10 +443,10 @@ def export_ref(repo, name, kind, head): marks.set_tip(ename, rev) def export_tag(repo, tag): - export_ref(repo, tag, 'tags', repo[tag]) + export_ref(repo, tag, 'tags', repo[hgref(tag)]) def export_bookmark(repo, bmark): - head = bmarks[bmark] + head = bmarks[hgref(bmark)] export_ref(repo, bmark, 'bookmarks', head) def export_branch(repo, branch): @@ -479,14 +485,14 @@ def branch_tip(repo, branch): def get_branch_tip(repo, branch): global branches - heads = branches.get(branch, None) + heads = branches.get(hgref(branch), None) if not heads: return None # verify there's only one head if (len(heads) > 1): warn("Branch '%s' has more than one head, consider merging" % branch) - return branch_tip(repo, branch) + return branch_tip(repo, hgref(branch)) return heads[0] @@ -508,6 +514,7 @@ def list_head(repo, cur): head = 'master' bmarks[head] = node + head = gitref(head) print "@refs/heads/%s HEAD" % head g_head = (head, node) @@ -529,15 +536,15 @@ def do_list(parser): branches[branch] = heads for branch in branches: - print "? refs/heads/branches/%s" % branch + print "? refs/heads/branches/%s" % gitref(branch) for bmark in bmarks: - print "? refs/heads/%s" % bmark + print "? refs/heads/%s" % gitref(bmark) for tag, node in repo.tagslist(): if tag == 'tip': continue - print "? refs/tags/%s" % tag + print "? refs/tags/%s" % gitref(tag) print @@ -674,7 +681,8 @@ def parse_commit(parser): # Check if the ref is supposed to be a named branch if ref.startswith('refs/heads/branches/'): - extra['branch'] = ref[len('refs/heads/branches/'):] + branch = ref[len('refs/heads/branches/'):] + extra['branch'] = hgref(branch) if mode == 'hg': i = data.find('\n--HG--\n') @@ -803,6 +811,7 @@ def do_export(parser): continue elif ref.startswith('refs/tags/'): tag = ref[len('refs/tags/'):] + tag = hgref(tag) author, msg = parsed_tags.get(tag, (None, None)) if mode == 'git': if not msg: -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 14/16] remote-hg: small performance improvement Message-ID: <1366667724-567-15-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-15-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` Load previous manifest first as Mercurial does; for caching reasons. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index 6f4afd7..f80236b 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -230,8 +230,9 @@ def get_filechanges(repo, ctx, parent): added = set() removed = set() - cur = ctx.manifest() + # load earliest manifest first for caching reasons prev = repo[parent].manifest().copy() + cur = ctx.manifest() for fn in cur: if fn in prev: -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 15/16] remote-hg: use marks instead of inlined files Message-ID: <1366667724-567-16-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-16-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` So that we can find already exported ones. We can never be 100% sure that we already exported such data, due to mercurial design, it at least sometimes we should detect them, and so should give ups some performance boost. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 41 +++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index f80236b..d0e552c 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -126,6 +126,10 @@ class Marks: def to_rev(self, mark): return self.rev_marks[mark] + def next_mark(self): + self.last_mark += 1 + return self.last_mark + def get_mark(self, rev): self.last_mark += 1 self.marks[str(rev)] = self.last_mark @@ -218,12 +222,29 @@ def fix_file_path(path): return path return os.path.relpath(path, '/') -def export_file(fc): - d = fc.data() - path = fix_file_path(fc.path()) - print "M %s inline %s" % (gitmode(fc.flags()), path) - print "data %d" % len(d) - print d +def export_files(files): + global marks, filenodes + + final = [] + for f in files: + fid = node.hex(f.filenode()) + + if fid in filenodes: + mark = filenodes[fid] + else: + mark = marks.next_mark() + filenodes[fid] = mark + d = f.data() + + print "blob" + print "mark :%u" % mark + print "data %d" % len(d) + print d + + path = fix_file_path(f.path()) + final.append((gitmode(f.flags()), mark, path)) + + return final def get_filechanges(repo, ctx, parent): modified = set() @@ -413,6 +434,8 @@ def export_ref(repo, name, kind, head): if len(parents) == 0 and rev: print 'reset %s/%s' % (prefix, ename) + modified_final = export_files(c.filectx(f) for f in modified) + print "commit %s/%s" % (prefix, ename) print "mark :%d" % (marks.get_mark(rev)) print "author %s" % (author) @@ -425,8 +448,8 @@ def export_ref(repo, name, kind, head): if len(parents) > 1: print "merge :%s" % (rev_to_mark(parents[1])) - for f in modified: - export_file(c.filectx(f)) + for f in modified_final: + print "M %s :%u %s" % f for f in removed: print "D %s" % (fix_file_path(f)) print @@ -878,6 +901,7 @@ def main(args): global peer, mode, bad_mail, bad_name global track_branches, force_push, is_tmp global parsed_tags + global filenodes alias = args[1] url = args[2] @@ -921,6 +945,7 @@ def main(args): parsed_refs = {} marks = None parsed_tags = {} + filenodes = {} repo = get_repo(url, alias) prefix = 'refs/hg/%s' % alias -- 1.8.2.1.790.g4588561 ``` ## Felipe Contreras, 2013-04-22 21:55 Subject: [PATCH 16/16] remote-hg: strip extra newline Message-ID: <1366667724-567-17-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366667724-567-17-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366667724-567-1-git-send-email-felipe.contreras@gmail.com> ``` There's no functional change since mercurial commit operation strips that anyway, but that's no excuse for us not to do the right thing. So let's be explicit about it. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-hg | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg index d0e552c..fda4199 100755 --- a/contrib/remote-helpers/git-remote-hg +++ b/contrib/remote-helpers/git-remote-hg @@ -652,6 +652,10 @@ def parse_commit(parser): if parser.check('merge'): die('octopus merges are not supported yet') + # fast-export adds an extra newline + if data[-1] == '\n': + data = data[:-1] + files = {} for line in parser: -- 1.8.2.1.790.g4588561 ``` ## Junio C Hamano, 2013-04-22 22:28 Subject: Re: [PATCH 01/16] remote-helpers: avoid has_key Message-ID: <7va9oqnqs9.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7va9oqnqs9.fsf%40alter.siamese.dyndns.org In-Reply-To: <1366667724-567-2-git-send-email-felipe.contreras@gmail.com> ``` Felipe Contreras writes: > From: Dusty Phillips > > It is deprecated. > > [fc: do the same in remote-bzr] > > Signed-off-by: Felipe Contreras > --- No sign-off by the author? It does not matter for something so trivial like this that there is no other way to write, but it is a good habit you should encourage your contributors to acquire, so that you do not have to waste time with "please sign off" when their next contribution that is more substantial comes. > contrib/remote-helpers/git-remote-bzr | 2 +- > contrib/remote-helpers/git-remote-hg | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr > index aa7bc97..cc6609b 100755 > --- a/contrib/remote-helpers/git-remote-bzr > +++ b/contrib/remote-helpers/git-remote-bzr > @@ -94,7 +94,7 @@ class Marks: > return self.last_mark > > def is_marked(self, rev): > - return self.marks.has_key(rev) > + return str(rev) in self.marks > > def new_mark(self, rev, mark): > self.marks[rev] = mark > diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg > index 5481331..2cd1996 100755 > --- a/contrib/remote-helpers/git-remote-hg > +++ b/contrib/remote-helpers/git-remote-hg > @@ -129,7 +129,7 @@ class Marks: > self.last_mark = mark > > def is_marked(self, rev): > - return self.marks.has_key(str(rev)) > + return str(rev) in self.marks > > def get_tip(self, branch): > return self.tips.get(branch, 0) ``` ## Junio C Hamano, 2013-04-22 22:32 Subject: Re: [PATCH 13/16] remote-hg: allow refs with spaces Message-ID: <7v61zenqm4.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v61zenqm4.fsf%40alter.siamese.dyndns.org In-Reply-To: <1366667724-567-14-git-send-email-felipe.contreras@gmail.com> ``` Felipe Contreras writes: > Mercurial supports them, Git doesn't. Another important thing to note is that you represent a SP with three underscores on our side, no? > Signed-off-by: Felipe Contreras > --- > contrib/remote-helpers/git-remote-hg | 25 +++++++++++++++++-------- > 1 file changed, 17 insertions(+), 8 deletions(-) > > diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg > index dbb4091..6f4afd7 100755 > --- a/contrib/remote-helpers/git-remote-hg > +++ b/contrib/remote-helpers/git-remote-hg > @@ -75,6 +75,12 @@ def hgmode(mode): > def hghex(node): > return hg.node.hex(node) > > +def hgref(ref): > + return ref.replace('___', ' ') > + > +def gitref(ref): > + return ref.replace(' ', '___') > + > def get_config(config): > cmd = ['git', 'config', '--get', config] > process = subprocess.Popen(cmd, stdout=subprocess.PIPE) > @@ -437,10 +443,10 @@ def export_ref(repo, name, kind, head): > marks.set_tip(ename, rev) > > def export_tag(repo, tag): > - export_ref(repo, tag, 'tags', repo[tag]) > + export_ref(repo, tag, 'tags', repo[hgref(tag)]) > > def export_bookmark(repo, bmark): > - head = bmarks[bmark] > + head = bmarks[hgref(bmark)] > export_ref(repo, bmark, 'bookmarks', head) > > def export_branch(repo, branch): > @@ -479,14 +485,14 @@ def branch_tip(repo, branch): > def get_branch_tip(repo, branch): > global branches > > - heads = branches.get(branch, None) > + heads = branches.get(hgref(branch), None) > if not heads: > return None > > # verify there's only one head > if (len(heads) > 1): > warn("Branch '%s' has more than one head, consider merging" % branch) > - return branch_tip(repo, branch) > + return branch_tip(repo, hgref(branch)) > > return heads[0] > > @@ -508,6 +514,7 @@ def list_head(repo, cur): > head = 'master' > bmarks[head] = node > > + head = gitref(head) > print "@refs/heads/%s HEAD" % head > g_head = (head, node) > > @@ -529,15 +536,15 @@ def do_list(parser): > branches[branch] = heads > > for branch in branches: > - print "? refs/heads/branches/%s" % branch > + print "? refs/heads/branches/%s" % gitref(branch) > > for bmark in bmarks: > - print "? refs/heads/%s" % bmark > + print "? refs/heads/%s" % gitref(bmark) > > for tag, node in repo.tagslist(): > if tag == 'tip': > continue > - print "? refs/tags/%s" % tag > + print "? refs/tags/%s" % gitref(tag) > > print > > @@ -674,7 +681,8 @@ def parse_commit(parser): > > # Check if the ref is supposed to be a named branch > if ref.startswith('refs/heads/branches/'): > - extra['branch'] = ref[len('refs/heads/branches/'):] > + branch = ref[len('refs/heads/branches/'):] > + extra['branch'] = hgref(branch) > > if mode == 'hg': > i = data.find('\n--HG--\n') > @@ -803,6 +811,7 @@ def do_export(parser): > continue > elif ref.startswith('refs/tags/'): > tag = ref[len('refs/tags/'):] > + tag = hgref(tag) > author, msg = parsed_tags.get(tag, (None, None)) > if mode == 'git': > if not msg: ``` ## Junio C Hamano, 2013-04-22 22:33 Subject: Re: [PATCH 15/16] remote-hg: use marks instead of inlined files Message-ID: <7v1ua2nqky.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v1ua2nqky.fsf%40alter.siamese.dyndns.org In-Reply-To: <1366667724-567-16-git-send-email-felipe.contreras@gmail.com> ``` Felipe Contreras writes: > So that we can find already exported ones. We can never be 100% sure > that we already exported such data, due to mercurial design, it at least > sometimes we should detect them, and so should give ups some performance s/ups/us/ (will locally tweak). > boost. > > Signed-off-by: Felipe Contreras > --- > contrib/remote-helpers/git-remote-hg | 41 +++++++++++++++++++++++++++++------- > 1 file changed, 33 insertions(+), 8 deletions(-) > > diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg > index f80236b..d0e552c 100755 > --- a/contrib/remote-helpers/git-remote-hg > +++ b/contrib/remote-helpers/git-remote-hg > @@ -126,6 +126,10 @@ class Marks: > def to_rev(self, mark): > return self.rev_marks[mark] > > + def next_mark(self): > + self.last_mark += 1 > + return self.last_mark > + > def get_mark(self, rev): > self.last_mark += 1 > self.marks[str(rev)] = self.last_mark > @@ -218,12 +222,29 @@ def fix_file_path(path): > return path > return os.path.relpath(path, '/') > > -def export_file(fc): > - d = fc.data() > - path = fix_file_path(fc.path()) > - print "M %s inline %s" % (gitmode(fc.flags()), path) > - print "data %d" % len(d) > - print d > +def export_files(files): > + global marks, filenodes > + > + final = [] > + for f in files: > + fid = node.hex(f.filenode()) > + > + if fid in filenodes: > + mark = filenodes[fid] > + else: > + mark = marks.next_mark() > + filenodes[fid] = mark > + d = f.data() > + > + print "blob" > + print "mark :%u" % mark > + print "data %d" % len(d) > + print d > + > + path = fix_file_path(f.path()) > + final.append((gitmode(f.flags()), mark, path)) > + > + return final > > def get_filechanges(repo, ctx, parent): > modified = set() > @@ -413,6 +434,8 @@ def export_ref(repo, name, kind, head): > if len(parents) == 0 and rev: > print 'reset %s/%s' % (prefix, ename) > > + modified_final = export_files(c.filectx(f) for f in modified) > + > print "commit %s/%s" % (prefix, ename) > print "mark :%d" % (marks.get_mark(rev)) > print "author %s" % (author) > @@ -425,8 +448,8 @@ def export_ref(repo, name, kind, head): > if len(parents) > 1: > print "merge :%s" % (rev_to_mark(parents[1])) > > - for f in modified: > - export_file(c.filectx(f)) > + for f in modified_final: > + print "M %s :%u %s" % f > for f in removed: > print "D %s" % (fix_file_path(f)) > print > @@ -878,6 +901,7 @@ def main(args): > global peer, mode, bad_mail, bad_name > global track_branches, force_push, is_tmp > global parsed_tags > + global filenodes > > alias = args[1] > url = args[2] > @@ -921,6 +945,7 @@ def main(args): > parsed_refs = {} > marks = None > parsed_tags = {} > + filenodes = {} > > repo = get_repo(url, alias) > prefix = 'refs/hg/%s' % alias ``` ## Felipe Contreras, 2013-04-22 22:35 Subject: Re: [PATCH 01/16] remote-helpers: avoid has_key Message-ID: URL: https://gitlist.dev/e/CAMP44s1XaOOnHCCAHL3C%3DZQFe6v4vOD%2Bn7mwzFT0nEZAwGfqrg%40mail.gmail.com In-Reply-To: <7va9oqnqs9.fsf@alter.siamese.dyndns.org> ``` On Mon, Apr 22, 2013 at 5:28 PM, Junio C Hamano wrote: > Felipe Contreras writes: > >> From: Dusty Phillips >> >> It is deprecated. >> >> [fc: do the same in remote-bzr] >> >> Signed-off-by: Felipe Contreras >> --- > > No sign-off by the author? > > It does not matter for something so trivial like this that there is > no other way to write, but it is a good habit you should encourage > your contributors to acquire, so that you do not have to waste time > with "please sign off" when their next contribution that is more > substantial comes. He is not a contributor, he sent a few patches, and then immediately started gitifyhg, I haven't heard a word from him. Still, he was the original author, so... -- Felipe Contreras ``` ## Felipe Contreras, 2013-04-22 22:38 Subject: Re: [PATCH 13/16] remote-hg: allow refs with spaces Message-ID: URL: https://gitlist.dev/e/CAMP44s0f%2BgJNNESJbYXQgG05Xn0xkkFJMJq6%3DXC9RsnCdAGzMw%40mail.gmail.com In-Reply-To: <7v61zenqm4.fsf@alter.siamese.dyndns.org> ``` On Mon, Apr 22, 2013 at 5:32 PM, Junio C Hamano wrote: > Felipe Contreras writes: > >> Mercurial supports them, Git doesn't. > > Another important thing to note is that you represent a SP with > three underscores on our side, no? Probably, I don't think it's really important. Perhaps some users would benefit from this information, but they would be better served by a README, or a wiki, the commit message doesn't help them. -- Felipe Contreras ``` ## Dusty Phillips, 2013-04-22 22:51 Subject: Re: [PATCH 01/16] remote-helpers: avoid has_key Message-ID: <5175BEF0.7040509@buchuki.com> URL: https://gitlist.dev/e/5175BEF0.7040509%40buchuki.com In-Reply-To: ``` On 04/22/2013 04:35 PM, Felipe Contreras wrote: > On Mon, Apr 22, 2013 at 5:28 PM, Junio C Hamano wrote: >> Felipe Contreras writes: >> >>> From: Dusty Phillips >>> >>> It is deprecated. >>> >>> [fc: do the same in remote-bzr] >>> >>> Signed-off-by: Felipe Contreras >>> --- >> No sign-off by the author? >> >> It does not matter for something so trivial like this that there is >> no other way to write, but it is a good habit you should encourage >> your contributors to acquire, so that you do not have to waste time >> with "please sign off" when their next contribution that is more >> substantial comes. > He is not a contributor, he sent a few patches, and then immediately > started gitifyhg, I haven't heard a word from him. > > Still, he was the original author, so... Acknowledged that I will not likely be contributing any more patches. I apologize for the lack of signoffs on the pull requests; I wasn't actually aware that they were going into git mainline already. I will state for the record that all patches I submitted to Felipe's github fork of git in November and December, 2012 were authored by me and only me and I submitted them with the understanding and expectation that they would be licensed under the GPL. Dusty ``` ## Junio C Hamano, 2013-04-22 23:03 Subject: Re: [PATCH 13/16] remote-hg: allow refs with spaces Message-ID: <7vmwsqmamr.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vmwsqmamr.fsf%40alter.siamese.dyndns.org In-Reply-To: ``` Felipe Contreras writes: > On Mon, Apr 22, 2013 at 5:32 PM, Junio C Hamano wrote: >> Felipe Contreras writes: >> >>> Mercurial supports them, Git doesn't. >> >> Another important thing to note is that you represent a SP with >> three underscores on our side, no? > > Probably, I don't think it's really important. Perhaps some users > would benefit from this information, but they would be better served > by a README, or a wiki, the commit message doesn't help them. True, but it helps a reviewer to notice the missing documentation ;-) ```