threads / patch / 33575

patch, 16 partsremote-hg: second round of improvements

Subject: [PATCH 00/16] remote-hg: second round of improvements

## tl;dr

24 messages between Apr 22, 2013 and Apr 22, 2013. Diffs are folded; open one to read it.

replies: 23people: 3as markdown or json

Felipe Contreras· Apr 22, 2013, 21:55 UTC · lore
Hi,

Now that the safter first round is merged to master, it's time for the second round which is a little tricker. Most of the patches should be safe, but the later ones might be not. All the tests pass, even gitifyhg ones, so everything seems to be fine, but we shall see.

Among the important changes are:
 * Proper support for tags (they are global, and we can push them)
 * Improved e-mail sanitaztion (a bit similar to gitifyhg, but better)
 * Support for hg schemes (like bb://felipec/repo)
 * Support for tags, bookmarks and branches with spaces
 * Performance improvements
Good stuff we should probably merge to master if there are no issues.
Cheers.
Dusty Phillips (1):
  remote-helpers: avoid has_key
Felipe Contreras (15):
  remote-hg: safer bookmark pushing
  remote-hg: use python urlparse
  remote-hg: properly mark branches up-to-date
  remote-hg: add branch_tip() helper
  remote-hg: add support for tag objects
  remote-hg: custom method to write tags
  remote-hg: write tags in the appropriate branch
  remote-hg: add custom local tag write code
  remote-hg: improve email sanitation
  remote-hg: add support for schemes extension
  remote-hg: don't update bookmarks unnecessarily
  remote-hg: allow refs with spaces
  remote-hg: small performance improvement
  remote-hg: use marks instead of inlined files
  remote-hg: strip extra newline
 contrib/remote-helpers/git-remote-bzr |   2 +-
 contrib/remote-helpers/git-remote-hg  | 167 +++++++++++++++++++++++++++-------
 contrib/remote-helpers/test-hg.sh     |   8 +-
 3 files changed, 141 insertions(+), 36 deletions(-)
-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 01/16] remote-helpers: avoid has_key

From: Dusty Phillips <dusty@linux.ca>
It is deprecated.
[fc: do the same in remote-bzr]
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 2 +-
 contrib/remote-helpers/git-remote-hg  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Show changes to 2 files +2 −2

contrib/remote-helpers/git-remote-bzr, contrib/remote-helpers/git-remote-hg

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index aa7bc97..cc6609b 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -94,7 +94,7 @@ class Marks:
         return self.last_mark
 
     def is_marked(self, rev):
-        return self.marks.has_key(rev)
+        return str(rev) in self.marks
 
     def new_mark(self, rev, mark):
         self.marks[rev] = mark
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 5481331..2cd1996 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -129,7 +129,7 @@ class Marks:
         self.last_mark = mark
 
     def is_marked(self, rev):
-        return self.marks.has_key(str(rev))
+        return str(rev) in self.marks
 
     def get_tip(self, branch):
         return self.tips.get(branch, 0)

-- 
1.8.2.1.790.g4588561
Junio C Hamano· Apr 22, 2013, 22:28 UTC · re: Felipe Contreras · lore

Re: [PATCH 01/16] remote-helpers: avoid has_key

Felipe Contreras <felipe.contreras@gmail.com> writes:
Show 8 quoted lines
> From: Dusty Phillips <dusty@linux.ca>
>
> It is deprecated.
>
> [fc: do the same in remote-bzr]
>
> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
> ---
No sign-off by the author?

It does not matter for something so trivial like this that there is no other way to write, but it is a good habit you should encourage your contributors to acquire, so that you do not have to waste time with "please sign off" when their next contribution that is more substantial comes.

Show 30 quoted lines
>  contrib/remote-helpers/git-remote-bzr | 2 +-
>  contrib/remote-helpers/git-remote-hg  | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
> index aa7bc97..cc6609b 100755
> --- a/contrib/remote-helpers/git-remote-bzr
> +++ b/contrib/remote-helpers/git-remote-bzr
> @@ -94,7 +94,7 @@ class Marks:
>          return self.last_mark
>  
>      def is_marked(self, rev):
> -        return self.marks.has_key(rev)
> +        return str(rev) in self.marks
>  
>      def new_mark(self, rev, mark):
>          self.marks[rev] = mark
> diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
> index 5481331..2cd1996 100755
> --- a/contrib/remote-helpers/git-remote-hg
> +++ b/contrib/remote-helpers/git-remote-hg
> @@ -129,7 +129,7 @@ class Marks:
>          self.last_mark = mark
>  
>      def is_marked(self, rev):
> -        return self.marks.has_key(str(rev))
> +        return str(rev) in self.marks
>  
>      def get_tip(self, branch):
>          return self.tips.get(branch, 0)
Felipe Contreras· Apr 22, 2013, 22:35 UTC · re: Junio C Hamano · lore

Re: [PATCH 01/16] remote-helpers: avoid has_key

On Mon, Apr 22, 2013 at 5:28 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 18 quoted lines
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> From: Dusty Phillips <dusty@linux.ca>
>>
>> It is deprecated.
>>
>> [fc: do the same in remote-bzr]
>>
>> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
>> ---
>
> No sign-off by the author?
>
> It does not matter for something so trivial like this that there is
> no other way to write, but it is a good habit you should encourage
> your contributors to acquire, so that you do not have to waste time
> with "please sign off" when their next contribution that is more
> substantial comes.

He is not a contributor, he sent a few patches, and then immediately started gitifyhg, I haven't heard a word from him.

Still, he was the original author, so...
-- 
Felipe Contreras
Dusty Phillips· Apr 22, 2013, 22:51 UTC · re: Felipe Contreras · lore

Re: [PATCH 01/16] remote-helpers: avoid has_key

On 04/22/2013 04:35 PM, Felipe Contreras wrote:
Show 22 quoted lines
> On Mon, Apr 22, 2013 at 5:28 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>
>>> From: Dusty Phillips <dusty@linux.ca>
>>>
>>> It is deprecated.
>>>
>>> [fc: do the same in remote-bzr]
>>>
>>> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
>>> ---
>> No sign-off by the author?
>>
>> It does not matter for something so trivial like this that there is
>> no other way to write, but it is a good habit you should encourage
>> your contributors to acquire, so that you do not have to waste time
>> with "please sign off" when their next contribution that is more
>> substantial comes.
> He is not a contributor, he sent a few patches, and then immediately
> started gitifyhg, I haven't heard a word from him.
>
> Still, he was the original author, so...

Acknowledged that I will not likely be contributing any more patches. I apologize for the lack of signoffs on the pull requests; I wasn't actually aware that they were going into git mainline already. I will state for the record that all patches I submitted to Felipe's github fork of git in November and December, 2012 were authored by me and only me and I submitted them with the understanding and expectation that they would be licensed under the GPL.

Dusty
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 02/16] remote-hg: safer bookmark pushing

It is possible that the remote has changed the bookmarks, so let's fetch them before we make any assumptions, just the way mercurial does.

Probably doesn't make a difference, but better be safe than sorry.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 2 ++
 1 file changed, 2 insertions(+)
Show changes to contrib/remote-helpers/git-remote-hg +2 −0
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 2cd1996..dcf6c98 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -782,6 +782,8 @@ def do_export(parser):
             continue
 
         if peer:
+            rb = peer.listkeys('bookmarks')
+            old = rb.get(bmark, '')
             if not peer.pushkey('bookmarks', bmark, old, new):
                 print "error %s" % ref
                 continue

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 03/16] remote-hg: use python urlparse

It's simpler, and we don't need to depend on certain Mercurial versions.
Also, now we don't update the URL if 'file://' is not present.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
Show changes to contrib/remote-helpers/git-remote-hg +5 −4
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index dcf6c98..b6589a3 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -22,6 +22,7 @@ import shutil
 import subprocess
 import urllib
 import atexit
+import urlparse
 
 #
 # If you want to switch to hg-git compatibility mode:
@@ -793,11 +794,11 @@ def do_export(parser):
     print
 
 def fix_path(alias, repo, orig_url):
-    repo_url = util.url(repo.url())
-    url = util.url(orig_url)
-    if str(url) == str(repo_url):
+    url = urlparse.urlparse(orig_url, 'file')
+    if url.scheme != 'file' or os.path.isabs(url.path):
         return
-    cmd = ['git', 'config', 'remote.%s.url' % alias, "hg::%s" % repo_url]
+    abs_url = urlparse.urljoin("%s/" % os.getcwd(), orig_url)
+    cmd = ['git', 'config', 'remote.%s.url' % alias, "hg::%s" % abs_url]
     subprocess.call(cmd)
 
 def main(args):

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 04/16] remote-hg: properly mark branches up-to-date

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 4 ++++
 1 file changed, 4 insertions(+)
Show changes to contrib/remote-helpers/git-remote-hg +4 −0
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index b6589a3..a4db5b0 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -742,6 +742,10 @@ def do_export(parser):
 
     for ref, node in parsed_refs.iteritems():
         if ref.startswith('refs/heads/branches'):
+            branch = ref[len('refs/heads/branches/'):]
+            if branch in branches and node in branches[branch]:
+                # up to date
+                continue
             print "ok %s" % ref
         elif ref.startswith('refs/heads/'):
             bmark = ref[len('refs/heads/'):]

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 05/16] remote-hg: add branch_tip() helper

Idea from gitifyhg, the backwards compatibility is how Mercurial used to do it.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
Show changes to contrib/remote-helpers/git-remote-hg +8 −3
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index a4db5b0..bd93f82 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -457,6 +457,13 @@ def do_capabilities(parser):
 
     print
 
+def branch_tip(repo, branch):
+    # older versions of mercurial don't have this
+    if hasattr(repo, 'branchtip'):
+        return repo.branchtip(branch)
+    else:
+        return repo.branchtags()[branch]
+
 def get_branch_tip(repo, branch):
     global branches
 
@@ -467,9 +474,7 @@ def get_branch_tip(repo, branch):
     # verify there's only one head
     if (len(heads) > 1):
         warn("Branch '%s' has more than one head, consider merging" % branch)
-        # older versions of mercurial don't have this
-        if hasattr(repo, "branchtip"):
-            return repo.branchtip(branch)
+        return branch_tip(repo, branch)
 
     return heads[0]
 

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 06/16] remote-hg: add support for tag objects

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
Show changes to contrib/remote-helpers/git-remote-hg +7 −3
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index bd93f82..4a1c637 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -722,7 +722,7 @@ def parse_tag(parser):
     data = parser.get_data()
     parser.next()
 
-    # nothing to do
+    parsed_tags[name] = (tagger, data)
 
 def do_export(parser):
     global parsed_refs, bmarks, peer
@@ -758,9 +758,11 @@ def do_export(parser):
             continue
         elif ref.startswith('refs/tags/'):
             tag = ref[len('refs/tags/'):]
+            author, msg = parsed_tags.get(tag, (None, None))
             if mode == 'git':
-                msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));
-                parser.repo.tag([tag], node, msg, False, None, {})
+                if not msg:
+                    msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));
+                parser.repo.tag([tag], node, msg, False, author[0], {})
             else:
                 parser.repo.tag([tag], node, None, True, None, {})
             print "ok %s" % ref
@@ -815,6 +817,7 @@ def main(args):
     global marks, blob_marks, parsed_refs
     global peer, mode, bad_mail, bad_name
     global track_branches, force_push, is_tmp
+    global parsed_tags
 
     alias = args[1]
     url = args[2]
@@ -857,6 +860,7 @@ def main(args):
     blob_marks = {}
     parsed_refs = {}
     marks = None
+    parsed_tags = {}
 
     repo = get_repo(url, alias)
     prefix = 'refs/hg/%s' % alias

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 07/16] remote-hg: custom method to write tags

The one from mercurial is meant for users, on top of the latest tip.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)
Show changes to contrib/remote-helpers/git-remote-hg +32 −1
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 4a1c637..f5e4ba7 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -724,6 +724,37 @@ def parse_tag(parser):
 
     parsed_tags[name] = (tagger, data)
 
+def write_tag(repo, tag, node, msg, author):
+    tip = repo['tip']
+
+    def getfilectx(repo, memctx, f):
+        try:
+            fctx = tip.filectx(f)
+            data = fctx.data()
+        except error.ManifestLookupError:
+            data = ""
+        content = data + "%s %s\n" % (hghex(node), tag)
+        return context.memfilectx(f, content, False, False, None)
+
+    p1 = tip.hex()
+    p2 = '\0' * 20
+    if not author:
+        author = (None, 0, 0)
+    user, date, tz = author
+
+    ctx = context.memctx(repo, (p1, p2), msg,
+            ['.hgtags'], getfilectx,
+            user, (date, tz), {})
+
+    tmp = encoding.encoding
+    encoding.encoding = 'utf-8'
+
+    tagnode = repo.commitctx(ctx)
+
+    encoding.encoding = tmp
+
+    return tagnode
+
 def do_export(parser):
     global parsed_refs, bmarks, peer
 
@@ -762,7 +793,7 @@ def do_export(parser):
             if mode == 'git':
                 if not msg:
                     msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));
-                parser.repo.tag([tag], node, msg, False, author[0], {})
+                write_tag(parser.repo, tag, node, msg, author)
             else:
                 parser.repo.tag([tag], node, None, True, None, {})
             print "ok %s" % ref

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 08/16] remote-hg: write tags in the appropriate branch

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Show changes to contrib/remote-helpers/git-remote-hg +4 −2
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index f5e4ba7..f685990 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -725,7 +725,9 @@ def parse_tag(parser):
     parsed_tags[name] = (tagger, data)
 
 def write_tag(repo, tag, node, msg, author):
-    tip = repo['tip']
+    branch = repo[node].branch()
+    tip = branch_tip(repo, branch)
+    tip = repo[tip]
 
     def getfilectx(repo, memctx, f):
         try:
@@ -744,7 +746,7 @@ def write_tag(repo, tag, node, msg, author):
 
     ctx = context.memctx(repo, (p1, p2), msg,
             ['.hgtags'], getfilectx,
-            user, (date, tz), {})
+            user, (date, tz), {'branch' : branch})
 
     tmp = encoding.encoding
     encoding.encoding = 'utf-8'

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 09/16] remote-hg: add custom local tag write code

There's no point in calling the tag method for such simple action. Not that we care much about the hg-git compat mode, it's mostly just for comparison testing purposes.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Show changes to contrib/remote-helpers/git-remote-hg +3 −1
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index f685990..3c6eeb7 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -797,7 +797,9 @@ def do_export(parser):
                     msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));
                 write_tag(parser.repo, tag, node, msg, author)
             else:
-                parser.repo.tag([tag], node, None, True, None, {})
+                fp = parser.repo.opener('localtags', 'a')
+                fp.write('%s %s\n' % (hghex(node), tag))
+                fp.close()
             print "ok %s" % ref
         else:
             # transport-helper/fast-export bugs

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 10/16] remote-hg: improve email sanitation

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 10 ++++++++--
 contrib/remote-helpers/test-hg.sh    |  8 ++++----
 2 files changed, 12 insertions(+), 6 deletions(-)
Show changes to 2 files +12 −6

contrib/remote-helpers/git-remote-hg, contrib/remote-helpers/test-hg.sh

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 3c6eeb7..0084709 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -51,6 +51,7 @@ import urlparse
 
 NAME_RE = re.compile('^([^<>]+)')
 AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$')
+EMAIL_RE = re.compile('^([^<>]+[^ \\\t<>])?\\b(?:[ \\t<>]*?)\\b([^ \\t<>]+@[^ \\t<>]+)')
 AUTHOR_HG_RE = re.compile('^(.*?) ?<(.*?)(?:>(.+)?)?$')
 RAW_AUTHOR_RE = re.compile('^(\w+) (?:(.+)? )?<(.*)> (\d+) ([+-]\d+)')
 
@@ -245,9 +246,14 @@ def fixup_user_git(user):
         name = m.group(1)
         mail = m.group(2).strip()
     else:
-        m = NAME_RE.match(user)
+        m = EMAIL_RE.match(user)
         if m:
-            name = m.group(1).strip()
+            name = m.group(1)
+            mail = m.group(2)
+        else:
+            m = NAME_RE.match(user)
+            if m:
+                name = m.group(1).strip()
     return (name, mail)
 
 def fixup_user_hg(user):
diff --git a/contrib/remote-helpers/test-hg.sh b/contrib/remote-helpers/test-hg.sh
index d5b8730..8de2aa7 100755
--- a/contrib/remote-helpers/test-hg.sh
+++ b/contrib/remote-helpers/test-hg.sh
@@ -137,15 +137,15 @@ test_expect_success 'authors' '
 
   author_test alpha "" "H G Wells <wells@example.com>" &&
   author_test beta "test" "test <unknown>" &&
-  author_test beta "test <test@example.com> (comment)" "test <unknown>" &&
+  author_test beta "test <test@example.com> (comment)" "test <test@example.com>" &&
   author_test gamma "<test@example.com>" "Unknown <test@example.com>" &&
   author_test delta "name<test@example.com>" "name <test@example.com>" &&
-  author_test epsilon "name <test@example.com" "name <unknown>" &&
+  author_test epsilon "name <test@example.com" "name <test@example.com>" &&
   author_test zeta " test " "test <unknown>" &&
   author_test eta "test < test@example.com >" "test <test@example.com>" &&
-  author_test theta "test >test@example.com>" "test <unknown>" &&
+  author_test theta "test >test@example.com>" "test <test@example.com>" &&
   author_test iota "test < test <at> example <dot> com>" "test <unknown>" &&
-  author_test kappa "test@example.com" "test@example.com <unknown>"
+  author_test kappa "test@example.com" "Unknown <test@example.com>"
   ) &&
 
   git clone "hg::$PWD/hgrepo" gitrepo &&

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 11/16] remote-hg: add support for schemes extension

So that we can use shortened URLs, for example 'bb:://felipec/repo' (Bitbucket).

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Show changes to contrib/remote-helpers/git-remote-hg +7 −1
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 0084709..4f6c7b7 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -12,7 +12,7 @@
 # For remote repositories a local clone is stored in
 # "$GIT_DIR/hg/origin/clone/.hg/".
 
-from mercurial import hg, ui, bookmarks, context, util, encoding, node, error
+from mercurial import hg, ui, bookmarks, context, util, encoding, node, error, extensions
 
 import re
 import sys
@@ -305,6 +305,12 @@ def get_repo(url, alias):
     except subprocess.CalledProcessError:
         pass
 
+    try:
+        mod = extensions.load(myui, 'hgext.schemes', None)
+        mod.extsetup(myui)
+    except ImportError:
+        pass
+
     if hg.islocal(url):
         repo = hg.repository(myui, url)
     else:

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 12/16] remote-hg: don't update bookmarks unnecessarily

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 3 +++
 1 file changed, 3 insertions(+)
Show changes to contrib/remote-helpers/git-remote-hg +3 −0
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 4f6c7b7..dbb4091 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -830,6 +830,9 @@ def do_export(parser):
         else:
             old = ''
 
+        if old == new:
+            continue
+
         if bmark == 'master' and 'master' not in parser.repo._bookmarks:
             # fake bookmark
             pass

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 13/16] remote-hg: allow refs with spaces

Mercurial supports them, Git doesn't.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
Show changes to contrib/remote-helpers/git-remote-hg +17 −8
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index dbb4091..6f4afd7 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -75,6 +75,12 @@ def hgmode(mode):
 def hghex(node):
     return hg.node.hex(node)
 
+def hgref(ref):
+    return ref.replace('___', ' ')
+
+def gitref(ref):
+    return ref.replace(' ', '___')
+
 def get_config(config):
     cmd = ['git', 'config', '--get', config]
     process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
@@ -437,10 +443,10 @@ def export_ref(repo, name, kind, head):
     marks.set_tip(ename, rev)
 
 def export_tag(repo, tag):
-    export_ref(repo, tag, 'tags', repo[tag])
+    export_ref(repo, tag, 'tags', repo[hgref(tag)])
 
 def export_bookmark(repo, bmark):
-    head = bmarks[bmark]
+    head = bmarks[hgref(bmark)]
     export_ref(repo, bmark, 'bookmarks', head)
 
 def export_branch(repo, branch):
@@ -479,14 +485,14 @@ def branch_tip(repo, branch):
 def get_branch_tip(repo, branch):
     global branches
 
-    heads = branches.get(branch, None)
+    heads = branches.get(hgref(branch), None)
     if not heads:
         return None
 
     # verify there's only one head
     if (len(heads) > 1):
         warn("Branch '%s' has more than one head, consider merging" % branch)
-        return branch_tip(repo, branch)
+        return branch_tip(repo, hgref(branch))
 
     return heads[0]
 
@@ -508,6 +514,7 @@ def list_head(repo, cur):
             head = 'master'
         bmarks[head] = node
 
+    head = gitref(head)
     print "@refs/heads/%s HEAD" % head
     g_head = (head, node)
 
@@ -529,15 +536,15 @@ def do_list(parser):
                 branches[branch] = heads
 
         for branch in branches:
-            print "? refs/heads/branches/%s" % branch
+            print "? refs/heads/branches/%s" % gitref(branch)
 
     for bmark in bmarks:
-        print "? refs/heads/%s" % bmark
+        print "? refs/heads/%s" % gitref(bmark)
 
     for tag, node in repo.tagslist():
         if tag == 'tip':
             continue
-        print "? refs/tags/%s" % tag
+        print "? refs/tags/%s" % gitref(tag)
 
     print
 
@@ -674,7 +681,8 @@ def parse_commit(parser):
 
     # Check if the ref is supposed to be a named branch
     if ref.startswith('refs/heads/branches/'):
-        extra['branch'] = ref[len('refs/heads/branches/'):]
+        branch = ref[len('refs/heads/branches/'):]
+        extra['branch'] = hgref(branch)
 
     if mode == 'hg':
         i = data.find('\n--HG--\n')
@@ -803,6 +811,7 @@ def do_export(parser):
             continue
         elif ref.startswith('refs/tags/'):
             tag = ref[len('refs/tags/'):]
+            tag = hgref(tag)
             author, msg = parsed_tags.get(tag, (None, None))
             if mode == 'git':
                 if not msg:

-- 
1.8.2.1.790.g4588561
Junio C Hamano· Apr 22, 2013, 22:32 UTC · re: Felipe Contreras · lore

Re: [PATCH 13/16] remote-hg: allow refs with spaces

Felipe Contreras <felipe.contreras@gmail.com> writes:
> Mercurial supports them, Git doesn't.

Another important thing to note is that you represent a SP with three underscores on our side, no?

Show 97 quoted lines
> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
> ---
>  contrib/remote-helpers/git-remote-hg | 25 +++++++++++++++++--------
>  1 file changed, 17 insertions(+), 8 deletions(-)
>
> diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
> index dbb4091..6f4afd7 100755
> --- a/contrib/remote-helpers/git-remote-hg
> +++ b/contrib/remote-helpers/git-remote-hg
> @@ -75,6 +75,12 @@ def hgmode(mode):
>  def hghex(node):
>      return hg.node.hex(node)
>  
> +def hgref(ref):
> +    return ref.replace('___', ' ')
> +
> +def gitref(ref):
> +    return ref.replace(' ', '___')
> +
>  def get_config(config):
>      cmd = ['git', 'config', '--get', config]
>      process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
> @@ -437,10 +443,10 @@ def export_ref(repo, name, kind, head):
>      marks.set_tip(ename, rev)
>  
>  def export_tag(repo, tag):
> -    export_ref(repo, tag, 'tags', repo[tag])
> +    export_ref(repo, tag, 'tags', repo[hgref(tag)])
>  
>  def export_bookmark(repo, bmark):
> -    head = bmarks[bmark]
> +    head = bmarks[hgref(bmark)]
>      export_ref(repo, bmark, 'bookmarks', head)
>  
>  def export_branch(repo, branch):
> @@ -479,14 +485,14 @@ def branch_tip(repo, branch):
>  def get_branch_tip(repo, branch):
>      global branches
>  
> -    heads = branches.get(branch, None)
> +    heads = branches.get(hgref(branch), None)
>      if not heads:
>          return None
>  
>      # verify there's only one head
>      if (len(heads) > 1):
>          warn("Branch '%s' has more than one head, consider merging" % branch)
> -        return branch_tip(repo, branch)
> +        return branch_tip(repo, hgref(branch))
>  
>      return heads[0]
>  
> @@ -508,6 +514,7 @@ def list_head(repo, cur):
>              head = 'master'
>          bmarks[head] = node
>  
> +    head = gitref(head)
>      print "@refs/heads/%s HEAD" % head
>      g_head = (head, node)
>  
> @@ -529,15 +536,15 @@ def do_list(parser):
>                  branches[branch] = heads
>  
>          for branch in branches:
> -            print "? refs/heads/branches/%s" % branch
> +            print "? refs/heads/branches/%s" % gitref(branch)
>  
>      for bmark in bmarks:
> -        print "? refs/heads/%s" % bmark
> +        print "? refs/heads/%s" % gitref(bmark)
>  
>      for tag, node in repo.tagslist():
>          if tag == 'tip':
>              continue
> -        print "? refs/tags/%s" % tag
> +        print "? refs/tags/%s" % gitref(tag)
>  
>      print
>  
> @@ -674,7 +681,8 @@ def parse_commit(parser):
>  
>      # Check if the ref is supposed to be a named branch
>      if ref.startswith('refs/heads/branches/'):
> -        extra['branch'] = ref[len('refs/heads/branches/'):]
> +        branch = ref[len('refs/heads/branches/'):]
> +        extra['branch'] = hgref(branch)
>  
>      if mode == 'hg':
>          i = data.find('\n--HG--\n')
> @@ -803,6 +811,7 @@ def do_export(parser):
>              continue
>          elif ref.startswith('refs/tags/'):
>              tag = ref[len('refs/tags/'):]
> +            tag = hgref(tag)
>              author, msg = parsed_tags.get(tag, (None, None))
>              if mode == 'git':
>                  if not msg:
Felipe Contreras· Apr 22, 2013, 22:38 UTC · re: Junio C Hamano · lore

Re: [PATCH 13/16] remote-hg: allow refs with spaces

On Mon, Apr 22, 2013 at 5:32 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 6 quoted lines
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> Mercurial supports them, Git doesn't.
>
> Another important thing to note is that you represent a SP with
> three underscores on our side, no?

Probably, I don't think it's really important. Perhaps some users would benefit from this information, but they would be better served by a README, or a wiki, the commit message doesn't help them.

-- 
Felipe Contreras
Junio C Hamano· Apr 22, 2013, 23:03 UTC · re: Felipe Contreras · lore

Re: [PATCH 13/16] remote-hg: allow refs with spaces

Felipe Contreras <felipe.contreras@gmail.com> writes:
Show 11 quoted lines
> On Mon, Apr 22, 2013 at 5:32 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>
>>> Mercurial supports them, Git doesn't.
>>
>> Another important thing to note is that you represent a SP with
>> three underscores on our side, no?
>
> Probably, I don't think it's really important. Perhaps some users
> would benefit from this information, but they would be better served
> by a README, or a wiki, the commit message doesn't help them.

True, but it helps a reviewer to notice the missing documentation ;-)

Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 14/16] remote-hg: small performance improvement

Load previous manifest first as Mercurial does; for caching reasons.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Show changes to contrib/remote-helpers/git-remote-hg +2 −1
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 6f4afd7..f80236b 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -230,8 +230,9 @@ def get_filechanges(repo, ctx, parent):
     added = set()
     removed = set()
 
-    cur = ctx.manifest()
+    # load earliest manifest first for caching reasons
     prev = repo[parent].manifest().copy()
+    cur = ctx.manifest()
 
     for fn in cur:
         if fn in prev:

-- 
1.8.2.1.790.g4588561
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 15/16] remote-hg: use marks instead of inlined files

So that we can find already exported ones. We can never be 100% sure that we already exported such data, due to mercurial design, it at least sometimes we should detect them, and so should give ups some performance boost.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 41 +++++++++++++++++++++++++++++-------
 1 file changed, 33 insertions(+), 8 deletions(-)
Show changes to contrib/remote-helpers/git-remote-hg +33 −8
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index f80236b..d0e552c 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -126,6 +126,10 @@ class Marks:
     def to_rev(self, mark):
         return self.rev_marks[mark]
 
+    def next_mark(self):
+        self.last_mark += 1
+        return self.last_mark
+
     def get_mark(self, rev):
         self.last_mark += 1
         self.marks[str(rev)] = self.last_mark
@@ -218,12 +222,29 @@ def fix_file_path(path):
         return path
     return os.path.relpath(path, '/')
 
-def export_file(fc):
-    d = fc.data()
-    path = fix_file_path(fc.path())
-    print "M %s inline %s" % (gitmode(fc.flags()), path)
-    print "data %d" % len(d)
-    print d
+def export_files(files):
+    global marks, filenodes
+
+    final = []
+    for f in files:
+        fid = node.hex(f.filenode())
+
+        if fid in filenodes:
+            mark = filenodes[fid]
+        else:
+            mark = marks.next_mark()
+            filenodes[fid] = mark
+            d = f.data()
+
+            print "blob"
+            print "mark :%u" % mark
+            print "data %d" % len(d)
+            print d
+
+        path = fix_file_path(f.path())
+        final.append((gitmode(f.flags()), mark, path))
+
+    return final
 
 def get_filechanges(repo, ctx, parent):
     modified = set()
@@ -413,6 +434,8 @@ def export_ref(repo, name, kind, head):
         if len(parents) == 0 and rev:
             print 'reset %s/%s' % (prefix, ename)
 
+        modified_final = export_files(c.filectx(f) for f in modified)
+
         print "commit %s/%s" % (prefix, ename)
         print "mark :%d" % (marks.get_mark(rev))
         print "author %s" % (author)
@@ -425,8 +448,8 @@ def export_ref(repo, name, kind, head):
             if len(parents) > 1:
                 print "merge :%s" % (rev_to_mark(parents[1]))
 
-        for f in modified:
-            export_file(c.filectx(f))
+        for f in modified_final:
+            print "M %s :%u %s" % f
         for f in removed:
             print "D %s" % (fix_file_path(f))
         print
@@ -878,6 +901,7 @@ def main(args):
     global peer, mode, bad_mail, bad_name
     global track_branches, force_push, is_tmp
     global parsed_tags
+    global filenodes
 
     alias = args[1]
     url = args[2]
@@ -921,6 +945,7 @@ def main(args):
     parsed_refs = {}
     marks = None
     parsed_tags = {}
+    filenodes = {}
 
     repo = get_repo(url, alias)
     prefix = 'refs/hg/%s' % alias

-- 
1.8.2.1.790.g4588561
Junio C Hamano· Apr 22, 2013, 22:33 UTC · re: Felipe Contreras · lore

Re: [PATCH 15/16] remote-hg: use marks instead of inlined files

Felipe Contreras <felipe.contreras@gmail.com> writes:
> So that we can find already exported ones. We can never be 100% sure
> that we already exported such data, due to mercurial design, it at least
> sometimes we should detect them, and so should give ups some performance
s/ups/us/ (will locally tweak).
Show 94 quoted lines
> boost.
>
> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
> ---
>  contrib/remote-helpers/git-remote-hg | 41 +++++++++++++++++++++++++++++-------
>  1 file changed, 33 insertions(+), 8 deletions(-)
>
> diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
> index f80236b..d0e552c 100755
> --- a/contrib/remote-helpers/git-remote-hg
> +++ b/contrib/remote-helpers/git-remote-hg
> @@ -126,6 +126,10 @@ class Marks:
>      def to_rev(self, mark):
>          return self.rev_marks[mark]
>  
> +    def next_mark(self):
> +        self.last_mark += 1
> +        return self.last_mark
> +
>      def get_mark(self, rev):
>          self.last_mark += 1
>          self.marks[str(rev)] = self.last_mark
> @@ -218,12 +222,29 @@ def fix_file_path(path):
>          return path
>      return os.path.relpath(path, '/')
>  
> -def export_file(fc):
> -    d = fc.data()
> -    path = fix_file_path(fc.path())
> -    print "M %s inline %s" % (gitmode(fc.flags()), path)
> -    print "data %d" % len(d)
> -    print d
> +def export_files(files):
> +    global marks, filenodes
> +
> +    final = []
> +    for f in files:
> +        fid = node.hex(f.filenode())
> +
> +        if fid in filenodes:
> +            mark = filenodes[fid]
> +        else:
> +            mark = marks.next_mark()
> +            filenodes[fid] = mark
> +            d = f.data()
> +
> +            print "blob"
> +            print "mark :%u" % mark
> +            print "data %d" % len(d)
> +            print d
> +
> +        path = fix_file_path(f.path())
> +        final.append((gitmode(f.flags()), mark, path))
> +
> +    return final
>  
>  def get_filechanges(repo, ctx, parent):
>      modified = set()
> @@ -413,6 +434,8 @@ def export_ref(repo, name, kind, head):
>          if len(parents) == 0 and rev:
>              print 'reset %s/%s' % (prefix, ename)
>  
> +        modified_final = export_files(c.filectx(f) for f in modified)
> +
>          print "commit %s/%s" % (prefix, ename)
>          print "mark :%d" % (marks.get_mark(rev))
>          print "author %s" % (author)
> @@ -425,8 +448,8 @@ def export_ref(repo, name, kind, head):
>              if len(parents) > 1:
>                  print "merge :%s" % (rev_to_mark(parents[1]))
>  
> -        for f in modified:
> -            export_file(c.filectx(f))
> +        for f in modified_final:
> +            print "M %s :%u %s" % f
>          for f in removed:
>              print "D %s" % (fix_file_path(f))
>          print
> @@ -878,6 +901,7 @@ def main(args):
>      global peer, mode, bad_mail, bad_name
>      global track_branches, force_push, is_tmp
>      global parsed_tags
> +    global filenodes
>  
>      alias = args[1]
>      url = args[2]
> @@ -921,6 +945,7 @@ def main(args):
>      parsed_refs = {}
>      marks = None
>      parsed_tags = {}
> +    filenodes = {}
>  
>      repo = get_repo(url, alias)
>      prefix = 'refs/hg/%s' % alias
Felipe Contreras· Apr 22, 2013, 21:55 UTC · re: Felipe Contreras · lore

[PATCH 16/16] remote-hg: strip extra newline

There's no functional change since mercurial commit operation strips that anyway, but that's no excuse for us not to do the right thing. So let's be explicit about it.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 4 ++++
 1 file changed, 4 insertions(+)
Show changes to contrib/remote-helpers/git-remote-hg +4 −0
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index d0e552c..fda4199 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -652,6 +652,10 @@ def parse_commit(parser):
         if parser.check('merge'):
             die('octopus merges are not supported yet')
 
+    # fast-export adds an extra newline
+    if data[-1] == '\n':
+        data = data[:-1]
+
     files = {}
 
     for line in parser:

-- 
1.8.2.1.790.g4588561

← back to recent threads