threads / patch / 33689

v2, 18 partsremote-bzr: massive changes

Subject: [PATCH v2 00/18] remote-bzr: massive changes

## tl;dr

25 messages between May 1, 2013 and Jan 3, 2014. Diffs are folded; open one to read it.

replies: 24people: 3as markdown or json

Felipe Contreras· May 1, 2013, 01:09 UTC · lore

Hi,

This is the same as last series, except that I dropped a couple of conflicting patches, and added one test.

After being contacted by the emacs developers and others who are stuck with Bazaar, which at this point seems to be utterly abandoned, I realized the current implementation is too crude.

Bazaar branches are very simplistic, and our support for them is the same; we need to create one remote per branch. This works nicely if you work on small projects with few branches, but doesn't scale.

Big projects like emacs take a lot of space, and creating a remote per branch is unrealistic, because each remote will have the whole Bazaar repository copied, wasting space, and time each time a remote (for a bzr branch) is set up.

Moreoever, a developer needs to constantly reset the master branch to the commit he/she wants to push before pushing, since the transport-helper infraestructure doesn't support pushing with refspecs (xwidget:master).

That is of course, if pushing actually worked (which in many cases doesn't).

In short, our support for real-world projects suck.

These patches fix all the issues I encountrered.

 1) First of all, there are several improvements for pushing. Before, we failed
    when trying to push a merge, now, even tricky merges work.

 2) Secondly, bzr branches are tied to a transport, so they time out if not
    used for a period of time, and importing/exporting huge chunks of a
    repository do take some time. So now they are only opened when they are
    about to be used.

 3) Then the big one; now bzr repositories are supported. They are very simple:
    basically an object store with no notion of branches, so to find the
    branches we need to traverse a directory (sometimes) remote, to find them.
    This is how Bazaar does it, ableit very slowly. Naturally, a lot of code
    had to be changed to support more than one branch.

 4) In addition, now remotes share all the objects, so adding a new remote
    doesn't imply fetchng a bunch of duplicated objects. They are just re-used
    automagically.

 5) Since the bzr objects are shared now, it only makes sense to share the
    remote-bzr marks, so we don't have to fast-import them again.

 6) The code was also reorganized to keep referenced as few objects as
    possible, since Bazaar seems to be need *a ton* of memory for them.

Finally, after all these changes I was finally able to clone the whole emacs repository, all 130685 commits, and 56 branches without running out of memory in my modest laptop.

Since the purpose of remote-bzr is to actually be usable for the poor souls stucks in DSCMs that are not git, these changes are a must. I propose they be merged for the next major version of git (v1.8.3) if no issues are found. They changes pass all the tests, and work on various repositories I've tried. I'll ask the emacs developers to give them a try, and let's see how it goes.

Felipe Contreras (18):
  remote-bzr: cleanup CustomTree
  remote-bzr: delay blob fetching until the very end
  remote-bzr: fix order of locking in CustomTree
  remote-bzr: always try to update the worktree
  remote-bzr: add support to push merges
  remote-bzr: fixes for branch diverge
  remote-bzr: fix partially pushed merge
  remote-bzr: use branch variable when appropriate
  remote-bzr: add support for bzr repos
  remote-bzr: fix branch names
  remote-bzr: add support for shared repo
  remote-bzr: improve author sanitazion
  remote-bzr: add custom method to find branches
  remote-bzr: add option to specify branches
  remote-bzr: improve progress reporting
  remote-bzr: iterate revisions properly
  remote-bzr: delay peer branch usage
  remote-bzr: access branches only when needed

 contrib/remote-helpers/git-remote-bzr | 305 ++++++++++++++++++++++++----------
 contrib/remote-helpers/test-bzr.sh    |  72 ++++++++
 2 files changed, 293 insertions(+), 84 deletions(-)

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:09 UTC · re: Felipe Contreras · lore

[PATCH v2 01/18] remote-bzr: cleanup CustomTree

This code was not used at all.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +1 −7

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index c19ed0e..7452a57 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -385,9 +385,6 @@ class CustomTree():
     def __init__(self, repo, revid, parents, files):
         global files_cache
 
-        self.repo = repo
-        self.revid = revid
-        self.parents = parents
         self.updates = {}
 
         def copy_tree(revid):
@@ -435,7 +432,7 @@ class CustomTree():
             if basename == '':
                 return None
             fid = bzrlib.generate_ids.gen_file_id(path)
-            d = add_entry(fid, dirname, 'directory')
+            add_entry(fid, dirname, 'directory')
             return fid
 
         def add_entry(fid, path, kind, mode = None):
@@ -458,7 +455,6 @@ class CustomTree():
                     (None, executable))
             self.files[path] = change[0]
             changes.append(change)
-            return change
 
         def update_entry(fid, path, kind, mode = None):
             dirname, basename = os.path.split(path)
@@ -480,7 +476,6 @@ class CustomTree():
                     (None, executable))
             self.files[path] = change[0]
             changes.append(change)
-            return change
 
         def remove_entry(fid, path, kind):
             dirname, basename = os.path.split(path)
@@ -495,7 +490,6 @@ class CustomTree():
                     (None, None))
             del self.files[path]
             changes.append(change)
-            return change
 
         for fid, f in self.updates.iteritems():
             path = f['path']

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:09 UTC · re: Felipe Contreras · lore

[PATCH v2 02/18] remote-bzr: delay blob fetching until the very end

Might be more efficient, but the real reason to use the marks will be revealed in upcoming patches.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +5 −3

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 7452a57..91b5cda 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -506,10 +506,12 @@ class CustomTree():
         return changes
 
     def get_file_with_stat(self, file_id, path=None):
-        return (StringIO.StringIO(self.updates[file_id]['data']), None)
+        mark = self.updates[file_id]['mark']
+        return (StringIO.StringIO(blob_marks[mark]), None)
 
     def get_symlink_target(self, file_id):
-        return self.updates[file_id]['data']
+        mark = self.updates[file_id]['mark']
+        return blob_marks[mark]
 
 def c_style_unescape(string):
     if string[0] == string[-1] == '"':
@@ -553,7 +555,7 @@ def parse_commit(parser):
         if parser.check('M'):
             t, m, mark_ref, path = line.split(' ', 3)
             mark = int(mark_ref[1:])
-            f = { 'mode' : m, 'data' : blob_marks[mark] }
+            f = { 'mode' : m, 'mark' : mark }
         elif parser.check('D'):
             t, path = line.split(' ')
             f = { 'deleted' : True }

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:09 UTC · re: Felipe Contreras · lore

[PATCH v2 03/18] remote-bzr: fix order of locking in CustomTree

It doesn't seem to make any difference, but revision_tree() requires a lock.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Show changes to contrib/remote-helpers/git-remote-bzr +1 −1

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 91b5cda..b428244 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -389,8 +389,8 @@ class CustomTree():
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
-            tree = repo.repository.revision_tree(revid)
             repo.lock_read()
+            tree = repo.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
                     files[path] = entry.file_id

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:09 UTC · re: Felipe Contreras · lore

[PATCH v2 04/18] remote-bzr: always try to update the worktree

And fail properly when we can't.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Show changes to contrib/remote-helpers/git-remote-bzr +5 −1

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index b428244..eb91d28 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -641,9 +641,13 @@ def do_export(parser):
                 except bzrlib.errors.DivergedBranches:
                     print "error %s non-fast forward" % ref
                     continue
-            else:
+
+            try:
                 wt = repo.bzrdir.open_workingtree()
                 wt.update()
+            except bzrlib.errors.NoWorkingTree:
+                pass
+
         print "ok %s" % ref
 
     print

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:09 UTC · re: Felipe Contreras · lore

[PATCH v2 05/18] remote-bzr: add support to push merges

In order to do that, we need to store the marks of every file, so that they can be fetched when needed. Unfortunately we can't tell bazaar that nothing changed, we need to send the data so that it can figure it out by itself.

And since it will be requesting a bunch of information by the file_id, it's better to have a helper dict (rev_files), so that we can fetch it quickly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 29 ++++++++++++++++++++--------
 contrib/remote-helpers/test-bzr.sh    | 36 +++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+), 8 deletions(-)

Show changes to 2 files +57 −8

contrib/remote-helpers/git-remote-bzr, contrib/remote-helpers/test-bzr.sh

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index eb91d28..6a7f836 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -393,7 +393,7 @@ class CustomTree():
             tree = repo.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
-                    files[path] = entry.file_id
+                    files[path] = [entry.file_id, None]
             finally:
                 repo.unlock()
             return files
@@ -408,12 +408,18 @@ class CustomTree():
                 self.base_files = copy_tree(self.base_id)
 
         self.files = files_cache[revid] = self.base_files.copy()
+        self.rev_files = {}
+
+        for path, data in self.files.iteritems():
+            fid, mark = data
+            self.rev_files[fid] = [path, mark]
 
         for path, f in files.iteritems():
-            fid = self.files.get(path, None)
+            fid, mark = self.files.get(path, [None, None])
             if not fid:
                 fid = bzrlib.generate_ids.gen_file_id(path)
             f['path'] = path
+            self.rev_files[fid] = [path, mark]
             self.updates[fid] = f
 
     def last_revision(self):
@@ -423,10 +429,10 @@ class CustomTree():
         changes = []
 
         def get_parent(dirname, basename):
-            parent_fid = self.base_files.get(dirname, None)
+            parent_fid, mark = self.base_files.get(dirname, [None, None])
             if parent_fid:
                 return parent_fid
-            parent_fid = self.files.get(dirname, None)
+            parent_fid, mark = self.files.get(dirname, [None, None])
             if parent_fid:
                 return parent_fid
             if basename == '':
@@ -453,7 +459,7 @@ class CustomTree():
                     (None, basename),
                     (None, kind),
                     (None, executable))
-            self.files[path] = change[0]
+            self.files[path] = [change[0], None]
             changes.append(change)
 
         def update_entry(fid, path, kind, mode = None):
@@ -474,7 +480,7 @@ class CustomTree():
                     (None, basename),
                     (None, kind),
                     (None, executable))
-            self.files[path] = change[0]
+            self.files[path] = [change[0], None]
             changes.append(change)
 
         def remove_entry(fid, path, kind):
@@ -503,16 +509,23 @@ class CustomTree():
             else:
                 add_entry(fid, path, 'file', f['mode'])
 
+            self.files[path][1] = f['mark']
+            self.rev_files[fid][1] = f['mark']
+
         return changes
 
     def get_file_with_stat(self, file_id, path=None):
-        mark = self.updates[file_id]['mark']
+        path, mark = self.rev_files[file_id]
         return (StringIO.StringIO(blob_marks[mark]), None)
 
     def get_symlink_target(self, file_id):
-        mark = self.updates[file_id]['mark']
+        path, mark = self.rev_files[file_id]
         return blob_marks[mark]
 
+    def id2path(self, file_id):
+        path, mark = self.rev_files[file_id]
+        return path
+
 def c_style_unescape(string):
     if string[0] == string[-1] == '"':
         return string.decode('string-escape')[1:-1]
diff --git a/contrib/remote-helpers/test-bzr.sh b/contrib/remote-helpers/test-bzr.sh
index 34666e1..eca3476 100755
--- a/contrib/remote-helpers/test-bzr.sh
+++ b/contrib/remote-helpers/test-bzr.sh
@@ -228,4 +228,40 @@ test_expect_success 'push utf-8 filenames' '
   test_cmp expected actual
 '
 
+test_expect_success 'pushing a merge' '
+  mkdir -p tmp && cd tmp &&
+  test_when_finished "cd .. && rm -rf tmp" &&
+
+  (
+  bzr init bzrrepo &&
+  cd bzrrepo &&
+  echo one > content &&
+  bzr add content &&
+  bzr commit -m one
+  ) &&
+
+  git clone "bzr::$PWD/bzrrepo" gitrepo &&
+
+  (
+  cd bzrrepo &&
+  echo two > content &&
+  bzr commit -m two
+  ) &&
+
+  (
+  cd gitrepo &&
+  echo three > content &&
+  git commit -a -m three &&
+  git fetch &&
+  git merge origin/master || true &&
+  echo three > content &&
+  git commit -a --no-edit &&
+  git push
+  ) &&
+
+  echo three > expected &&
+  cat bzrrepo/content > actual &&
+  test_cmp expected actual
+'
+
 test_done

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:09 UTC · re: Felipe Contreras · lore

[PATCH v2 06/18] remote-bzr: fixes for branch diverge

If the branches diverge we want to reset the pointer to where the remote actually is. Since we can access remote branches just as easily as local ones, let's do so.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

Show changes to contrib/remote-helpers/git-remote-bzr +7 −1

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 6a7f836..bf254a0 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -354,6 +354,7 @@ def do_import(parser):
     if os.path.exists(path):
         print "feature import-marks=%s" % path
     print "feature export-marks=%s" % path
+    print "feature force"
     sys.stdout.flush()
 
     while parser.check('import'):
@@ -716,7 +717,12 @@ def get_repo(url, alias):
             # pull
             d = bzrlib.bzrdir.BzrDir.open(clone_path)
             branch = d.open_branch()
-            result = branch.pull(remote_branch, [], None, False)
+            try:
+                result = branch.pull(remote_branch, [], None, False)
+            except bzrlib.errors.DivergedBranches:
+                # use remote branch for now
+                peer = None
+                return remote_branch
         else:
             # clone
             d = origin.sprout(clone_path, None,

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:09 UTC · re: Felipe Contreras · lore

[PATCH v2 07/18] remote-bzr: fix partially pushed merge

If part of the merge was already pushed, we don't have the blob_marks available, however, the commits are already stored in bazaar, so we can use the revision_tree to fetch the contents.

We want to do this only when there's no other option.

There's no easy way to test this.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +13 −4

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index bf254a0..fdead31 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -387,6 +387,7 @@ class CustomTree():
         global files_cache
 
         self.updates = {}
+        self.branch = repo
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
@@ -515,13 +516,21 @@ class CustomTree():
 
         return changes
 
-    def get_file_with_stat(self, file_id, path=None):
+    def get_content(self, file_id):
         path, mark = self.rev_files[file_id]
-        return (StringIO.StringIO(blob_marks[mark]), None)
+        if mark:
+            return blob_marks[mark]
+
+        # last resort
+        tree = self.branch.repository.revision_tree(self.base_id)
+        return tree.get_file_text(file_id)
+
+    def get_file_with_stat(self, file_id, path=None):
+        content = self.get_content(file_id)
+        return (StringIO.StringIO(content), None)
 
     def get_symlink_target(self, file_id):
-        path, mark = self.rev_files[file_id]
-        return blob_marks[mark]
+        return self.get_content(file_id)
 
     def id2path(self, file_id):
         path, mark = self.rev_files[file_id]

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 08/18] remote-bzr: use branch variable when appropriate

There should be no functional changes. Basically we want to reserve the 'repo' variable.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +18 −17

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index fdead31..0c67236 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -257,7 +257,8 @@ def export_branch(branch, name):
     tip = marks.get_tip(name)
 
     repo = branch.repository
-    repo.lock_read()
+
+    branch.lock_read()
     revs = branch.iter_merge_sorted_revisions(None, tip, 'exclude', 'forward')
     count = 0
 
@@ -325,7 +326,7 @@ def export_branch(branch, name):
             print "progress revision %s (%d/%d)" % (revid, count, len(revs))
             print "#############################################################"
 
-    repo.unlock()
+    branch.unlock()
 
     revid = branch.last_revision()
 
@@ -383,21 +384,21 @@ def parse_blob(parser):
 
 class CustomTree():
 
-    def __init__(self, repo, revid, parents, files):
+    def __init__(self, branch, revid, parents, files):
         global files_cache
 
         self.updates = {}
-        self.branch = repo
+        self.branch = branch
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
-            repo.lock_read()
-            tree = repo.repository.revision_tree(revid)
+            branch.lock_read()
+            tree = branch.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
                     files[path] = [entry.file_id, None]
             finally:
-                repo.unlock()
+                branch.unlock()
             return files
 
         if len(parents) == 0:
@@ -587,20 +588,20 @@ def parse_commit(parser):
         path = c_style_unescape(path).decode('utf-8')
         files[path] = f
 
-    repo = parser.repo
+    branch = parser.repo
 
     committer, date, tz = committer
     parents = [str(mark_to_rev(p)) for p in parents]
     revid = bzrlib.generate_ids.gen_revision_id(committer, date)
     props = {}
-    props['branch-nick'] = repo.nick
+    props['branch-nick'] = branch.nick
 
-    mtree = CustomTree(repo, revid, parents, files)
+    mtree = CustomTree(branch, revid, parents, files)
     changes = mtree.iter_changes()
 
-    repo.lock_write()
+    branch.lock_write()
     try:
-        builder = repo.get_commit_builder(parents, None, date, tz, committer, props, revid)
+        builder = branch.get_commit_builder(parents, None, date, tz, committer, props, revid)
         try:
             list(builder.record_iter_changes(mtree, mtree.last_revision(), changes))
             builder.finish_inventory()
@@ -609,7 +610,7 @@ def parse_commit(parser):
             builder.abort()
             raise
     finally:
-        repo.unlock()
+        branch.unlock()
 
     parsed_refs[ref] = revid
     marks.new_mark(revid, commit_mark)
@@ -653,20 +654,20 @@ def do_export(parser):
         else:
             die('unhandled export command: %s' % line)
 
-    repo = parser.repo
+    branch = parser.repo
 
     for ref, revid in parsed_refs.iteritems():
         if ref == 'refs/heads/master':
-            repo.generate_revision_history(revid, marks.get_tip('master'))
+            branch.generate_revision_history(revid, marks.get_tip('master'))
             if peer:
                 try:
-                    repo.push(peer, stop_revision=revid)
+                    branch.push(peer, stop_revision=revid)
                 except bzrlib.errors.DivergedBranches:
                     print "error %s non-fast forward" % ref
                     continue
 
             try:
-                wt = repo.bzrdir.open_workingtree()
+                wt = branch.bzrdir.open_workingtree()
                 wt.update()
             except bzrlib.errors.NoWorkingTree:
                 pass

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 09/18] remote-bzr: add support for bzr repos

In bazaar, a repository can contain multiple branches, and previously we were supporting only one branch at a time. Now we fetch them all.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 158 ++++++++++++++++++++++------------
 contrib/remote-helpers/test-bzr.sh    |  36 ++++++++
 2 files changed, 138 insertions(+), 56 deletions(-)

Show changes to 2 files +138 −56

contrib/remote-helpers/git-remote-bzr, contrib/remote-helpers/test-bzr.sh

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 0c67236..3b33061 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -27,6 +27,7 @@ import bzrlib.generate_ids
 import bzrlib.transport
 import bzrlib.errors
 import bzrlib.ui
+import bzrlib.urlutils
 
 import sys
 import os
@@ -250,12 +251,13 @@ def export_files(tree, files):
 
     return final
 
-def export_branch(branch, name):
+def export_branch(repo, name):
     global prefix
 
     ref = '%s/heads/%s' % (prefix, name)
     tip = marks.get_tip(name)
 
+    branch = branches[name]
     repo = branch.repository
 
     branch.lock_read()
@@ -323,7 +325,7 @@ def export_branch(branch, name):
 
         count += 1
         if (count % 100 == 0):
-            print "progress revision %s (%d/%d)" % (revid, count, len(revs))
+            print "progress revision %s '%s' (%d/%d)" % (revid, name, count, len(revs))
             print "#############################################################"
 
     branch.unlock()
@@ -348,7 +350,7 @@ def export_tag(repo, name):
 def do_import(parser):
     global dirname
 
-    branch = parser.repo
+    repo = parser.repo
     path = os.path.join(dirname, 'marks-git')
 
     print "feature done"
@@ -362,10 +364,10 @@ def do_import(parser):
         ref = parser[1]
         if ref.startswith('refs/heads/'):
             name = ref[len('refs/heads/'):]
-            export_branch(branch, name)
+            export_branch(repo, name)
         if ref.startswith('refs/tags/'):
             name = ref[len('refs/tags/'):]
-            export_tag(branch, name)
+            export_tag(repo, name)
         parser.next()
 
     print 'done'
@@ -551,8 +553,11 @@ def parse_commit(parser):
     ref = parser[1]
     parser.next()
 
-    if ref != 'refs/heads/master':
-        die("bzr doesn't support multiple branches; use 'master'")
+    if ref.startswith('refs/heads/'):
+        name = ref[len('refs/heads/'):]
+        branch = branches[name]
+    else:
+        die('unknown ref')
 
     commit_mark = parser.get_mark()
     parser.next()
@@ -588,8 +593,6 @@ def parse_commit(parser):
         path = c_style_unescape(path).decode('utf-8')
         files[path] = f
 
-    branch = parser.repo
-
     committer, date, tz = committer
     parents = [str(mark_to_rev(p)) for p in parents]
     revid = bzrlib.generate_ids.gen_revision_id(committer, date)
@@ -621,9 +624,6 @@ def parse_reset(parser):
     ref = parser[1]
     parser.next()
 
-    if ref != 'refs/heads/master':
-        die("bzr doesn't support multiple branches; use 'master'")
-
     # ugh
     if parser.check('commit'):
         parse_commit(parser)
@@ -636,7 +636,7 @@ def parse_reset(parser):
     parsed_refs[ref] = mark_to_rev(from_mark)
 
 def do_export(parser):
-    global parsed_refs, dirname, peer
+    global parsed_refs, dirname
 
     parser.next()
 
@@ -654,23 +654,24 @@ def do_export(parser):
         else:
             die('unhandled export command: %s' % line)
 
-    branch = parser.repo
-
     for ref, revid in parsed_refs.iteritems():
-        if ref == 'refs/heads/master':
-            branch.generate_revision_history(revid, marks.get_tip('master'))
-            if peer:
-                try:
-                    branch.push(peer, stop_revision=revid)
-                except bzrlib.errors.DivergedBranches:
-                    print "error %s non-fast forward" % ref
-                    continue
+        name = ref[len('refs/heads/'):]
+        branch = branches[name]
+        branch.generate_revision_history(revid, marks.get_tip(name))
 
+        if name in peers:
+            peer = peers[name]
             try:
-                wt = branch.bzrdir.open_workingtree()
-                wt.update()
-            except bzrlib.errors.NoWorkingTree:
-                pass
+                peer.bzrdir.push_branch(branch, revision_id=revid)
+            except bzrlib.errors.DivergedBranches:
+                print "error %s non-fast forward" % ref
+                continue
+
+        try:
+            wt = branch.bzrdir.open_workingtree()
+            wt.update()
+        except bzrlib.errors.NoWorkingTree:
+            pass
 
         print "ok %s" % ref
 
@@ -697,9 +698,15 @@ def ref_is_valid(name):
 
 def do_list(parser):
     global tags
-    print "? refs/heads/%s" % 'master'
 
-    branch = parser.repo
+    master_branch = None
+
+    for name in branches:
+        if not master_branch:
+            master_branch = name
+        print "? refs/heads/%s" % name
+
+    branch = branches[master_branch]
     branch.lock_read()
     for tag, revid in branch.tags.get_tag_dict().items():
         try:
@@ -711,41 +718,77 @@ def do_list(parser):
         print "? refs/tags/%s" % tag
         tags[tag] = revid
     branch.unlock()
-    print "@refs/heads/%s HEAD" % 'master'
+
+    print "@refs/heads/%s HEAD" % master_branch
     print
 
+def get_remote_branch(origin, remote_branch, name):
+    global dirname, peers
+
+    branch_path = os.path.join(dirname, 'clone', name)
+    if os.path.exists(branch_path):
+        # pull
+        d = bzrlib.bzrdir.BzrDir.open(branch_path)
+        branch = d.open_branch()
+        try:
+            branch.pull(remote_branch, [], None, False)
+        except bzrlib.errors.DivergedBranches:
+            # use remote branch for now
+            return remote_branch
+    else:
+        # clone
+        d = origin.sprout(branch_path, None,
+                hardlink=True, create_tree_if_local=False,
+                force_new_repo=False,
+                source_branch=remote_branch)
+        branch = d.open_branch()
+
+    return branch
+
 def get_repo(url, alias):
-    global dirname, peer
+    global dirname, peer, branches
 
+    normal_url = bzrlib.urlutils.normalize_url(url)
     origin = bzrlib.bzrdir.BzrDir.open(url)
-    branch = origin.open_branch()
-
-    if not isinstance(origin.transport, bzrlib.transport.local.LocalTransport):
-        clone_path = os.path.join(dirname, 'clone')
-        remote_branch = branch
-        if os.path.exists(clone_path):
-            # pull
-            d = bzrlib.bzrdir.BzrDir.open(clone_path)
-            branch = d.open_branch()
-            try:
-                result = branch.pull(remote_branch, [], None, False)
-            except bzrlib.errors.DivergedBranches:
-                # use remote branch for now
-                peer = None
-                return remote_branch
+    is_local = isinstance(origin.transport, bzrlib.transport.local.LocalTransport)
+
+    clone_path = os.path.join(dirname, 'clone')
+
+    try:
+        repo = origin.open_repository()
+    except bzrlib.errors.NoRepositoryPresent:
+        # branch
+
+        name = 'master'
+        branch = origin.open_branch()
+
+        if not is_local:
+            if not os.path.exists(clone_path):
+                os.mkdir(clone_path)
+            peers[name] = branch
+            branches[name] = get_remote_branch(origin, branch, name)
         else:
-            # clone
-            d = origin.sprout(clone_path, None,
-                    hardlink=True, create_tree_if_local=False,
-                    source_branch=remote_branch)
-            branch = d.open_branch()
-            branch.bind(remote_branch)
-
-        peer = remote_branch
+            branches[name] = branch
+
+        return branch.repository
     else:
-        peer = None
+        # repository
 
-    return branch
+        if not is_local and not os.path.exists(clone_path):
+            clonedir = bzrlib.bzrdir.BzrDir.create(clone_path)
+
+        for branch in repo.find_branches():
+
+            name = repo.user_transport.relpath(branch.base)
+            name = name if name != '' else 'master'
+
+            if not is_local:
+                peers[name] = branch
+                branches[name] = get_remote_branch(origin, branch, name)
+            else:
+                branches[name] = branch
+
+        return repo
 
 def fix_path(alias, orig_url):
     url = urlparse.urlparse(orig_url, 'file')
@@ -762,6 +805,7 @@ def main(args):
     global parsed_refs
     global files_cache
     global is_tmp
+    global branches, peers
 
     alias = args[1]
     url = args[2]
@@ -772,6 +816,8 @@ def main(args):
     parsed_refs = {}
     files_cache = {}
     marks = None
+    branches = {}
+    peers = {}
 
     if alias[5:] == url:
         is_tmp = True
diff --git a/contrib/remote-helpers/test-bzr.sh b/contrib/remote-helpers/test-bzr.sh
index eca3476..cec55f1 100755
--- a/contrib/remote-helpers/test-bzr.sh
+++ b/contrib/remote-helpers/test-bzr.sh
@@ -264,4 +264,40 @@ test_expect_success 'pushing a merge' '
   test_cmp expected actual
 '
 
+cat > expected <<EOF
+origin/HEAD
+origin/branch
+origin/trunk
+EOF
+
+test_expect_success 'proper bzr repo' '
+  mkdir -p tmp && cd tmp &&
+  test_when_finished "cd .. && rm -rf tmp" &&
+
+  bzr init-repo bzrrepo &&
+
+  bzr init bzrrepo/trunk &&
+  (
+  cd bzrrepo/trunk &&
+  echo one >> content &&
+  bzr add content &&
+  bzr commit -m one
+  ) &&
+
+  bzr branch bzrrepo/trunk bzrrepo/branch &&
+  (
+  cd bzrrepo/branch &&
+  echo two >> content &&
+  bzr commit -m one
+  ) &&
+
+  git clone "bzr::$PWD/bzrrepo" gitrepo &&
+  (
+  cd gitrepo &&
+  git for-each-ref --format "%(refname:short)" refs/remotes/origin > ../actual
+  ) &&
+
+  test_cmp ../expected actual
+'
+
 test_done

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 10/18] remote-bzr: fix branch names

When branches have '/' in their name (aka. sub-branches), bazaar seems to choke while creating the new directory.

Also, git cannot have both 'foo' and 'foo/bar'.

So let's replace slashes with a plus sign.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 1 +
 1 file changed, 1 insertion(+)

Show changes to contrib/remote-helpers/git-remote-bzr +1 −0

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 3b33061..38cec07 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -781,6 +781,7 @@ def get_repo(url, alias):
 
             name = repo.user_transport.relpath(branch.base)
             name = name if name != '' else 'master'
+            name = name.replace('/', '+')
 
             if not is_local:
                 peers[name] = branch

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 11/18] remote-bzr: add support for shared repo

This way all the remotes share the same data, so adding multiple remotes, or renaming them doesn't create extra overhead.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +15 −7

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 38cec07..9fe830e 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -752,7 +752,20 @@ def get_repo(url, alias):
     origin = bzrlib.bzrdir.BzrDir.open(url)
     is_local = isinstance(origin.transport, bzrlib.transport.local.LocalTransport)
 
-    clone_path = os.path.join(dirname, 'clone')
+    shared_path = os.path.join(gitdir, 'bzr')
+    try:
+        shared_dir = bzrlib.bzrdir.BzrDir.open(shared_path)
+    except bzrlib.errors.NotBranchError:
+        shared_dir = bzrlib.bzrdir.BzrDir.create(shared_path)
+    try:
+        shared_repo = shared_dir.open_repository()
+    except bzrlib.errors.NoRepositoryPresent:
+        shared_repo = shared_dir.create_repository(shared=True)
+
+    if not is_local:
+        clone_path = os.path.join(dirname, 'clone')
+        if not os.path.exists(clone_path):
+            os.mkdir(clone_path)
 
     try:
         repo = origin.open_repository()
@@ -763,8 +776,6 @@ def get_repo(url, alias):
         branch = origin.open_branch()
 
         if not is_local:
-            if not os.path.exists(clone_path):
-                os.mkdir(clone_path)
             peers[name] = branch
             branches[name] = get_remote_branch(origin, branch, name)
         else:
@@ -774,9 +785,6 @@ def get_repo(url, alias):
     else:
         # repository
 
-        if not is_local and not os.path.exists(clone_path):
-            clonedir = bzrlib.bzrdir.BzrDir.create(clone_path)
-
         for branch in repo.find_branches():
 
             name = repo.user_transport.relpath(branch.base)
@@ -800,7 +808,7 @@ def fix_path(alias, orig_url):
     subprocess.call(cmd)
 
 def main(args):
-    global marks, prefix, dirname
+    global marks, prefix, gitdir, dirname
     global tags, filenodes
     global blob_marks
     global parsed_refs

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 12/18] remote-bzr: improve author sanitazion

So that we don't end up with '<None>', and also synchronize it with the one from remote-hg.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +13 −2

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 9fe830e..af46016 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -38,6 +38,7 @@ import atexit, shutil, hashlib, urlparse, subprocess
 
 NAME_RE = re.compile('^([^<>]+)')
 AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$')
+EMAIL_RE = re.compile('^([^<>]+[^ \\\t<>])?\\b(?:[ \\t<>]*?)\\b([^ \\t<>]+@[^ \\t<>]+)')
 RAW_AUTHOR_RE = re.compile('^(\w+) (.+)? <(.*)> (\d+) ([+-]\d+)')
 
 def die(msg, *args):
@@ -175,9 +176,19 @@ def fixup_user(user):
         name = m.group(1)
         mail = m.group(2).strip()
     else:
-        m = NAME_RE.match(user)
+        m = EMAIL_RE.match(user)
         if m:
-            name = m.group(1).strip()
+            name = m.group(1)
+            mail = m.group(2)
+        else:
+            m = NAME_RE.match(user)
+            if m:
+                name = m.group(1).strip()
+
+    if not name:
+        name = 'unknown'
+    if not mail:
+        mail = 'Unknown'
 
     return '%s <%s>' % (name, mail)

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 13/18] remote-bzr: add custom method to find branches

The official method is incredibly inefficient and slow.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +20 −5

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index af46016..11f2415 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -756,6 +756,25 @@ def get_remote_branch(origin, remote_branch, name):
 
     return branch
 
+def find_branches(repo):
+    transport = repo.user_transport
+
+    for fn in transport.iter_files_recursive():
+        if not fn.endswith('.bzr/branch-format'):
+            continue
+
+        name = subdir = fn[:-len('/.bzr/branch-format')]
+        name = name if name != '' else 'master'
+        name = name.replace('/', '+')
+
+        try:
+            cur = transport.clone(subdir)
+            branch = bzrlib.branch.Branch.open_from_transport(cur)
+        except bzrlib.errors.NotBranchError:
+            continue
+        else:
+            yield name, branch
+
 def get_repo(url, alias):
     global dirname, peer, branches
 
@@ -796,11 +815,7 @@ def get_repo(url, alias):
     else:
         # repository
 
-        for branch in repo.find_branches():
-
-            name = repo.user_transport.relpath(branch.base)
-            name = name if name != '' else 'master'
-            name = name.replace('/', '+')
+        for name, branch in find_branches(repo):
 
             if not is_local:
                 peers[name] = branch

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 14/18] remote-bzr: add option to specify branches

We might not want all the branches. And branch handling in bazaar is rather tricky, so it's safer to simply specify them.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +18 −2

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 11f2415..d284afc 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -13,6 +13,9 @@
 # or
 # % git clone bzr::lp:myrepo
 #
+# If you want to specify which branches you want track (per repo):
+# git config remote-bzr.branches 'trunk, devel, test'
+#
 
 import sys
 
@@ -51,6 +54,12 @@ def warn(msg, *args):
 def gittz(tz):
     return '%+03d%02d' % (tz / 3600, tz % 3600 / 60)
 
+def get_config(config):
+    cmd = ['git', 'config', '--get', config]
+    process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+    output, _ = process.communicate()
+    return output
+
 class Marks:
 
     def __init__(self, path):
@@ -756,7 +765,7 @@ def get_remote_branch(origin, remote_branch, name):
 
     return branch
 
-def find_branches(repo):
+def find_branches(repo, wanted):
     transport = repo.user_transport
 
     for fn in transport.iter_files_recursive():
@@ -767,6 +776,9 @@ def find_branches(repo):
         name = name if name != '' else 'master'
         name = name.replace('/', '+')
 
+        if wanted and not name in wanted:
+            continue
+
         try:
             cur = transport.clone(subdir)
             branch = bzrlib.branch.Branch.open_from_transport(cur)
@@ -815,7 +827,11 @@ def get_repo(url, alias):
     else:
         # repository
 
-        for name, branch in find_branches(repo):
+        wanted = get_config('remote-bzr.branches').rstrip().split(', ')
+        # stupid python
+        wanted = [e for e in wanted if e]
+
+        for name, branch in find_branches(repo, wanted):
 
             if not is_local:
                 peers[name] = branch

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 15/18] remote-bzr: improve progress reporting

No need to manually count the revisions, and also, this would help to iterate more properly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +13 −7

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index d284afc..2e21008 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -282,13 +282,16 @@ def export_branch(repo, name):
 
     branch.lock_read()
     revs = branch.iter_merge_sorted_revisions(None, tip, 'exclude', 'forward')
-    count = 0
+    tip_revno = branch.revision_id_to_revno(tip)
+    last_revno, _ = branch.last_revision_info()
+    total = last_revno - tip_revno
 
-    revs = [revid for revid, _, _, _ in revs if not marks.is_marked(revid)]
+    revs = [revid, seq for revid, _, seq, _ in revs if not marks.is_marked(revid)]
 
-    for revid in revs:
+    for revid, seq in revs:
 
         rev = repo.get_revision(revid)
+        revno = seq[0]
 
         parents = rev.parent_ids
         time = rev.timestamp
@@ -343,10 +346,13 @@ def export_branch(repo, name):
             print "M %s :%u %s" % f
         print
 
-        count += 1
-        if (count % 100 == 0):
-            print "progress revision %s '%s' (%d/%d)" % (revid, name, count, len(revs))
-            print "#############################################################"
+        if len(seq) > 1:
+            # let's skip branch revisions from the progress report
+            continue
+
+        progress = (revno - tip_revno)
+        if (progress % 100 == 0):
+            print "progress revision %d '%s' (%d/%d)" % (revno, name, progress, total)
 
     branch.unlock()

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 16/18] remote-bzr: iterate revisions properly

This way we don't need to store the list of all the revisions, which doesn't seem to be very memory efficient with bazaar's design, for whatever reason.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +3 −2

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 2e21008..66e3861 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -286,9 +286,10 @@ def export_branch(repo, name):
     last_revno, _ = branch.last_revision_info()
     total = last_revno - tip_revno
 
-    revs = [revid, seq for revid, _, seq, _ in revs if not marks.is_marked(revid)]
+    for revid, _, seq, _ in revs:
 
-    for revid, seq in revs:
+        if marks.is_marked(revid):
+            continue
 
         rev = repo.get_revision(revid)
         revno = seq[0]

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 17/18] remote-bzr: delay peer branch usage

So it doesn't time out.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +3 −3

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 66e3861..b2d67f7 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -687,7 +687,7 @@ def do_export(parser):
         branch.generate_revision_history(revid, marks.get_tip(name))
 
         if name in peers:
-            peer = peers[name]
+            peer = bzrlib.branch.Branch.open(peers[name])
             try:
                 peer.bzrdir.push_branch(branch, revision_id=revid)
             except bzrlib.errors.DivergedBranches:
@@ -825,7 +825,7 @@ def get_repo(url, alias):
         branch = origin.open_branch()
 
         if not is_local:
-            peers[name] = branch
+            peers[name] = branch.base
             branches[name] = get_remote_branch(origin, branch, name)
         else:
             branches[name] = branch
@@ -841,7 +841,7 @@ def get_repo(url, alias):
         for name, branch in find_branches(repo, wanted):
 
             if not is_local:
-                peers[name] = branch
+                peers[name] = branch.base
                 branches[name] = get_remote_branch(origin, branch, name)
             else:
                 branches[name] = branch

-- 
1.8.3.rc0.399.gc96a135

Felipe Contreras· May 1, 2013, 01:10 UTC · re: Felipe Contreras · lore

[PATCH v2 18/18] remote-bzr: access branches only when needed

Bazaar doesn't seem to be tested for multiple usage of branches, so resources seem to be leaked all over. Let's try to minimize this by accessing the Branch objects only when needed.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)

Show changes to contrib/remote-helpers/git-remote-bzr +16 −12

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index b2d67f7..161f831 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -277,7 +277,7 @@ def export_branch(repo, name):
     ref = '%s/heads/%s' % (prefix, name)
     tip = marks.get_tip(name)
 
-    branch = branches[name]
+    branch = bzrlib.branch.Branch.open(branches[name])
     repo = branch.repository
 
     branch.lock_read()
@@ -582,7 +582,7 @@ def parse_commit(parser):
 
     if ref.startswith('refs/heads/'):
         name = ref[len('refs/heads/'):]
-        branch = branches[name]
+        branch = bzrlib.branch.Branch.open(branches[name])
     else:
         die('unknown ref')
 
@@ -683,7 +683,7 @@ def do_export(parser):
 
     for ref, revid in parsed_refs.iteritems():
         name = ref[len('refs/heads/'):]
-        branch = branches[name]
+        branch = bzrlib.branch.Branch.open(branches[name])
         branch.generate_revision_history(revid, marks.get_tip(name))
 
         if name in peers:
@@ -733,7 +733,7 @@ def do_list(parser):
             master_branch = name
         print "? refs/heads/%s" % name
 
-    branch = branches[master_branch]
+    branch = bzrlib.branch.Branch.open(branches[master_branch])
     branch.lock_read()
     for tag, revid in branch.tags.get_tag_dict().items():
         try:
@@ -822,13 +822,15 @@ def get_repo(url, alias):
         # branch
 
         name = 'master'
-        branch = origin.open_branch()
+        remote_branch = origin.open_branch()
 
         if not is_local:
-            peers[name] = branch.base
-            branches[name] = get_remote_branch(origin, branch, name)
+            peers[name] = remote_branch.base
+            branch = get_remote_branch(origin, remote_branch, name)
         else:
-            branches[name] = branch
+            branch = remote_branch
+
+        branches[name] = branch.base
 
         return branch.repository
     else:
@@ -838,13 +840,15 @@ def get_repo(url, alias):
         # stupid python
         wanted = [e for e in wanted if e]
 
-        for name, branch in find_branches(repo, wanted):
+        for name, remote_branch in find_branches(repo, wanted):
 
             if not is_local:
-                peers[name] = branch.base
-                branches[name] = get_remote_branch(origin, branch, name)
+                peers[name] = remote_branch.base
+                branch = get_remote_branch(origin, remote_branch, name)
             else:
-                branches[name] = branch
+                branch = remote_branch
+
+            branches[name] = branch.base
 
         return repo

-- 
1.8.3.rc0.399.gc96a135

Junio C Hamano· May 1, 2013, 05:44 UTC · re: Felipe Contreras · lore

Re: [PATCH v2 00/18] remote-bzr: massive changes

Felipe Contreras <felipe.contreras@gmail.com> writes:

Show 13 quoted lines

> After being contacted by the emacs developers and others who are stuck with
> Bazaar, which at this point seems to be utterly abandoned, I realized the
> current implementation is too crude.
> ...
> That is of course, if pushing actually worked (which in many cases doesn't).
>
> In short, our support for real-world projects suck.
>
> These patches fix all the issues I encountrered.
> ...
> Finally, after all these changes I was finally able to clone the whole emacs
> repository, all 130685 commits, and 56 branches without running out of memory
> in my modest laptop.

Yay ;-)

I assume that the trees at a handful of key points (e.g. releases) were verified to be identical with the original history and the conversion result.

Show 5 quoted lines

> Since the purpose of remote-bzr is to actually be usable for the
> poor souls stucks in DSCMs that are not git, these changes are a
> must. I propose they be merged for the next major version of git
> (v1.8.3) if no issues are found.  They changes pass all the tests,
> and work on various repositories I've tried.

Nice.

> I'll ask the emacs developers to give them a try, and let's see
> how it goes.

Yeah, that's the least we can do for both existing and future users.

Generally speaking, post -rc0 is too late for "if no issues are found", simply because no existing user has enough time to find corner case regressions in her work using the new software (I do not expect a trivial bug that can be uncovered in a few weeks of use would remain in a version that has successfully converted the Emacs history; but real world users always have different needs than what we anticipate).

I however am finding myself moderately receptive to this series. That is primarily because this series touches only two files that are totally isolated from the rest of the system. Even if they did not work at all, there is no risk for the remainder of Git. Nobody other than existing users of remote-bzr will even notice if we merged this by the final.

For existing users of remote-bzr that we shipped in 1.8.2, the story is a bit different, though. If this series makes things worse in a way your tests did not reveal, and if such a regression is not reported and/or cannot be fixed by 1.8.3 final, that will mean a real regression in the released version for them.

If that ever happens, that would be the time for us to regret the hasty decision to merge remote-bzr in 1.8.2, justifying that with a "There wasn't anything working for interoperating with bzr, and here is one to do so; anything is better than nothing", and learn from that mistake (it is not an option to say "the 1.8.2 users chose to use contrib/ material that are clearly marked as sub-par quality with their own risk". If we did not ship it in 1.8.2, they did not have to get burned with any regression and could have kept working with bzr a bit longer. "Anything" is not necessarily better than "nothing").

Hopefully, such a regression will not have to happen (for one thing, I would expect that the existing 1.8.2 remote-bzr user base would be very small). Also I somehow have a feeling that it is very unlikely to happen, especially given your report:

 (1) the series converts Emacs history without barfing; and

 (2) you have some confidence in the conversion result after
     inspecting at least a handful of key release points and trees
     and metainformation match between the original and the
     converted history.

So let's go ahead and apply these directly on top of 'master', once we hear from Emacs folks and they are happy with it. I'll queue it on 'pu' so that I do not have to go back to the list archive when it happens.

Thanks.

Felipe Contreras· May 1, 2013, 06:36 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 00/18] remote-bzr: massive changes

On Wed, May 1, 2013 at 12:44 AM, Junio C Hamano <gitster@pobox.com> wrote:

Show 21 quoted lines

> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> After being contacted by the emacs developers and others who are stuck with
>> Bazaar, which at this point seems to be utterly abandoned, I realized the
>> current implementation is too crude.
>> ...
>> That is of course, if pushing actually worked (which in many cases doesn't).
>>
>> In short, our support for real-world projects suck.
>>
>> These patches fix all the issues I encountrered.
>> ...
>> Finally, after all these changes I was finally able to clone the whole emacs
>> repository, all 130685 commits, and 56 branches without running out of memory
>> in my modest laptop.
>
> Yay ;-)
>
> I assume that the trees at a handful of key points (e.g. releases)
> were verified to be identical with the original history and the
> conversion result.

Not really. I don't think the users are that interested in the history being identical at this point, merely that they can use it as a proxy to interact with bazaar repositories.

People have found discrepancies, so I assume they have compared at least the tip of the branches, and found them. This probably means that the history is correct, since bazaar deals with changesets (git is one of the few DSCMs that don't).

Also, there's further news on this, pushes seem to work correctly to the emacs' repo[1].

Show 36 quoted lines

>> I'll ask the emacs developers to give them a try, and let's see
>> how it goes.
>
> Yeah, that's the least we can do for both existing and future users.
>
> Generally speaking, post -rc0 is too late for "if no issues are
> found", simply because no existing user has enough time to find
> corner case regressions in her work using the new software (I do
> not expect a trivial bug that can be uncovered in a few weeks of use
> would remain in a version that has successfully converted the Emacs
> history; but real world users always have different needs than what
> we anticipate).
>
> I however am finding myself moderately receptive to this series.
> That is primarily because this series touches only two files that
> are totally isolated from the rest of the system.  Even if they did
> not work at all, there is no risk for the remainder of Git.  Nobody
> other than existing users of remote-bzr will even notice if we
> merged this by the final.
>
> For existing users of remote-bzr that we shipped in 1.8.2, the story
> is a bit different, though.  If this series makes things worse in a
> way your tests did not reveal, and if such a regression is not
> reported and/or cannot be fixed by 1.8.3 final, that will mean a
> real regression in the released version for them.
>
> If that ever happens, that would be the time for us to regret the
> hasty decision to merge remote-bzr in 1.8.2, justifying that with a
> "There wasn't anything working for interoperating with bzr, and here
> is one to do so; anything is better than nothing", and learn from
> that mistake (it is not an option to say "the 1.8.2 users chose to
> use contrib/ material that are clearly marked as sub-par quality
> with their own risk".  If we did not ship it in 1.8.2, they did not
> have to get burned with any regression and could have kept working
> with bzr a bit longer.  "Anything" is not necessarily better than
> "nothing").

Fortunately there seem to be at least some users that find what is in 1.8.2 working to some extent, not in all the repositories, and not all the features, but at least something, which is much better than the alternatives, even the best one has been blocked for years, even when a solution is known[2].

Show 16 quoted lines

> Hopefully, such a regression will not have to happen (for one thing,
> I would expect that the existing 1.8.2 remote-bzr user base would be
> very small).  Also I somehow have a feeling that it is very unlikely
> to happen, especially given your report:
>
>  (1) the series converts Emacs history without barfing; and
>
>  (2) you have some confidence in the conversion result after
>      inspecting at least a handful of key release points and trees
>      and metainformation match between the original and the
>      converted history.
>
> So let's go ahead and apply these directly on top of 'master', once
> we hear from Emacs folks and they are happy with it.  I'll queue it
> on 'pu' so that I do not have to go back to the list archive when it
> happens.

I already heard that everything seems to be working correctly, except one feature, the biggest change, which I screwed up with a one-liner commit. That's why I added a test. Anyway, I've fixed it in my github branch and in this patch series, and I've told them to try the fix.

Let's see.

Cheers.

[1] http://bzr.savannah.gnu.org/lh/emacs/xwidget/revision/101292 [2] https://bugs.launchpad.net/bzr/+bug/541626

-- 
Felipe Contreras

Junio C Hamano· May 1, 2013, 16:39 UTC · re: Felipe Contreras · lore

Re: [PATCH v2 00/18] remote-bzr: massive changes

Felipe Contreras <felipe.contreras@gmail.com> writes:

Show 9 quoted lines

>> So let's go ahead and apply these directly on top of 'master', once
>> we hear from Emacs folks and they are happy with it.  I'll queue it
>> on 'pu' so that I do not have to go back to the list archive when it
>> happens.
>
> I already heard that everything seems to be working correctly, except
> one feature, the biggest change, which I screwed up with a one-liner
> commit. That's why I added a test. Anyway, I've fixed it in my github
> branch and in this patch series, and I've told them to try the fix.

Let us know when they make progress on that front.

If Emacs decides to switch to Git and decides to use this version of remote-bzr for their conversion, or at least a nontrivial group of developers favor to do so, without seeing concrete technical points that say remote-bzr is not yet ready (e.g. "the conversion is still wrong and X, Y and Z needs to be fixed"), that would be a very welcome solid vote of confidence in favor of us going ahead with this.

Thanks.

Felipe Contreras· May 1, 2013, 18:35 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 00/18] remote-bzr: massive changes

On Wed, May 1, 2013 at 11:39 AM, Junio C Hamano <gitster@pobox.com> wrote:

Show 21 quoted lines

> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>>> So let's go ahead and apply these directly on top of 'master', once
>>> we hear from Emacs folks and they are happy with it.  I'll queue it
>>> on 'pu' so that I do not have to go back to the list archive when it
>>> happens.
>>
>> I already heard that everything seems to be working correctly, except
>> one feature, the biggest change, which I screwed up with a one-liner
>> commit. That's why I added a test. Anyway, I've fixed it in my github
>> branch and in this patch series, and I've told them to try the fix.
>
> Let us know when they make progress on that front.
>
> If Emacs decides to switch to Git and decides to use this version of
> remote-bzr for their conversion, or at least a nontrivial group of
> developers favor to do so, without seeing concrete technical points
> that say remote-bzr is not yet ready (e.g. "the conversion is still
> wrong and X, Y and Z needs to be fixed"), that would be a very
> welcome solid vote of confidence in favor of us going ahead with
> this.

Seems unlikely for political reasons (isn't it always for GNU?), since RMS is heavily involved in the decision.

-- 
Felipe Contreras

Junio C Hamano· May 1, 2013, 18:38 UTC · re: Felipe Contreras · lore

Re: [PATCH v2 00/18] remote-bzr: massive changes

Felipe Contreras <felipe.contreras@gmail.com> writes:

Show 25 quoted lines

> On Wed, May 1, 2013 at 11:39 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>
>>>> So let's go ahead and apply these directly on top of 'master', once
>>>> we hear from Emacs folks and they are happy with it.  I'll queue it
>>>> on 'pu' so that I do not have to go back to the list archive when it
>>>> happens.
>>>
>>> I already heard that everything seems to be working correctly, except
>>> one feature, the biggest change, which I screwed up with a one-liner
>>> commit. That's why I added a test. Anyway, I've fixed it in my github
>>> branch and in this patch series, and I've told them to try the fix.
>>
>> Let us know when they make progress on that front.
>>
>> If Emacs decides to switch to Git and decides to use this version of
>> remote-bzr for their conversion, or at least a nontrivial group of
>> developers favor to do so, without seeing concrete technical points
>> that say remote-bzr is not yet ready (e.g. "the conversion is still
>> wrong and X, Y and Z needs to be fixed"), that would be a very
>> welcome solid vote of confidence in favor of us going ahead with
>> this.
>
> Seems unlikely for political reasons (isn't it always for GNU?), since
> RMS is heavily involved in the decision.

I am very aware of that discussion (and the original one when they decided to use bzr). That is exactly why I said "at least ... favor to do so".

Ted Zlatanov· Jan 3, 2014, 20:17 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 00/18] remote-bzr: massive changes

On Wed, 01 May 2013 11:38:47 -0700 Junio C Hamano <gitster@pobox.com> wrote:

JCH> Felipe Contreras <felipe.contreras@gmail.com> writes:

Show 25 quoted lines

>> On Wed, May 1, 2013 at 11:39 AM, Junio C Hamano <gitster@pobox.com> wrote:
>>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>> 
>>>>> So let's go ahead and apply these directly on top of 'master', once
>>>>> we hear from Emacs folks and they are happy with it.  I'll queue it
>>>>> on 'pu' so that I do not have to go back to the list archive when it
>>>>> happens.
>>>> 
>>>> I already heard that everything seems to be working correctly, except
>>>> one feature, the biggest change, which I screwed up with a one-liner
>>>> commit. That's why I added a test. Anyway, I've fixed it in my github
>>>> branch and in this patch series, and I've told them to try the fix.
>>> 
>>> Let us know when they make progress on that front.
>>> 
>>> If Emacs decides to switch to Git and decides to use this version of
>>> remote-bzr for their conversion, or at least a nontrivial group of
>>> developers favor to do so, without seeing concrete technical points
>>> that say remote-bzr is not yet ready (e.g. "the conversion is still
>>> wrong and X, Y and Z needs to be fixed"), that would be a very
>>> welcome solid vote of confidence in favor of us going ahead with
>>> this.
>> 
>> Seems unlikely for political reasons (isn't it always for GNU?), since
>> RMS is heavily involved in the decision.

JCH> I am very aware of that discussion (and the original one when they JCH> decided to use bzr). That is exactly why I said "at least ... favor JCH> to do so".

FYI, in case you're not aware, there's a pretty strong feeling on emacs-devel that the switch to Git will happen and RMS is not opposed.

I don't know if they'll use remote-bzr, though. It's more likely they'll use one of the already-existing mirrors and sync it up, based on the feedback so far. It's a good time to bring remote-bzr up on emacs-devel if you want it to be considered.

HTH Ted