# [PATCH 00/20] remote-bzr: massive changes 29 messages from 2013-04-26 to 2013-04-29. Participants: Felipe Contreras, Junio C Hamano. Thread: https://gitlist.dev/t/33612 ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 00/20] remote-bzr: massive changes Message-ID: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-1-git-send-email-felipe.contreras%40gmail.com ``` Hi, After being contacted by the emacs developers and others who are stuck with Bazaar, which at this point seems to be utterly abandoned, I realized the current implementation is too crude. Bazaar branches are very simplistic, and our support for them is the same; we need to create one remote per branch. This works nicely if you work on small projects with few branches, but doesn't scale. Big projects like emacs take a lot of space, and creating a remote per branch is unrealistic, because each remote will have the whole Bazaar repository copied, wasting space, and time each time a remote (for a bzr branch) is set up. Moreoever, a developer needs to constantly reset the master branch to the commit he/she wants to push before pushing, since the transport-helper infraestructure doesn't support pushing with refspecs (xwidget:master). That is of course, if pushing actually worked (which in many cases doesn't). In short, our support for real-world projects suck. These patches fix all the issues I encountrered. 1) First of all, there are several improvements for pushing. Before, we failed when trying to push a merge, now, even tricky merges work. 2) Secondly, bzr branches are tied to a transport, so they time out if not used for a period of time, and importing/exporting huge chunks of a repository do take some time. So now they are only opened when they are about to be used. 3) Then the big one; now bzr repositories are supported. They are very simple: basically an object store with no notion of branches, so to find the branches we need to traverse a directory (sometimes) remote, to find them. This is how Bazaar does it, ableit very slowly. Naturally, a lot of code had to be changed to support more than one branch. 4) In addition, now remotes share all the objects, so adding a new remote doesn't imply fetchng a bunch of duplicated objects. They are just re-used automagically. 5) Since the bzr objects are shared now, it only makes sense to share the remote-bzr marks, so we don't have to fast-import them again. 6) The code was also reorganized to keep referenced as few objects as possible, since Bazaar seems to be need *a ton* of memory for them. Finally, after all these changes I was finally able to clone the whole emacs repository, all 130685 commits, and 56 branches without running out of memory in my modest laptop. Since the purpose of remote-bzr is to actually be usable for the poor souls stucks in DSCMs that are not git, these changes are a must. I propose they be merged for the next major version of git (v1.8.3) if no issues are found. They changes pass all the tests, and work on various repositories I've tried. I'll ask the emacs developers to give them a try, and let's see how it goes. Cheers. Felipe Contreras (20): remote-bzr: cleanup CustomTree remote-bzr: delay blob fetching until the very end remote-bzr: fix order of locking in CustomTree remote-bzr: always try to update the worktree remote-bzr: add support to push merges remote-bzr: fixes for branch diverge remote-bzr: fix partially pushed merge remote-bzr: use branch variable when appropriate remote-bzr: add support for bzr repos remote-bzr: fix branch names remote-bzr: add support for shared repo remote-bzr: split marks file remote-bzr: improve author sanitazion remote-bzr: add custom method to find branches remote-bzr: add option to specify branches remote-bzr: improve progress reporting remote-bzr: iterate revisions properly remote-bzr: delay peer branch usage remote-bzr: access branches only when needed remote-bzr: export HEAD contrib/remote-helpers/git-remote-bzr | 365 +++++++++++++++++++++++++--------- contrib/remote-helpers/test-bzr.sh | 36 ++++ 2 files changed, 302 insertions(+), 99 deletions(-) -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 01/20] remote-bzr: cleanup CustomTree Message-ID: <1366938488-25425-2-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-2-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` This code was not used at all. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index c19ed0e..7452a57 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -385,9 +385,6 @@ class CustomTree(): def __init__(self, repo, revid, parents, files): global files_cache - self.repo = repo - self.revid = revid - self.parents = parents self.updates = {} def copy_tree(revid): @@ -435,7 +432,7 @@ class CustomTree(): if basename == '': return None fid = bzrlib.generate_ids.gen_file_id(path) - d = add_entry(fid, dirname, 'directory') + add_entry(fid, dirname, 'directory') return fid def add_entry(fid, path, kind, mode = None): @@ -458,7 +455,6 @@ class CustomTree(): (None, executable)) self.files[path] = change[0] changes.append(change) - return change def update_entry(fid, path, kind, mode = None): dirname, basename = os.path.split(path) @@ -480,7 +476,6 @@ class CustomTree(): (None, executable)) self.files[path] = change[0] changes.append(change) - return change def remove_entry(fid, path, kind): dirname, basename = os.path.split(path) @@ -495,7 +490,6 @@ class CustomTree(): (None, None)) del self.files[path] changes.append(change) - return change for fid, f in self.updates.iteritems(): path = f['path'] -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 02/20] remote-bzr: delay blob fetching until the very end Message-ID: <1366938488-25425-3-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-3-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` Might be more efficient, but the real reason to use the marks will be revealed in upcoming patches. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 7452a57..91b5cda 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -506,10 +506,12 @@ class CustomTree(): return changes def get_file_with_stat(self, file_id, path=None): - return (StringIO.StringIO(self.updates[file_id]['data']), None) + mark = self.updates[file_id]['mark'] + return (StringIO.StringIO(blob_marks[mark]), None) def get_symlink_target(self, file_id): - return self.updates[file_id]['data'] + mark = self.updates[file_id]['mark'] + return blob_marks[mark] def c_style_unescape(string): if string[0] == string[-1] == '"': @@ -553,7 +555,7 @@ def parse_commit(parser): if parser.check('M'): t, m, mark_ref, path = line.split(' ', 3) mark = int(mark_ref[1:]) - f = { 'mode' : m, 'data' : blob_marks[mark] } + f = { 'mode' : m, 'mark' : mark } elif parser.check('D'): t, path = line.split(' ') f = { 'deleted' : True } -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 03/20] remote-bzr: fix order of locking in CustomTree Message-ID: <1366938488-25425-4-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-4-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` It doesn't seem to make any difference, but revision_tree() requires a lock. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 91b5cda..b428244 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -389,8 +389,8 @@ class CustomTree(): def copy_tree(revid): files = files_cache[revid] = {} - tree = repo.repository.revision_tree(revid) repo.lock_read() + tree = repo.repository.revision_tree(revid) try: for path, entry in tree.iter_entries_by_dir(): files[path] = entry.file_id -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 04/20] remote-bzr: always try to update the worktree Message-ID: <1366938488-25425-5-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-5-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` And fail properly when we can't. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index b428244..eb91d28 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -641,9 +641,13 @@ def do_export(parser): except bzrlib.errors.DivergedBranches: print "error %s non-fast forward" % ref continue - else: + + try: wt = repo.bzrdir.open_workingtree() wt.update() + except bzrlib.errors.NoWorkingTree: + pass + print "ok %s" % ref print -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 05/20] remote-bzr: add support to push merges Message-ID: <1366938488-25425-6-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-6-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` In order to do that, we need to store the marks of every file, so that they can be fetched when needed. Unfortunately we can't tell bazaar that nothing changed, we need to send the data so that it can figure it out by itself. And since it will be requesting a bunch of information by the file_id, it's better to have a helper dict (rev_files), so that we can fetch it quickly. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 29 ++++++++++++++++++++-------- contrib/remote-helpers/test-bzr.sh | 36 +++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 8 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index eb91d28..6a7f836 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -393,7 +393,7 @@ class CustomTree(): tree = repo.repository.revision_tree(revid) try: for path, entry in tree.iter_entries_by_dir(): - files[path] = entry.file_id + files[path] = [entry.file_id, None] finally: repo.unlock() return files @@ -408,12 +408,18 @@ class CustomTree(): self.base_files = copy_tree(self.base_id) self.files = files_cache[revid] = self.base_files.copy() + self.rev_files = {} + + for path, data in self.files.iteritems(): + fid, mark = data + self.rev_files[fid] = [path, mark] for path, f in files.iteritems(): - fid = self.files.get(path, None) + fid, mark = self.files.get(path, [None, None]) if not fid: fid = bzrlib.generate_ids.gen_file_id(path) f['path'] = path + self.rev_files[fid] = [path, mark] self.updates[fid] = f def last_revision(self): @@ -423,10 +429,10 @@ class CustomTree(): changes = [] def get_parent(dirname, basename): - parent_fid = self.base_files.get(dirname, None) + parent_fid, mark = self.base_files.get(dirname, [None, None]) if parent_fid: return parent_fid - parent_fid = self.files.get(dirname, None) + parent_fid, mark = self.files.get(dirname, [None, None]) if parent_fid: return parent_fid if basename == '': @@ -453,7 +459,7 @@ class CustomTree(): (None, basename), (None, kind), (None, executable)) - self.files[path] = change[0] + self.files[path] = [change[0], None] changes.append(change) def update_entry(fid, path, kind, mode = None): @@ -474,7 +480,7 @@ class CustomTree(): (None, basename), (None, kind), (None, executable)) - self.files[path] = change[0] + self.files[path] = [change[0], None] changes.append(change) def remove_entry(fid, path, kind): @@ -503,16 +509,23 @@ class CustomTree(): else: add_entry(fid, path, 'file', f['mode']) + self.files[path][1] = f['mark'] + self.rev_files[fid][1] = f['mark'] + return changes def get_file_with_stat(self, file_id, path=None): - mark = self.updates[file_id]['mark'] + path, mark = self.rev_files[file_id] return (StringIO.StringIO(blob_marks[mark]), None) def get_symlink_target(self, file_id): - mark = self.updates[file_id]['mark'] + path, mark = self.rev_files[file_id] return blob_marks[mark] + def id2path(self, file_id): + path, mark = self.rev_files[file_id] + return path + def c_style_unescape(string): if string[0] == string[-1] == '"': return string.decode('string-escape')[1:-1] diff --git a/contrib/remote-helpers/test-bzr.sh b/contrib/remote-helpers/test-bzr.sh index 34666e1..eca3476 100755 --- a/contrib/remote-helpers/test-bzr.sh +++ b/contrib/remote-helpers/test-bzr.sh @@ -228,4 +228,40 @@ test_expect_success 'push utf-8 filenames' ' test_cmp expected actual ' +test_expect_success 'pushing a merge' ' + mkdir -p tmp && cd tmp && + test_when_finished "cd .. && rm -rf tmp" && + + ( + bzr init bzrrepo && + cd bzrrepo && + echo one > content && + bzr add content && + bzr commit -m one + ) && + + git clone "bzr::$PWD/bzrrepo" gitrepo && + + ( + cd bzrrepo && + echo two > content && + bzr commit -m two + ) && + + ( + cd gitrepo && + echo three > content && + git commit -a -m three && + git fetch && + git merge origin/master || true && + echo three > content && + git commit -a --no-edit && + git push + ) && + + echo three > expected && + cat bzrrepo/content > actual && + test_cmp expected actual +' + test_done -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 06/20] remote-bzr: fixes for branch diverge Message-ID: <1366938488-25425-7-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-7-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` If the branches diverge we want to reset the pointer to where the remote actually is. Since we can access remote branches just as easily as local ones, let's do so. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 6a7f836..bf254a0 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -354,6 +354,7 @@ def do_import(parser): if os.path.exists(path): print "feature import-marks=%s" % path print "feature export-marks=%s" % path + print "feature force" sys.stdout.flush() while parser.check('import'): @@ -716,7 +717,12 @@ def get_repo(url, alias): # pull d = bzrlib.bzrdir.BzrDir.open(clone_path) branch = d.open_branch() - result = branch.pull(remote_branch, [], None, False) + try: + result = branch.pull(remote_branch, [], None, False) + except bzrlib.errors.DivergedBranches: + # use remote branch for now + peer = None + return remote_branch else: # clone d = origin.sprout(clone_path, None, -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 07/20] remote-bzr: fix partially pushed merge Message-ID: <1366938488-25425-8-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-8-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` If part of the merge was already pushed, we don't have the blob_marks available, however, the commits are already stored in bazaar, so we can use the revision_tree to fetch the contents. We want to do this only when there's no other option. There's no easy way to test this. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index bf254a0..fdead31 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -387,6 +387,7 @@ class CustomTree(): global files_cache self.updates = {} + self.branch = repo def copy_tree(revid): files = files_cache[revid] = {} @@ -515,13 +516,21 @@ class CustomTree(): return changes - def get_file_with_stat(self, file_id, path=None): + def get_content(self, file_id): path, mark = self.rev_files[file_id] - return (StringIO.StringIO(blob_marks[mark]), None) + if mark: + return blob_marks[mark] + + # last resort + tree = self.branch.repository.revision_tree(self.base_id) + return tree.get_file_text(file_id) + + def get_file_with_stat(self, file_id, path=None): + content = self.get_content(file_id) + return (StringIO.StringIO(content), None) def get_symlink_target(self, file_id): - path, mark = self.rev_files[file_id] - return blob_marks[mark] + return self.get_content(file_id) def id2path(self, file_id): path, mark = self.rev_files[file_id] -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 08/20] remote-bzr: use branch variable when appropriate Message-ID: <1366938488-25425-9-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-9-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` There should be no functional changes. Basically we want to reserve the 'repo' variable. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index fdead31..0c67236 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -257,7 +257,8 @@ def export_branch(branch, name): tip = marks.get_tip(name) repo = branch.repository - repo.lock_read() + + branch.lock_read() revs = branch.iter_merge_sorted_revisions(None, tip, 'exclude', 'forward') count = 0 @@ -325,7 +326,7 @@ def export_branch(branch, name): print "progress revision %s (%d/%d)" % (revid, count, len(revs)) print "#############################################################" - repo.unlock() + branch.unlock() revid = branch.last_revision() @@ -383,21 +384,21 @@ def parse_blob(parser): class CustomTree(): - def __init__(self, repo, revid, parents, files): + def __init__(self, branch, revid, parents, files): global files_cache self.updates = {} - self.branch = repo + self.branch = branch def copy_tree(revid): files = files_cache[revid] = {} - repo.lock_read() - tree = repo.repository.revision_tree(revid) + branch.lock_read() + tree = branch.repository.revision_tree(revid) try: for path, entry in tree.iter_entries_by_dir(): files[path] = [entry.file_id, None] finally: - repo.unlock() + branch.unlock() return files if len(parents) == 0: @@ -587,20 +588,20 @@ def parse_commit(parser): path = c_style_unescape(path).decode('utf-8') files[path] = f - repo = parser.repo + branch = parser.repo committer, date, tz = committer parents = [str(mark_to_rev(p)) for p in parents] revid = bzrlib.generate_ids.gen_revision_id(committer, date) props = {} - props['branch-nick'] = repo.nick + props['branch-nick'] = branch.nick - mtree = CustomTree(repo, revid, parents, files) + mtree = CustomTree(branch, revid, parents, files) changes = mtree.iter_changes() - repo.lock_write() + branch.lock_write() try: - builder = repo.get_commit_builder(parents, None, date, tz, committer, props, revid) + builder = branch.get_commit_builder(parents, None, date, tz, committer, props, revid) try: list(builder.record_iter_changes(mtree, mtree.last_revision(), changes)) builder.finish_inventory() @@ -609,7 +610,7 @@ def parse_commit(parser): builder.abort() raise finally: - repo.unlock() + branch.unlock() parsed_refs[ref] = revid marks.new_mark(revid, commit_mark) @@ -653,20 +654,20 @@ def do_export(parser): else: die('unhandled export command: %s' % line) - repo = parser.repo + branch = parser.repo for ref, revid in parsed_refs.iteritems(): if ref == 'refs/heads/master': - repo.generate_revision_history(revid, marks.get_tip('master')) + branch.generate_revision_history(revid, marks.get_tip('master')) if peer: try: - repo.push(peer, stop_revision=revid) + branch.push(peer, stop_revision=revid) except bzrlib.errors.DivergedBranches: print "error %s non-fast forward" % ref continue try: - wt = repo.bzrdir.open_workingtree() + wt = branch.bzrdir.open_workingtree() wt.update() except bzrlib.errors.NoWorkingTree: pass -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 09/20] remote-bzr: add support for bzr repos Message-ID: <1366938488-25425-10-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-10-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` In bazaar, a repository can contain multiple branches, and previously we were supporting only one branch at a time. Now we fetch them all. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 158 ++++++++++++++++++++++------------ 1 file changed, 102 insertions(+), 56 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 0c67236..3b33061 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -27,6 +27,7 @@ import bzrlib.generate_ids import bzrlib.transport import bzrlib.errors import bzrlib.ui +import bzrlib.urlutils import sys import os @@ -250,12 +251,13 @@ def export_files(tree, files): return final -def export_branch(branch, name): +def export_branch(repo, name): global prefix ref = '%s/heads/%s' % (prefix, name) tip = marks.get_tip(name) + branch = branches[name] repo = branch.repository branch.lock_read() @@ -323,7 +325,7 @@ def export_branch(branch, name): count += 1 if (count % 100 == 0): - print "progress revision %s (%d/%d)" % (revid, count, len(revs)) + print "progress revision %s '%s' (%d/%d)" % (revid, name, count, len(revs)) print "#############################################################" branch.unlock() @@ -348,7 +350,7 @@ def export_tag(repo, name): def do_import(parser): global dirname - branch = parser.repo + repo = parser.repo path = os.path.join(dirname, 'marks-git') print "feature done" @@ -362,10 +364,10 @@ def do_import(parser): ref = parser[1] if ref.startswith('refs/heads/'): name = ref[len('refs/heads/'):] - export_branch(branch, name) + export_branch(repo, name) if ref.startswith('refs/tags/'): name = ref[len('refs/tags/'):] - export_tag(branch, name) + export_tag(repo, name) parser.next() print 'done' @@ -551,8 +553,11 @@ def parse_commit(parser): ref = parser[1] parser.next() - if ref != 'refs/heads/master': - die("bzr doesn't support multiple branches; use 'master'") + if ref.startswith('refs/heads/'): + name = ref[len('refs/heads/'):] + branch = branches[name] + else: + die('unknown ref') commit_mark = parser.get_mark() parser.next() @@ -588,8 +593,6 @@ def parse_commit(parser): path = c_style_unescape(path).decode('utf-8') files[path] = f - branch = parser.repo - committer, date, tz = committer parents = [str(mark_to_rev(p)) for p in parents] revid = bzrlib.generate_ids.gen_revision_id(committer, date) @@ -621,9 +624,6 @@ def parse_reset(parser): ref = parser[1] parser.next() - if ref != 'refs/heads/master': - die("bzr doesn't support multiple branches; use 'master'") - # ugh if parser.check('commit'): parse_commit(parser) @@ -636,7 +636,7 @@ def parse_reset(parser): parsed_refs[ref] = mark_to_rev(from_mark) def do_export(parser): - global parsed_refs, dirname, peer + global parsed_refs, dirname parser.next() @@ -654,23 +654,24 @@ def do_export(parser): else: die('unhandled export command: %s' % line) - branch = parser.repo - for ref, revid in parsed_refs.iteritems(): - if ref == 'refs/heads/master': - branch.generate_revision_history(revid, marks.get_tip('master')) - if peer: - try: - branch.push(peer, stop_revision=revid) - except bzrlib.errors.DivergedBranches: - print "error %s non-fast forward" % ref - continue + name = ref[len('refs/heads/'):] + branch = branches[name] + branch.generate_revision_history(revid, marks.get_tip(name)) + if name in peers: + peer = peers[name] try: - wt = branch.bzrdir.open_workingtree() - wt.update() - except bzrlib.errors.NoWorkingTree: - pass + peer.bzrdir.push_branch(branch, revision_id=revid) + except bzrlib.errors.DivergedBranches: + print "error %s non-fast forward" % ref + continue + + try: + wt = branch.bzrdir.open_workingtree() + wt.update() + except bzrlib.errors.NoWorkingTree: + pass print "ok %s" % ref @@ -697,9 +698,15 @@ def ref_is_valid(name): def do_list(parser): global tags - print "? refs/heads/%s" % 'master' - branch = parser.repo + master_branch = None + + for name in branches: + if not master_branch: + master_branch = name + print "? refs/heads/%s" % name + + branch = branches[master_branch] branch.lock_read() for tag, revid in branch.tags.get_tag_dict().items(): try: @@ -711,41 +718,77 @@ def do_list(parser): print "? refs/tags/%s" % tag tags[tag] = revid branch.unlock() - print "@refs/heads/%s HEAD" % 'master' + + print "@refs/heads/%s HEAD" % master_branch print +def get_remote_branch(origin, remote_branch, name): + global dirname, peers + + branch_path = os.path.join(dirname, 'clone', name) + if os.path.exists(branch_path): + # pull + d = bzrlib.bzrdir.BzrDir.open(branch_path) + branch = d.open_branch() + try: + branch.pull(remote_branch, [], None, False) + except bzrlib.errors.DivergedBranches: + # use remote branch for now + return remote_branch + else: + # clone + d = origin.sprout(branch_path, None, + hardlink=True, create_tree_if_local=False, + force_new_repo=False, + source_branch=remote_branch) + branch = d.open_branch() + + return branch + def get_repo(url, alias): - global dirname, peer + global dirname, peer, branches + normal_url = bzrlib.urlutils.normalize_url(url) origin = bzrlib.bzrdir.BzrDir.open(url) - branch = origin.open_branch() - - if not isinstance(origin.transport, bzrlib.transport.local.LocalTransport): - clone_path = os.path.join(dirname, 'clone') - remote_branch = branch - if os.path.exists(clone_path): - # pull - d = bzrlib.bzrdir.BzrDir.open(clone_path) - branch = d.open_branch() - try: - result = branch.pull(remote_branch, [], None, False) - except bzrlib.errors.DivergedBranches: - # use remote branch for now - peer = None - return remote_branch + is_local = isinstance(origin.transport, bzrlib.transport.local.LocalTransport) + + clone_path = os.path.join(dirname, 'clone') + + try: + repo = origin.open_repository() + except bzrlib.errors.NoRepositoryPresent: + # branch + + name = 'master' + branch = origin.open_branch() + + if not is_local: + if not os.path.exists(clone_path): + os.mkdir(clone_path) + peers[name] = branch + branches[name] = get_remote_branch(origin, branch, name) else: - # clone - d = origin.sprout(clone_path, None, - hardlink=True, create_tree_if_local=False, - source_branch=remote_branch) - branch = d.open_branch() - branch.bind(remote_branch) - - peer = remote_branch + branches[name] = branch + + return branch.repository else: - peer = None + # repository - return branch + if not is_local and not os.path.exists(clone_path): + clonedir = bzrlib.bzrdir.BzrDir.create(clone_path) + + for branch in repo.find_branches(): + + name = repo.user_transport.relpath(branch.base) + name = name if name != '' else 'master' + + if not is_local: + peers[name] = branch + branches[name] = get_remote_branch(origin, branch, name) + else: + branches[name] = branch + + return repo def fix_path(alias, orig_url): url = urlparse.urlparse(orig_url, 'file') @@ -762,6 +805,7 @@ def main(args): global parsed_refs global files_cache global is_tmp + global branches, peers alias = args[1] url = args[2] @@ -772,6 +816,8 @@ def main(args): parsed_refs = {} files_cache = {} marks = None + branches = {} + peers = {} if alias[5:] == url: is_tmp = True -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 10/20] remote-bzr: fix branch names Message-ID: <1366938488-25425-11-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-11-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` When branches have '/' in their name (aka. sub-branches), bazaar seems to choke while creating the new directory. Also, git cannot have both 'foo' and 'foo/bar'. So let's replace slashes with a plus sign. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 1 + 1 file changed, 1 insertion(+) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 3b33061..38cec07 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -781,6 +781,7 @@ def get_repo(url, alias): name = repo.user_transport.relpath(branch.base) name = name if name != '' else 'master' + name = name.replace('/', '+') if not is_local: peers[name] = branch -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:07 Subject: [PATCH 11/20] remote-bzr: add support for shared repo Message-ID: <1366938488-25425-12-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-12-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` This way all the remotes share the same data, so adding multiple remotes, or renaming them doesn't create extra overhead. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 38cec07..9fe830e 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -752,7 +752,20 @@ def get_repo(url, alias): origin = bzrlib.bzrdir.BzrDir.open(url) is_local = isinstance(origin.transport, bzrlib.transport.local.LocalTransport) - clone_path = os.path.join(dirname, 'clone') + shared_path = os.path.join(gitdir, 'bzr') + try: + shared_dir = bzrlib.bzrdir.BzrDir.open(shared_path) + except bzrlib.errors.NotBranchError: + shared_dir = bzrlib.bzrdir.BzrDir.create(shared_path) + try: + shared_repo = shared_dir.open_repository() + except bzrlib.errors.NoRepositoryPresent: + shared_repo = shared_dir.create_repository(shared=True) + + if not is_local: + clone_path = os.path.join(dirname, 'clone') + if not os.path.exists(clone_path): + os.mkdir(clone_path) try: repo = origin.open_repository() @@ -763,8 +776,6 @@ def get_repo(url, alias): branch = origin.open_branch() if not is_local: - if not os.path.exists(clone_path): - os.mkdir(clone_path) peers[name] = branch branches[name] = get_remote_branch(origin, branch, name) else: @@ -774,9 +785,6 @@ def get_repo(url, alias): else: # repository - if not is_local and not os.path.exists(clone_path): - clonedir = bzrlib.bzrdir.BzrDir.create(clone_path) - for branch in repo.find_branches(): name = repo.user_transport.relpath(branch.base) @@ -800,7 +808,7 @@ def fix_path(alias, orig_url): subprocess.call(cmd) def main(args): - global marks, prefix, dirname + global marks, prefix, gitdir, dirname global tags, filenodes global blob_marks global parsed_refs -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 12/20] remote-bzr: split marks file Message-ID: <1366938488-25425-13-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-13-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` This way all the remotes can share the same git objects, and the same marks. The information we want to store per remote is very small. The code transparently converts from one organization of marks, to the other. It's rather smooth and there shouldn't be any issues. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 58 ++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 15 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 9fe830e..fcd8e41 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -52,22 +52,42 @@ def gittz(tz): class Marks: - def __init__(self, path): - self.path = path + def __init__(self, gitdir, dirname): + self.marks_path = os.path.join(gitdir, 'bzr', '.marks-bzr') + self.remote_path = os.path.join(dirname, '.marks-bzr') + old_path = os.path.join(dirname, 'marks-int') + self.tips = {} self.marks = {} self.rev_marks = {} self.last_mark = 0 - self.load() + + if not os.path.exists(old_path): + self.load() + else: + self.old_load(old_path) + os.remove(old_path) def load(self): - if not os.path.exists(self.path): - return - tmp = json.load(open(self.path)) + if os.path.exists(self.marks_path): + tmp = json.load(open(self.marks_path)) + self.marks = tmp['marks'] + self.last_mark = tmp['last-mark'] + + for rev, mark in self.marks.iteritems(): + self.rev_marks[mark] = rev + + if os.path.exists(self.remote_path): + tmp = json.load(open(self.remote_path)) + self.tips = tmp['tips'] + + def old_load(self, path): + + tmp = json.load(open(path)) self.tips = tmp['tips'] - self.marks = tmp['marks'] - self.last_mark = tmp['last-mark'] + self.marks = tmp['marks'] # TODO remove + self.last_mark = tmp['last-mark'] # TODO remove for rev, mark in self.marks.iteritems(): self.rev_marks[mark] = rev @@ -76,7 +96,10 @@ class Marks: return { 'tips': self.tips, 'marks': self.marks, 'last-mark' : self.last_mark } def store(self): - json.dump(self.dict(), open(self.path, 'w')) + d = { 'marks': self.marks, 'last-mark' : self.last_mark } + json.dump(d, open(self.marks_path, 'w')) + d = { 'tips': self.tips } + json.dump(d, open(self.remote_path, 'w')) def __str__(self): return str(self.dict()) @@ -348,10 +371,10 @@ def export_tag(repo, name): print def do_import(parser): - global dirname + global gitdir repo = parser.repo - path = os.path.join(dirname, 'marks-git') + path = os.path.join(gitdir, 'bzr', '.marks-git') print "feature done" if os.path.exists(path): @@ -678,14 +701,14 @@ def do_export(parser): print def do_capabilities(parser): - global dirname + global gitdir print "import" print "export" print "refspec refs/heads/*:%s/heads/*" % prefix print "refspec refs/tags/*:%s/tags/*" % prefix - path = os.path.join(dirname, 'marks-git') + path = os.path.join(gitdir, 'bzr', '.marks-git') if os.path.exists(path): print "*import-marks %s" % path @@ -848,8 +871,13 @@ def main(args): repo = get_repo(url, alias) - marks_path = os.path.join(dirname, 'marks-int') - marks = Marks(marks_path) + marks = Marks(gitdir, dirname) + + # move old marks + old_gitmarks = os.path.join(dirname, 'marks-git') + new_gitmarks = os.path.join(gitdir, 'bzr', '.marks-git') + if os.path.exists(old_gitmarks): + os.rename(old_gitmarks, new_gitmarks) parser = Parser(repo) for line in parser: -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 13/20] remote-bzr: improve author sanitazion Message-ID: <1366938488-25425-14-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-14-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` So that we don't end up with '', and also synchronize it with the one from remote-hg. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index fcd8e41..5c25b06 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -38,6 +38,7 @@ import atexit, shutil, hashlib, urlparse, subprocess NAME_RE = re.compile('^([^<>]+)') AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$') +EMAIL_RE = re.compile('^([^<>]+[^ \\\t<>])?\\b(?:[ \\t<>]*?)\\b([^ \\t<>]+@[^ \\t<>]+)') RAW_AUTHOR_RE = re.compile('^(\w+) (.+)? <(.*)> (\d+) ([+-]\d+)') def die(msg, *args): @@ -198,9 +199,19 @@ def fixup_user(user): name = m.group(1) mail = m.group(2).strip() else: - m = NAME_RE.match(user) + m = EMAIL_RE.match(user) if m: - name = m.group(1).strip() + name = m.group(1) + mail = m.group(2) + else: + m = NAME_RE.match(user) + if m: + name = m.group(1).strip() + + if not name: + name = 'unknown' + if not mail: + mail = 'Unknown' return '%s <%s>' % (name, mail) -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 14/20] remote-bzr: add custom method to find branches Message-ID: <1366938488-25425-15-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-15-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` The official method is incredibly inefficient and slow. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 5c25b06..5155b93 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -779,6 +779,25 @@ def get_remote_branch(origin, remote_branch, name): return branch +def find_branches(repo): + transport = repo.user_transport + + for fn in transport.iter_files_recursive(): + if not fn.endswith('.bzr/branch-format'): + continue + + name = subdir = fn[:-len('/.bzr/branch-format')] + name = name if name != '' else 'master' + name = name.replace('/', '+') + + try: + cur = transport.clone(subdir) + branch = bzrlib.branch.Branch.open_from_transport(cur) + except bzrlib.errors.NotBranchError: + continue + else: + yield name, branch + def get_repo(url, alias): global dirname, peer, branches @@ -819,11 +838,7 @@ def get_repo(url, alias): else: # repository - for branch in repo.find_branches(): - - name = repo.user_transport.relpath(branch.base) - name = name if name != '' else 'master' - name = name.replace('/', '+') + for name, branch in find_branches(repo): if not is_local: peers[name] = branch -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 15/20] remote-bzr: add option to specify branches Message-ID: <1366938488-25425-16-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-16-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` We might not want all the branches. And branch handling in bazaar is rather tricky, so it's safer to simply specify them. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 5155b93..06d9d24 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -13,6 +13,9 @@ # or # % git clone bzr::lp:myrepo # +# If you want to specify which branches you want track (per repo): +# git config remote-bzr.branches 'trunk, devel, test' +# import sys @@ -51,6 +54,12 @@ def warn(msg, *args): def gittz(tz): return '%+03d%02d' % (tz / 3600, tz % 3600 / 60) +def get_config(config): + cmd = ['git', 'config', '--get', config] + process = subprocess.Popen(cmd, stdout=subprocess.PIPE) + output, _ = process.communicate() + return output + class Marks: def __init__(self, gitdir, dirname): @@ -779,7 +788,7 @@ def get_remote_branch(origin, remote_branch, name): return branch -def find_branches(repo): +def find_branches(repo, wanted): transport = repo.user_transport for fn in transport.iter_files_recursive(): @@ -790,6 +799,9 @@ def find_branches(repo): name = name if name != '' else 'master' name = name.replace('/', '+') + if wanted and not name in wanted: + continue + try: cur = transport.clone(subdir) branch = bzrlib.branch.Branch.open_from_transport(cur) @@ -838,7 +850,11 @@ def get_repo(url, alias): else: # repository - for name, branch in find_branches(repo): + wanted = get_config('remote-bzr.branches').rstrip().split(', ') + # stupid python + wanted = [e for e in wanted if e] + + for name, branch in find_branches(repo, wanted): if not is_local: peers[name] = branch -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 16/20] remote-bzr: improve progress reporting Message-ID: <1366938488-25425-17-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-17-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` No need to manually count the revisions, and also, this would help to iterate more properly. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 06d9d24..7d3052a 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -305,13 +305,16 @@ def export_branch(repo, name): branch.lock_read() revs = branch.iter_merge_sorted_revisions(None, tip, 'exclude', 'forward') - count = 0 + tip_revno = branch.revision_id_to_revno(tip) + last_revno, _ = branch.last_revision_info() + total = last_revno - tip_revno - revs = [revid for revid, _, _, _ in revs if not marks.is_marked(revid)] + revs = [revid, seq for revid, _, seq, _ in revs if not marks.is_marked(revid)] - for revid in revs: + for revid, seq in revs: rev = repo.get_revision(revid) + revno = seq[0] parents = rev.parent_ids time = rev.timestamp @@ -366,10 +369,13 @@ def export_branch(repo, name): print "M %s :%u %s" % f print - count += 1 - if (count % 100 == 0): - print "progress revision %s '%s' (%d/%d)" % (revid, name, count, len(revs)) - print "#############################################################" + if len(seq) > 1: + # let's skip branch revisions from the progress report + continue + + progress = (revno - tip_revno) + if (progress % 100 == 0): + print "progress revision %d '%s' (%d/%d)" % (revno, name, progress, total) branch.unlock() -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 17/20] remote-bzr: iterate revisions properly Message-ID: <1366938488-25425-18-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-18-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` This way we don't need to store the list of all the revisions, which doesn't seem to be very memory efficient with bazaar's design, for whatever reason. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 7d3052a..5f1b9c0 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -309,9 +309,10 @@ def export_branch(repo, name): last_revno, _ = branch.last_revision_info() total = last_revno - tip_revno - revs = [revid, seq for revid, _, seq, _ in revs if not marks.is_marked(revid)] + for revid, _, seq, _ in revs: - for revid, seq in revs: + if marks.is_marked(revid): + continue rev = repo.get_revision(revid) revno = seq[0] -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 18/20] remote-bzr: delay peer branch usage Message-ID: <1366938488-25425-19-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-19-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` So it doesn't time out. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 5f1b9c0..7517645 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -710,7 +710,7 @@ def do_export(parser): branch.generate_revision_history(revid, marks.get_tip(name)) if name in peers: - peer = peers[name] + peer = bzrlib.branch.Branch.open(peers[name]) try: peer.bzrdir.push_branch(branch, revision_id=revid) except bzrlib.errors.DivergedBranches: @@ -848,7 +848,7 @@ def get_repo(url, alias): branch = origin.open_branch() if not is_local: - peers[name] = branch + peers[name] = branch.base branches[name] = get_remote_branch(origin, branch, name) else: branches[name] = branch @@ -864,7 +864,7 @@ def get_repo(url, alias): for name, branch in find_branches(repo, wanted): if not is_local: - peers[name] = branch + peers[name] = branch.base branches[name] = get_remote_branch(origin, branch, name) else: branches[name] = branch -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 19/20] remote-bzr: access branches only when needed Message-ID: <1366938488-25425-20-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-20-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` Bazaar doesn't seem to be tested for multiple usage of branches, so resources seem to be leaked all over. Let's try to minimize this by accessing the Branch objects only when needed. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 7517645..3c59a18 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -300,7 +300,7 @@ def export_branch(repo, name): ref = '%s/heads/%s' % (prefix, name) tip = marks.get_tip(name) - branch = branches[name] + branch = bzrlib.branch.Branch.open(branches[name]) repo = branch.repository branch.lock_read() @@ -605,7 +605,7 @@ def parse_commit(parser): if ref.startswith('refs/heads/'): name = ref[len('refs/heads/'):] - branch = branches[name] + branch = bzrlib.branch.Branch.open(branches[name]) else: die('unknown ref') @@ -706,7 +706,7 @@ def do_export(parser): for ref, revid in parsed_refs.iteritems(): name = ref[len('refs/heads/'):] - branch = branches[name] + branch = bzrlib.branch.Branch.open(branches[name]) branch.generate_revision_history(revid, marks.get_tip(name)) if name in peers: @@ -756,7 +756,7 @@ def do_list(parser): master_branch = name print "? refs/heads/%s" % name - branch = branches[master_branch] + branch = bzrlib.branch.Branch.open(branches[master_branch]) branch.lock_read() for tag, revid in branch.tags.get_tag_dict().items(): try: @@ -845,13 +845,15 @@ def get_repo(url, alias): # branch name = 'master' - branch = origin.open_branch() + remote_branch = origin.open_branch() if not is_local: - peers[name] = branch.base - branches[name] = get_remote_branch(origin, branch, name) + peers[name] = remote_branch.base + branch = get_remote_branch(origin, remote_branch, name) else: - branches[name] = branch + branch = remote_branch + + branches[name] = branch.base return branch.repository else: @@ -861,13 +863,15 @@ def get_repo(url, alias): # stupid python wanted = [e for e in wanted if e] - for name, branch in find_branches(repo, wanted): + for name, remote_branch in find_branches(repo, wanted): if not is_local: - peers[name] = branch.base - branches[name] = get_remote_branch(origin, branch, name) + peers[name] = remote_branch.base + branch = get_remote_branch(origin, remote_branch, name) else: - branches[name] = branch + branch = remote_branch + + branches[name] = branch.base return repo -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 01:08 Subject: [PATCH 20/20] remote-bzr: export HEAD Message-ID: <1366938488-25425-21-git-send-email-felipe.contreras@gmail.com> URL: https://gitlist.dev/e/1366938488-25425-21-git-send-email-felipe.contreras%40gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` It probably doesn't make any difference, since we usually fetch along the 'master' branch, and when we don't probably the current transport-helper code doesn't work correctly. But if it did work correctly, this might be useful. Signed-off-by: Felipe Contreras --- contrib/remote-helpers/git-remote-bzr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr index 3c59a18..41c26a7 100755 --- a/contrib/remote-helpers/git-remote-bzr +++ b/contrib/remote-helpers/git-remote-bzr @@ -412,6 +412,8 @@ def do_import(parser): while parser.check('import'): ref = parser[1] + if ref == 'HEAD': + export_branch(repo, 'master') if ref.startswith('refs/heads/'): name = ref[len('refs/heads/'):] export_branch(repo, name) -- 1.8.2.1.884.g3532a8d ``` ## Felipe Contreras, 2013-04-26 02:35 Subject: Re: [PATCH 00/20] remote-bzr: massive changes Message-ID: URL: https://gitlist.dev/e/CAMP44s0XHXbX5brcxv-NROJ%3DPSSzi3oEzQv81%3Df3-1-zcypQuQ%40mail.gmail.com In-Reply-To: <1366938488-25425-1-git-send-email-felipe.contreras@gmail.com> ``` On Thu, Apr 25, 2013 at 8:07 PM, Felipe Contreras wrote: I forgot to mention; these apply on top of the previous 'fixes and cleanups'. > Felipe Contreras (20): > remote-bzr: cleanup CustomTree > remote-bzr: delay blob fetching until the very end > remote-bzr: fix order of locking in CustomTree > remote-bzr: always try to update the worktree > remote-bzr: add support to push merges > remote-bzr: fixes for branch diverge > remote-bzr: fix partially pushed merge > remote-bzr: use branch variable when appropriate > remote-bzr: add support for bzr repos > remote-bzr: fix branch names > remote-bzr: add support for shared repo > remote-bzr: split marks file > remote-bzr: improve author sanitazion > remote-bzr: add custom method to find branches > remote-bzr: add option to specify branches > remote-bzr: improve progress reporting > remote-bzr: iterate revisions properly > remote-bzr: delay peer branch usage > remote-bzr: access branches only when needed > remote-bzr: export HEAD > > contrib/remote-helpers/git-remote-bzr | 365 +++++++++++++++++++++++++--------- > contrib/remote-helpers/test-bzr.sh | 36 ++++ > 2 files changed, 302 insertions(+), 99 deletions(-) -- Felipe Contreras ``` ## Junio C Hamano, 2013-04-26 16:51 Subject: Re: [PATCH 00/20] remote-bzr: massive changes Message-ID: <7vy5c5xmka.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vy5c5xmka.fsf%40alter.siamese.dyndns.org In-Reply-To: ``` Felipe Contreras writes: > On Thu, Apr 25, 2013 at 8:07 PM, Felipe Contreras > wrote: > > I forgot to mention; these apply on top of the previous 'fixes and cleanups'. Then I should probably wait for v2 6/9 of that series is rerolled. There could be ones that you may want to reroll other than 6/9 in that, I didn't have time to look at the backlog. I'll be whipping the -rc0 into shape today and it is unlikely I'll have time to look at this or the previous one today. Thanks. ``` ## Felipe Contreras, 2013-04-26 21:21 Subject: Re: [PATCH 00/20] remote-bzr: massive changes Message-ID: URL: https://gitlist.dev/e/CAMP44s1ZCD4qNH23Zv-Lb3aq9qGeB0LAkr8mOBwusMKPO9GVmg%40mail.gmail.com In-Reply-To: <7vy5c5xmka.fsf@alter.siamese.dyndns.org> ``` On Fri, Apr 26, 2013 at 11:51 AM, Junio C Hamano wrote: > Felipe Contreras writes: > >> On Thu, Apr 25, 2013 at 8:07 PM, Felipe Contreras >> wrote: >> >> I forgot to mention; these apply on top of the previous 'fixes and cleanups'. > > Then I should probably wait for v2 6/9 of that series is rerolled. Done. > There could be ones that you may want to reroll other than 6/9 in > that, I didn't have time to look at the backlog. I'll be whipping > the -rc0 into shape today and it is unlikely I'll have time to look > at this or the previous one today. I don't see any need to reroll more. All the concerns of v1 and v2 have been addressed in one way or the other. -- Felipe Contreras ``` ## Felipe Contreras, 2013-04-26 23:36 Subject: Re: [PATCH 12/20] remote-bzr: split marks file Message-ID: URL: https://gitlist.dev/e/CAMP44s2dzV428%2B_ZENg_Z5n%3Drnvqz1ZNAxnjsqCtLSJK-9PZZg%40mail.gmail.com In-Reply-To: <1366938488-25425-13-git-send-email-felipe.contreras@gmail.com> ``` On Thu, Apr 25, 2013 at 8:08 PM, Felipe Contreras wrote: > This way all the remotes can share the same git objects, and the same > marks. The information we want to store per remote is very small. > > The code transparently converts from one organization of marks, to the > other. It's rather smooth and there shouldn't be any issues. Please drop this patch. While testing essentially the same functionality in remote-hg I noticed that it doesn't work when dealing with more than one remote. It's not clear if the marks can be shared at all, and if possible, it would be very tricky. Better drop it for now. I've tested that dropping this patch doesn't cause an conflicts for the rest of the series. And in case anybody is thinking that remote-bzr is really a too fast moving target; even if this managed to land in 'master', it's likely that people were not able to push at all, and in fact, many were not even able to clone in 1.8.2. So, hardly could be considered a regression. Nevertheless, I caught it in time. Cheers. -- Felipe Contreras ``` ## Junio C Hamano, 2013-04-27 00:17 Subject: Re: [PATCH 12/20] remote-bzr: split marks file Message-ID: <7v4nesvncw.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v4nesvncw.fsf%40alter.siamese.dyndns.org In-Reply-To: ``` Felipe Contreras writes: > And in case anybody is thinking that remote-bzr is really a too fast > moving target; even if this managed to land in 'master', it's likely > that people were not able to push at all, and in fact, many were not > even able to clone in 1.8.2. So, hardly could be considered a > regression. Nevertheless, I caught it in time. You didn't. I am already way too deep into today's 1.8.3-rc0 integration cycle and I won't waste a couple of hours work just to revert this. But from your description it sounds like with or without the patch the helper is equally broken and it does not make that much of a difference. ``` ## Felipe Contreras, 2013-04-27 00:35 Subject: Re: [PATCH 12/20] remote-bzr: split marks file Message-ID: URL: https://gitlist.dev/e/CAMP44s3jw-uy_BJyWjkGEPPeEiD4A8M3ywAAbmWRn44vfvDTvQ%40mail.gmail.com In-Reply-To: <7v4nesvncw.fsf@alter.siamese.dyndns.org> ``` On Fri, Apr 26, 2013 at 7:17 PM, Junio C Hamano wrote: > Felipe Contreras writes: > >> And in case anybody is thinking that remote-bzr is really a too fast >> moving target; even if this managed to land in 'master', it's likely >> that people were not able to push at all, and in fact, many were not >> even able to clone in 1.8.2. So, hardly could be considered a >> regression. Nevertheless, I caught it in time. > > You didn't. I am already way too deep into today's 1.8.3-rc0 > integration cycle and I won't waste a couple of hours work just to > revert this. All right, if you already merged this patch series, you could do what I did in my public branch: revert it, it's a single command, doesn't take hours. But I don't see this series in origin/master, am I missing something? > But from your description it sounds like with or without the patch > the helper is equally broken and it does not make that much of a > difference. Indeed. Many people would need this patch series to push, and this patch breaks the push for more than one remote. So if you merged the whole thing quite likely their situation would improve; they could at least push to one remote, instead of zero. Of course, if you remove this patch, they would be able to push to many. But I don't see it merged. Cheers. -- Felipe Contreras ``` ## Junio C Hamano, 2013-04-28 18:47 Subject: Re: [PATCH 12/20] remote-bzr: split marks file Message-ID: <7v1u9usdag.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v1u9usdag.fsf%40alter.siamese.dyndns.org In-Reply-To: <7v4nesvncw.fsf@alter.siamese.dyndns.org> ``` Junio C Hamano writes: > Felipe Contreras writes: > >> And in case anybody is thinking that remote-bzr is really a too fast >> moving target; even if this managed to land in 'master', it's likely >> that people were not able to push at all, and in fact, many were not >> even able to clone in 1.8.2. So, hardly could be considered a >> regression. Nevertheless, I caught it in time. > > You didn't. I am already way too deep into today's 1.8.3-rc0 > integration cycle and I won't waste a couple of hours work just to > revert this. Oh, I was lucky ;-) I mistook this with the other 9-patch bzr clean-up series that I applied to 'master' for -rc0. Pushing out a tagged-tip takes a lot longer than the normal tip because a lot more than what people see have to happen on my end. Reverting a single patch is simple, but we do not want to do that on top of "Git 1.8.3-rc0" commit and move the unpublished tag to point at the revert. Which means pretty much everything needs to be redone (one example among many is that the tagname will propagate to the htmldocs and manpages repositories, so their unpublished histories need to be rewound). But I didn't have to do that in the end ;-) ``` ## Felipe Contreras, 2013-04-29 04:57 Subject: Re: [PATCH 12/20] remote-bzr: split marks file Message-ID: URL: https://gitlist.dev/e/CAMP44s22AJ574nzXiXyMxV3SUiv8xz5yokS6__4Ck6oK4DwcGg%40mail.gmail.com In-Reply-To: <7v1u9usdag.fsf@alter.siamese.dyndns.org> ``` On Sun, Apr 28, 2013 at 1:47 PM, Junio C Hamano wrote: > Junio C Hamano writes: > >> Felipe Contreras writes: >> >>> And in case anybody is thinking that remote-bzr is really a too fast >>> moving target; even if this managed to land in 'master', it's likely >>> that people were not able to push at all, and in fact, many were not >>> even able to clone in 1.8.2. So, hardly could be considered a >>> regression. Nevertheless, I caught it in time. >> >> You didn't. I am already way too deep into today's 1.8.3-rc0 >> integration cycle and I won't waste a couple of hours work just to >> revert this. > > Oh, I was lucky ;-) I mistook this with the other 9-patch bzr > clean-up series that I applied to 'master' for -rc0. > > Pushing out a tagged-tip takes a lot longer than the normal tip > because a lot more than what people see have to happen on my end. > > Reverting a single patch is simple, but we do not want to do that on > top of "Git 1.8.3-rc0" commit and move the unpublished tag to point > at the revert. > > Which means pretty much everything needs to be redone (one example > among many is that the tagname will propagate to the htmldocs and > manpages repositories, so their unpublished histories need to be > rewound). > > But I didn't have to do that in the end ;-) Yeah, I realized you were talking about that one later on. I haven't heard anything bad from this new branch from emacs developers, so I think it's OK to merge it. Cheers. -- Felipe Contreras ```