threads / rfc / 29423

RFC patchAdapt to new semantics of remote-helper "import" command

Subject: [RFC/PATCH git-remote-bzr] Adapt to new semantics of remote-helper "import" command

## tl;dr

4 messages between Jan 22, 2012 and Feb 3, 2012. Diffs are folded; open one to read it.

replies: 3people: 3as markdown or json

Jonathan Nieder· Jan 22, 2012, 05:46 UTC · lore

Git 1.7.7 (commit 9504bc9d, "transport-helper: change import semantics", 2011-07-16) incompatibly changed the interface of the "import" capability.

Before, git would always send a single import command, which the remote helper would respond to with a fast-import stream, terminated by end of file, meaning there was no way to fetch multiple refs in one connection. Nowadays, git instead sends a sequence of import lines:

	import refs/heads/foo
	import refs/heads/bar

terminated by a blank line. The helper is to respond with a fast-import stream terminated by the "done" command and process further commands until another blank line indicates the end of the command stream. --- Hi Simon and Gabriel,

Here's a rough patch against git://github.com/lelutin/git-remote-bzr.git master.

Without this patch, whenever I try to use "git clone bzr::<something>", after doing all the work it removes the resulting repo and exits with status 141 (SIGPIPE). Maybe the transport-helper should mask SIGPIPE when writing the final newline to avoid that.

I'd have prefered to write a patch for remote-bzr that works with older versions of git fast-import, too, but it wasn't obvious how. Hints welcome.

BTW, would you mind if I sent a patch to include git-remote-bzr in git.git under contrib/?

Thanks for git remote-bzr!  I'd be happy for any thoughts you have.

Ciao, Jonathan

[1] http://thread.gmane.org/gmane.comp.version-control.git/176002/focus=176606
 README.rst     |    2 +-
 git-remote-bzr |   33 ++++++++++++++++++++++++++++++---
 2 files changed, 31 insertions(+), 4 deletions(-)
Show changes to 2 files +31 −4

README.rst, git-remote-bzr

diff --git a/README.rst b/README.rst
index 3eb3e476..f4dbbeb2 100644
--- a/README.rst
+++ b/README.rst
@@ -34,7 +34,7 @@ Relevant bug reports
 Requirements
 ------------
 
-- git 1.6.6 or later
+- git 1.7.7 or later
 - python 2.5 +
 - bzr 2.x
 - bzr-fastimport
diff --git a/git-remote-bzr b/git-remote-bzr
index 1e3a05f9..501fffe3 100755
--- a/git-remote-bzr
+++ b/git-remote-bzr
@@ -49,7 +49,7 @@ def do_list(repo, args):
     print  # end list
 
 
-def do_import(repo, args):
+def import_one_ref(repo, args):
     """Import a fast-import stream that is exported from Bazaar."""
     if len(args) != 1:
         die("Import needs exactly one ref")
@@ -65,6 +65,23 @@ def do_import(repo, args):
     if bzrp.wait():
         die("'bzr fast-export' returned unexpectedly with code %d",
             bzrp.returncode)
+    print "done"
+
+
+def do_import(repo,args):
+    import_one_ref(repo, args)
+
+    cmdline = True
+    while cmdline:
+        cmdline = next_command()
+        if not cmdline:
+            # Return to main processing loop
+            return True
+        cmd = cmdline.pop(0)
+        if cmd != "import":
+            warn("Unexpected command %s during import" % cmd)
+            return False
+        import_one_ref(repo, cmdline)
 
 
 def do_push(repo, args):
@@ -123,8 +140,8 @@ def sanitize(value):
     return value
 
 
-def read_one_line(repo):
-    """Read and process one command."""
+def next_command():
+    """Read one command."""
     line = sys.stdin.readline()
 
     cmdline = line
@@ -138,6 +155,16 @@ def read_one_line(repo):
         # Blank line means we're about to quit
         return False
 
+    return cmdline
+
+
+def read_one_line(repo):
+    """Read and process one command."""
+    cmdline = next_command()
+
+    if not cmdline:
+        return False
+
     cmd = cmdline.pop(0)
     debug("Got command '%s' with args '%s'", cmd, ' '.join(cmdline))
 
-- 
1.7.9.rc2
Jelmer Vernooij· Jan 22, 2012, 23:35 UTC · re: Jonathan Nieder · lore

Re: [RFC/PATCH git-remote-bzr] Adapt to new semantics of remote-helper "import" command

On 01/22/2012 06:46 AM, Jonathan Nieder wrote:
Show 33 quoted lines
> Git 1.7.7 (commit 9504bc9d, "transport-helper: change import
> semantics", 2011-07-16) incompatibly changed the interface of the
> "import" capability.
>
> Before, git would always send a single import command, which the
> remote helper would respond to with a fast-import stream, terminated
> by end of file, meaning there was no way to fetch multiple refs in one
> connection.  Nowadays, git instead sends a sequence of import lines:
>
> 	import refs/heads/foo
> 	import refs/heads/bar
>
> terminated by a blank line.  The helper is to respond with a
> fast-import stream terminated by the "done" command and process
> further commands until another blank line indicates the end of the
> command stream.
> ---
> Hi Simon and Gabriel,
>
> Here's a rough patch against git://github.com/lelutin/git-remote-bzr.git
> master.
>
> Without this patch, whenever I try to use "git clone bzr::<something>",
> after doing all the work it removes the resulting repo and exits with
> status 141 (SIGPIPE).  Maybe the transport-helper should mask SIGPIPE
> when writing the final newline to avoid that.
>
> I'd have prefered to write a patch for remote-bzr that works with
> older versions of git fast-import, too, but it wasn't obvious how.
> Hints welcome.
>
> BTW, would you mind if I sent a patch to include git-remote-bzr in
> git.git under contrib/?

Please note that the bzr-git package, which provides git integration for bzr and vice versa, also includes a 'git-remote-bzr' command. Apart from the 'import' command, it includes experimental implementations of 'fetch' and push as well.

It would be nice to consolidate the efforts, or at the very least prevent name clashes.

Cheers,
Jelmer
Jonathan Nieder· Jan 23, 2012, 00:12 UTC · re: Jelmer Vernooij · lore

Re: [RFC/PATCH git-remote-bzr] Adapt to new semantics of remote-helper "import" command

Jelmer Vernooij wrote:
> On 01/22/2012 06:46 AM, Jonathan Nieder wrote:
Show 5 quoted lines
>> BTW, would you mind if I sent a patch to include git-remote-bzr in
>> git.git under contrib/?
>
> Please note that the bzr-git package, which provides git integration
> for bzr and vice versa, also includes a 'git-remote-bzr' command.

That's good to hear. Then there should be no need for git.git to have its own helper.

Unfortunately, when I try to clone any repo, at the last step I get
 Traceback (most recent call last):map 6/7
  File "/usr/lib/git-core/git-remote-bzr", line 220, in <module>
    commands[argv[0]](argv, shortname, remote_dir)
  File "/usr/lib/git-core/git-remote-bzr", line 89, in cmd_list
    for ref, git_sha1 in refs.as_dict().iteritems():
  File "/usr/lib/python2.7/dist-packages/dulwich/repo.py", line 196, in as_dict
    ret[key] = self[("%s/%s" % (base, key)).strip("/")]
  File "/usr/lib/python2.7/dist-packages/dulwich/repo.py", line 267, in __getitem__
    _, sha = self._follow(name)
  File "/usr/lib/python2.7/dist-packages/dulwich/repo.py", line 249, in _follow
    contents = self.read_ref(refname)
  File "/usr/lib/python2.7/dist-packages/dulwich/repo.py", line 225, in read_ref
    contents = self.read_loose_ref(refname)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/git/refs.py", line 129, in read_loose_ref
    tag_name = ref_to_tag_name(ref)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/git/refs.py", line 89, in ref_to_tag_name
    raise ValueError("unable to map ref %s back to tag name" % ref)
 ValueError: unable to map ref refs/heads back to tag name
Will file a bug.
Gabriel Filion· Feb 3, 2012, 09:43 UTC · re: Jonathan Nieder · lore

Re: [RFC/PATCH git-remote-bzr] Adapt to new semantics of remote-helper "import" command

Hello,
On 12-01-22 12:46 AM, Jonathan Nieder wrote:
> Hi Simon and Gabriel,
> 
> Here's a rough patch against git://github.com/lelutin/git-remote-bzr.git
> master.
great! thanks for your help.

I must admit that this project never got to completion and is now getting quite the low priority for my part. I'm no longer working for the company that was using mainly Bazaar as their VCS, and I've been a happy git-only user for some time now. So, I don't have the same incentive to complete the project as I had before.

But I'd be happy to see this get to a point where it's working fine.
Show 8 quoted lines
> Without this patch, whenever I try to use "git clone bzr::<something>",
> after doing all the work it removes the resulting repo and exits with
> status 141 (SIGPIPE).  Maybe the transport-helper should mask SIGPIPE
> when writing the final newline to avoid that.
> 
> I'd have prefered to write a patch for remote-bzr that works with
> older versions of git fast-import, too, but it wasn't obvious how.
> Hints welcome.

hmm.. I can wait some time to see if some ideas come out around this, and commit your patch as-is if there are no comments/reworks.

> BTW, would you mind if I sent a patch to include git-remote-bzr in
> git.git under contrib/?
absolutely not, that'd be great actually :)

I didn't do that up to now, though, since I bumped into so much bugs that I couldn't work out -- some very bad performance issues, and problems with handling mark files with bzr-fastimport.

> Thanks for git remote-bzr!  I'd be happy for any thoughts you have.

The idea behind git-remote-bzr was to be able to interact with Bazaar from within your git repository, i.e. to expose remote branches that you can pull from and push to using the default git commands, without having to learn to use yet another tool since the remote-helper would be interfacing with the tool for you.

I have dived for a short period of time into bzrlib, the python library behind Bazaar, to see how much work it would take to put together a simplified fast-import client.. and .... wechrk! It was a huge maze of version-dependant code (the API undergoes big changes frequently, and backwards compatibility is maintained pretty far) mixed with a 40-foot-deep class hierarchy. So I quickly gave up on that idea..

IIRC, at the time I started work on this, Simon and I used bzr-fastimport[1] because we were able to get farther with this tool. I haven't used bzr-git[2] that much, though, so I can't comment too extensively on it. But I would guess that it is maintained more frequently than bzr-fastimport is, so it could be a better choice for the backend fast-import client.

I would guess that Jelmer would be happy to help out with interfacing with bzr-git.

-- 
Gabriel Filion

← back to recent threads