Announcing a new (prototype) git-remote-hg tool

Hi,

I've (re)started work on a longstanding idea of mine of having a git tool talking the mercurial wire protocol directly. I'm now at a stage where the tool can clone and pull/fetch from mercurial.

As it is a prototype, there are many things that it doesn't handle (like named branches, bookmarks, phases, obsolescence markers), but it currently transposes a complete mercurial dag to git and maintains metadata about the original mercurial data.

Code on https://github.com/glandium/git-remote-hg

It doesn't support push, but support for that should come in the coming weeks.

More background on http://glandium.org/blog/?p=3382

This is meant to be a prototype, and will stay that way for now. It's a validation that this can actually work. Now that I have pull support I know I can make it push. I'm currently evaluating what the final tool would look like. I'm *very* tempted to implement it in C, based on core git code, because there are many things that this helper does that would be so much easier to do with direct access to git's guts. And that wouldn't require more dependencies than git currently has: it would "just" need curl and ssh, and git already uses both.

If I were to go in that direction, would you consider integrating it in git core? If not, would you rather see git helpers to make this git-remote-hg helper more efficient?

Cheers,

Mike

Mike Hommey wrote:

Yes --- I would like this a lot.

The general trend has been to carry fewer contrib-style tools in-tree, since the problem of discovering tools built on top of git is not as hard as it used to be. What you describe above seems to be a bit of an exception:

 - libgit.a in its current state evolves too quickly for it to be
   convenient for out-of-tree tools to use.  cgit <http://git.zx2c4.com/cgit/>
   uses git pinned to a particular version as a submodule to get around
   this, which is fussy and has bad implications for remembering to
   get security updates.

 - an in-tree user of libgit.a would be useful as a reference example
   to use to try to make libgit.a into be a better library internally
   (and eventually expose e.g. by merging with libgit2 as something
   outside tools can link to, I hope)

 - if it makes sense to help people using the current remote helper
   in contrib to migrate to this, it could be convenient for users

In other words, although in the long term I would be happiest if libgit becomes good enough to let this project live in a separate tree and link to it, it's tempting to build this in-tree because we're not there yet.

Some other alternatives:

 - using libgit2 <https://libgit2.github.com/>

 - improving git plumbing (e.g., with new fast-import commands) or
   exposing a small library with a stable API for the tool's use

I haven't thought it through carefully but at the moment I like the in-tree approach best.

Thanks, Jonathan

On Fri, Dec 05, 2014 at 02:13:19PM -0800, Jonathan Nieder wrote:

I'm concerned that this tool will have drawbacks that Felipe's remote-hg does not. And I can well imagine that it may, as that tool builds on Mercurial's API, which will probably handle some corner cases differently. This isn't to disparage Mike's attempt; it will probably have some upsides, too. But given that the approaches are so different, it does not seem obvious to me that one will always be better than the other.

One of the nice things about spinning remote-hg out of the core repo is that it means we do not have to endorse a particular implementation, and they can compete with each other on their merits. I would very much hate to see Felipe's remote-hg project wither and die just because another implementation becomes the de facto standard by being included in git.git. It's a proven tool, and this new thing is not yet.

It's a shame that both squat on the name "remote-hg", because it makes it difficult to tell the two apart. But of course that is the only way to make "git clone hg::..." work. Maybe we need a layer of indirection? :)

I'm not sure that this approach is any better than carrying something in contrib/ in git.git. If I refactor a function in libgit.a, I notice breakage in the callers because it no longer compiles, or because I am thorough and look at the implications to git callers.

I do _not_ want to be responsible for making sure that contrib/* still builds. That is the problem of the maintainer of the contrib/ project in question. That may sound a little selfish, but I think that is what it means to be in contrib, and not in the regular tree.

So once you realize that is the burden of the contrib/ author to fix breakages, then the process is:

  git pull
  cd contrib/c-remote-hg
  make
  # oops, it broke
  fix fix fix

That is not any different than:

  git submodule add git...
  git submodule update
  make
  # oops, it broke
  fix fix fix

The hard part is not how you pull changes from the new git into your tree. It is the fact that upstream may be breaking the interface behind your back. And your best bet is to aggressively merge with upstream, rather than trying to track only occasional release versions.

Of course, if you meant to _really_ carry it in-tree, not in contrib/, then none of that applies. But then I worry doubly about the "endorsement" issue.

-Peff

Jeff King wrote:

True.

[...]

If the helpers are roughly interchangeable (that is, if you can switch between fetching using each one into the same on-disk git repository), then picking one to symlink as git-remote-hg in your $PATH should be enough.

If they don't have that level of interoperability, then there's an argument to be made that the URLs shouldn't be the same. Unfortunately url.*.insteadof rules are resolved at fetch time instead of being resolved once and the result recorded in .git/config. So yes, it seems like a way to have abbreviations for URLs (e.g., hg:: meaning hg+mh:: or hg+fc::) that get resolved at clone time would be useful. It's a layer of indirection we don't provide. :/

Thanks, Jonathan

On Fri, Dec 05, 2014 at 03:13:30PM -0800, Jonathan Nieder wrote:

I don't think Felipe's and the one that uses hg-git under the hood are already interoperable. Mine is also different from both. They should all produce the same git trees. They don't produce the same commits. They also don't share the same metadata.

Mike

On Fri, Dec 05, 2014 at 03:13:30PM -0800, Jonathan Nieder wrote:

That may be enough. For the most part you do not need to agree with other members of the project on which implementation to use. My experience with import tools has been that either:

  1. you are using them personally (because you do not like the
     upstream's choice of VCS and would prefer to transparently work in
     your favorite tool), or

  2. there is a group of developers who want to use git, but
     somebody provides an unofficial git mirror. They do not have to
     agree on the tool, because they just use git directly from the
     mirror.

So it is mostly a personal choice. But the two confusions I'd still anticipate are:

  a. It's difficult to even _talk_ about the tools, because the names
     are the same (so searching for tips on the tool, reporting bugs,
     etc, are harder than necessary).

  b. You may want different tools for different projects. If one tool is
     much more efficient, you may need it for a large repo (e.g.,
     mozilla). But another tool may provide other features, and you
     would prefer it for smaller repos.

This is largely speculation, though, and I do not actively use the tools myself. So I'd be happy to push off dealing with it until it itches enough for somebody to scratch.

-Peff

On Fri, Dec 05, 2014 at 05:59:30PM -0500, Jeff King wrote:

FWIW, my tool only uses the mercurial code for the wire protocol. This can (and if I go the C route, will) be implemented without using mercurial code, it's really not a hard problem.

Note that I'm only talking about an hypothetical long term goal. If there's not even a slim chance that this may end up in git core, or in the git.git repo, I'm not sure it's worth writing the tool in C at all, considering the burden for users. IOW, I'm only trying to assess if I should follow my temptation or not. But I can probably reassess after I actually get my prototype to do more than it does now. But maybe there are ways to make it work for users outside of git.git even if it's in C. I don't know.

Yeah, that's an unfortunate consequence of how remote helpers work. There are already two different git-remote-hgs (there's felipe's, and there's another one using hg-git under the hood) that I know of. I'm adding a third one. For what it's worth, none of the existing one is satisfying on repos the size of Mozilla's, and apparently noone at Mozilla uses them because of that. Add to that the disk space inefficiency of actually keeping a copy of the mercurial repo locally. The existing tools can likely be improved to scale better, but that wouldn't change the disk space problem.

Mike

From: "Mike Hommey" <mh@glandium.org>

You may also be interested in https://felipec.wordpress.com/2013/08/26/whats-new-in-git-v1-8-4-remote-hg/ and https://github.com/felipec/git-remote-hg.

Though Filipe's unique work style hasn't found favour locally.

see also https://github.com/buchuki/gitifyhg/wiki/List-of-git-hg-bridges

Philip 

Hi,

Cinnabar is the common natural form in which mercury can be found on Earth. It contains mercury sulfide and its powder is used to make the vermillion pigment.

What does that have to do with git?

Hint: mercury.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

If you've already seen the original thread this message is spawned from, this is the same tool, with a new name, and two months worth of additional work and testing.

Where git-cinnabar stands out compared to other similar tools is that it doesn't use a local mercurial clone under the hood (unlike all the existing other such tools), and is close to an order of magnitude faster to clone a repository like http://hg.mozilla.org/mozilla-central than the git-remote-hg tool that used to be shipped in contrib/.

I won't claim it is exempt of problems and limitations, which is why it's not a 1.0. I'm however confident enough with its state to make the first "official" release.

Code on https://github.com/glandium/git-cinnabar

Cheers,

Mike