{"thread":{"id":"38120","subject":"Announcing a new (prototype) git-remote-hg tool","startedAt":"2014-12-05T20:53:35Z","lastAt":"2015-02-11T09:32:09Z","messageCount":9,"participants":["Mike Hommey","Jonathan Nieder","Philip Oakley","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"253188","messageId":"20141205205335.GA28935@glandium.org","threadId":"38120","inReplyTo":null,"subject":"Announcing a new (prototype) git-remote-hg tool","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-12-05T20:53:35Z","receivedAt":"2014-12-05T20:53:35Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nI've (re)started work on a longstanding idea of mine of having a git tool\ntalking the mercurial wire protocol directly. I'm now at a stage where\nthe tool can clone and pull/fetch from mercurial.\n\nAs it is a prototype, there are many things that it doesn't handle (like\nnamed branches, bookmarks, phases, obsolescence markers), but it\ncurrently transposes a complete mercurial dag to git and maintains\nmetadata about the original mercurial data.\n\nCode on https://github.com/glandium/git-remote-hg\n\nIt doesn't support push, but support for that should come in the coming\nweeks.\n\nMore background on http://glandium.org/blog/?p=3382\n\nThis is meant to be a prototype, and will stay that way for now.\nIt's a validation that this can actually work. Now that I have pull\nsupport I know I can make it push.\nI'm currently evaluating what the final tool would look like. I'm *very*\ntempted to implement it in C, based on core git code, because there are\nmany things that this helper does that would be so much easier to do\nwith direct access to git's guts. And that wouldn't require more\ndependencies than git currently has: it would \"just\" need curl and ssh,\nand git already uses both.\n\nIf I were to go in that direction, would you consider integrating it\nin git core? If not, would you rather see git helpers to make this\ngit-remote-hg helper more efficient?\n\nCheers,\n\nMike\n"},{"id":"253193","messageId":"20141205221319.GK16345@google.com","threadId":"38120","inReplyTo":"20141205205335.GA28935@glandium.org","subject":"Re: Announcing a new (prototype) git-remote-hg tool","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-12-05T22:13:19Z","receivedAt":"2014-12-05T22:13:19Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Mike Hommey wrote:\n\n> I'm currently evaluating what the final tool would look like. I'm *very*\n> tempted to implement it in C, based on core git code, because there are\n> many things that this helper does that would be so much easier to do\n> with direct access to git's guts. And that wouldn't require more\n> dependencies than git currently has: it would \"just\" need curl and ssh,\n> and git already uses both.\n>\n> If I were to go in that direction, would you consider integrating it\n> in git core?\n\nYes --- I would like this a lot.\n\nThe general trend has been to carry fewer contrib-style tools in-tree,\nsince the problem of discovering tools built on top of git is not as\nhard as it used to be.  What you describe above seems to be a bit of an\nexception:\n\n - libgit.a in its current state evolves too quickly for it to be\n   convenient for out-of-tree tools to use.  cgit <http://git.zx2c4.com/cgit/>\n   uses git pinned to a particular version as a submodule to get around\n   this, which is fussy and has bad implications for remembering to\n   get security updates.\n\n - an in-tree user of libgit.a would be useful as a reference example\n   to use to try to make libgit.a into be a better library internally\n   (and eventually expose e.g. by merging with libgit2 as something\n   outside tools can link to, I hope)\n\n - if it makes sense to help people using the current remote helper\n   in contrib to migrate to this, it could be convenient for users\n\nIn other words, although in the long term I would be happiest if\nlibgit becomes good enough to let this project live in a separate tree\nand link to it, it's tempting to build this in-tree because we're not\nthere yet.\n\nSome other alternatives:\n\n - using libgit2 <https://libgit2.github.com/>\n\n - improving git plumbing (e.g., with new fast-import commands) or\n   exposing a small library with a stable API for the tool's use\n\nI haven't thought it through carefully but at the moment I like the\nin-tree approach best.\n\nThanks,\nJonathan\n"},{"id":"253199","messageId":"0B0D868B05D243178E3A2AC2912938DA@PhilipOakley","threadId":"38120","inReplyTo":"20141205205335.GA28935@glandium.org","subject":"Re: Announcing a new (prototype) git-remote-hg tool","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2014-12-05T22:36:18Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Mike Hommey\" <mh@glandium.org>\n> Hi,\n>\n> I've (re)started work on a longstanding idea of mine of having a git \n> tool\n> talking the mercurial wire protocol directly. I'm now at a stage where\n> the tool can clone and pull/fetch from mercurial.\n>\n> As it is a prototype, there are many things that it doesn't handle \n> (like\n> named branches, bookmarks, phases, obsolescence markers), but it\n> currently transposes a complete mercurial dag to git and maintains\n> metadata about the original mercurial data.\n>\n> Code on https://github.com/glandium/git-remote-hg\n>\n> It doesn't support push, but support for that should come in the \n> coming\n> weeks.\n>\n> More background on http://glandium.org/blog/?p=3382\n>\n> This is meant to be a prototype, and will stay that way for now.\n> It's a validation that this can actually work. Now that I have pull\n> support I know I can make it push.\n> I'm currently evaluating what the final tool would look like. I'm \n> *very*\n> tempted to implement it in C, based on core git code, because there \n> are\n> many things that this helper does that would be so much easier to do\n> with direct access to git's guts. And that wouldn't require more\n> dependencies than git currently has: it would \"just\" need curl and \n> ssh,\n> and git already uses both.\n>\n> If I were to go in that direction, would you consider integrating it\n> in git core? If not, would you rather see git helpers to make this\n> git-remote-hg helper more efficient?\n>\nYou may also be interested in \nhttps://felipec.wordpress.com/2013/08/26/whats-new-in-git-v1-8-4-remote-hg/ \nand https://github.com/felipec/git-remote-hg.\n\nThough Filipe's unique work style hasn't found favour locally.\n\nsee also https://github.com/buchuki/gitifyhg/wiki/List-of-git-hg-bridges\n\nPhilip \n"},{"id":"253202","messageId":"20141205225930.GA29256@peff.net","threadId":"38120","inReplyTo":"20141205221319.GK16345@google.com","subject":"Re: Announcing a new (prototype) git-remote-hg tool","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-12-05T22:59:30Z","receivedAt":"2014-12-05T22:59:30Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 05, 2014 at 02:13:19PM -0800, Jonathan Nieder wrote:\n\n> Mike Hommey wrote:\n> \n> > I'm currently evaluating what the final tool would look like. I'm *very*\n> > tempted to implement it in C, based on core git code, because there are\n> > many things that this helper does that would be so much easier to do\n> > with direct access to git's guts. And that wouldn't require more\n> > dependencies than git currently has: it would \"just\" need curl and ssh,\n> > and git already uses both.\n> >\n> > If I were to go in that direction, would you consider integrating it\n> > in git core?\n> \n> Yes --- I would like this a lot.\n\nI'm concerned that this tool will have drawbacks that Felipe's remote-hg\ndoes not. And I can well imagine that it may, as that tool builds on\nMercurial's API, which will probably handle some corner cases\ndifferently. This isn't to disparage Mike's attempt; it will probably\nhave some upsides, too. But given that the approaches are so different,\nit does not seem obvious to me that one will always be better than the\nother.\n\nOne of the nice things about spinning remote-hg out of the core repo is\nthat it means we do not have to endorse a particular implementation, and\nthey can compete with each other on their merits.  I would very much\nhate to see Felipe's remote-hg project wither and die just because\nanother implementation becomes the de facto standard by being included\nin git.git. It's a proven tool, and this new thing is not yet.\n\nIt's a shame that both squat on the name \"remote-hg\", because it makes\nit difficult to tell the two apart. But of course that is the only way\nto make \"git clone hg::...\" work. Maybe we need a layer of indirection?\n:)\n\n>  - libgit.a in its current state evolves too quickly for it to be\n>    convenient for out-of-tree tools to use.  cgit <http://git.zx2c4.com/cgit/>\n>    uses git pinned to a particular version as a submodule to get around\n>    this, which is fussy and has bad implications for remembering to\n>    get security updates.\n\nI'm not sure that this approach is any better than carrying something in\ncontrib/ in git.git. If I refactor a function in libgit.a, I notice\nbreakage in the callers because it no longer compiles, or because I am\nthorough and look at the implications to git callers.\n\nI do _not_ want to be responsible for making sure that contrib/* still\nbuilds. That is the problem of the maintainer of the contrib/ project in\nquestion. That may sound a little selfish, but I think that is what it\nmeans to be in contrib, and not in the regular tree.\n\nSo once you realize that is the burden of the contrib/ author to fix\nbreakages, then the process is:\n\n  git pull\n  cd contrib/c-remote-hg\n  make\n  # oops, it broke\n  fix fix fix\n\nThat is not any different than:\n\n  git submodule add git...\n  git submodule update\n  make\n  # oops, it broke\n  fix fix fix\n\nThe hard part is not how you pull changes from the new git into your\ntree. It is the fact that upstream may be breaking the interface behind\nyour back.  And your best bet is to aggressively merge with upstream,\nrather than trying to track only occasional release versions.\n\nOf course, if you meant to _really_ carry it in-tree, not in contrib/,\nthen none of that applies. But then I worry doubly about the\n\"endorsement\" issue.\n\n-Peff\n"},{"id":"253205","messageId":"20141205231330.GL16345@google.com","threadId":"38120","inReplyTo":"20141205225930.GA29256@peff.net","subject":"Re: Announcing a new (prototype) git-remote-hg tool","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-12-05T23:13:30Z","receivedAt":"2014-12-05T23:13:30Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff King wrote:\n\n> One of the nice things about spinning remote-hg out of the core repo is\n> that it means we do not have to endorse a particular implementation, and\n> they can compete with each other on their merits.\n\nTrue.\n\n[...]\n> It's a shame that both squat on the name \"remote-hg\", because it makes\n> it difficult to tell the two apart. But of course that is the only way\n> to make \"git clone hg::...\" work. Maybe we need a layer of indirection?\n> :)\n\nIf the helpers are roughly interchangeable (that is, if you can switch\nbetween fetching using each one into the same on-disk git repository),\nthen picking one to symlink as git-remote-hg in your $PATH should be\nenough.\n\nIf they don't have that level of interoperability, then there's an\nargument to be made that the URLs shouldn't be the same.\nUnfortunately url.*.insteadof rules are resolved at fetch time instead\nof being resolved once and the result recorded in .git/config.  So\nyes, it seems like a way to have abbreviations for URLs (e.g., hg::\nmeaning hg+mh:: or hg+fc::) that get resolved at clone time would be\nuseful.  It's a layer of indirection we don't provide. :/\n\nThanks,\nJonathan\n"},{"id":"253208","messageId":"20141205233106.GA832@glandium.org","threadId":"38120","inReplyTo":"20141205225930.GA29256@peff.net","subject":"Re: Announcing a new (prototype) git-remote-hg tool","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-12-05T23:31:06Z","receivedAt":"2014-12-05T23:31:06Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, Dec 05, 2014 at 05:59:30PM -0500, Jeff King wrote:\n> On Fri, Dec 05, 2014 at 02:13:19PM -0800, Jonathan Nieder wrote:\n> \n> > Mike Hommey wrote:\n> > \n> > > I'm currently evaluating what the final tool would look like. I'm *very*\n> > > tempted to implement it in C, based on core git code, because there are\n> > > many things that this helper does that would be so much easier to do\n> > > with direct access to git's guts. And that wouldn't require more\n> > > dependencies than git currently has: it would \"just\" need curl and ssh,\n> > > and git already uses both.\n> > >\n> > > If I were to go in that direction, would you consider integrating it\n> > > in git core?\n> > \n> > Yes --- I would like this a lot.\n> \n> I'm concerned that this tool will have drawbacks that Felipe's remote-hg\n> does not. And I can well imagine that it may, as that tool builds on\n> Mercurial's API, which will probably handle some corner cases\n> differently.\n\nFWIW, my tool only uses the mercurial code for the wire protocol. This\ncan (and if I go the C route, will) be implemented without using\nmercurial code, it's really not a hard problem.\n\n> This isn't to disparage Mike's attempt; it will probably\n> have some upsides, too. But given that the approaches are so different,\n> it does not seem obvious to me that one will always be better than the\n> other.\n> \n> One of the nice things about spinning remote-hg out of the core repo is\n> that it means we do not have to endorse a particular implementation, and\n> they can compete with each other on their merits.  I would very much\n> hate to see Felipe's remote-hg project wither and die just because\n> another implementation becomes the de facto standard by being included\n> in git.git. It's a proven tool, and this new thing is not yet.\n\nNote that I'm only talking about an hypothetical long term goal. If\nthere's not even a slim chance that this may end up in git core, or in\nthe git.git repo, I'm not sure it's worth writing the tool in C at all,\nconsidering the burden for users. IOW, I'm only trying to assess if I\nshould follow my temptation or not. But I can probably reassess after I\nactually get my prototype to do more than it does now. But maybe there\nare ways to make it work for users outside of git.git even if it's in C.\nI don't know.\n\n> It's a shame that both squat on the name \"remote-hg\", because it makes\n> it difficult to tell the two apart. But of course that is the only way\n> to make \"git clone hg::...\" work. Maybe we need a layer of indirection?\n> :)\n\nYeah, that's an unfortunate consequence of how remote helpers work.\nThere are already two different git-remote-hgs (there's felipe's, and\nthere's another one using hg-git under the hood) that I know of. I'm\nadding a third one. For what it's worth, none of the existing one is\nsatisfying on repos the size of Mozilla's, and apparently noone at\nMozilla uses them because of that. Add to that the disk space\ninefficiency of actually keeping a copy of the mercurial repo locally.\nThe existing tools can likely be improved to scale better, but that\nwouldn't change the disk space problem.\n\nMike\n"},{"id":"253212","messageId":"20141205234622.GA2457@glandium.org","threadId":"38120","inReplyTo":"20141205231330.GL16345@google.com","subject":"Re: Announcing a new (prototype) git-remote-hg tool","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-12-05T23:46:22Z","receivedAt":"2014-12-05T23:46:22Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, Dec 05, 2014 at 03:13:30PM -0800, Jonathan Nieder wrote:\n> Jeff King wrote:\n> \n> > One of the nice things about spinning remote-hg out of the core repo is\n> > that it means we do not have to endorse a particular implementation, and\n> > they can compete with each other on their merits.\n> \n> True.\n> \n> [...]\n> > It's a shame that both squat on the name \"remote-hg\", because it makes\n> > it difficult to tell the two apart. But of course that is the only way\n> > to make \"git clone hg::...\" work. Maybe we need a layer of indirection?\n> > :)\n> \n> If the helpers are roughly interchangeable (that is, if you can switch\n> between fetching using each one into the same on-disk git repository),\n> then picking one to symlink as git-remote-hg in your $PATH should be\n> enough.\n> \n> If they don't have that level of interoperability, then there's an\n> argument to be made that the URLs shouldn't be the same.\n\nI don't think Felipe's and the one that uses hg-git under the hood are\nalready interoperable. Mine is also different from both. They should all\nproduce the same git trees. They don't produce the same commits. They\nalso don't share the same metadata.\n\nMike\n"},{"id":"253228","messageId":"20141206050628.GB31301@peff.net","threadId":"38120","inReplyTo":"20141205231330.GL16345@google.com","subject":"Re: Announcing a new (prototype) git-remote-hg tool","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-12-06T05:06:29Z","receivedAt":"2014-12-06T05:06:29Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 05, 2014 at 03:13:30PM -0800, Jonathan Nieder wrote:\n\n> > It's a shame that both squat on the name \"remote-hg\", because it makes\n> > it difficult to tell the two apart. But of course that is the only way\n> > to make \"git clone hg::...\" work. Maybe we need a layer of indirection?\n> > :)\n> \n> If the helpers are roughly interchangeable (that is, if you can switch\n> between fetching using each one into the same on-disk git repository),\n> then picking one to symlink as git-remote-hg in your $PATH should be\n> enough.\n\nThat may be enough. For the most part you do not need to agree with\nother members of the project on which implementation to use. My\nexperience with import tools has been that either:\n\n  1. you are using them personally (because you do not like the\n     upstream's choice of VCS and would prefer to transparently work in\n     your favorite tool), or\n\n  2. there is a group of developers who want to use git, but\n     somebody provides an unofficial git mirror. They do not have to\n     agree on the tool, because they just use git directly from the\n     mirror.\n\nSo it is mostly a personal choice. But the two confusions I'd still\nanticipate are:\n\n  a. It's difficult to even _talk_ about the tools, because the names\n     are the same (so searching for tips on the tool, reporting bugs,\n     etc, are harder than necessary).\n\n  b. You may want different tools for different projects. If one tool is\n     much more efficient, you may need it for a large repo (e.g.,\n     mozilla). But another tool may provide other features, and you\n     would prefer it for smaller repos.\n\nThis is largely speculation, though, and I do not actively use the tools\nmyself. So I'd be happy to push off dealing with it until it itches\nenough for somebody to scratch.\n\n-Peff\n"},{"id":"255922","messageId":"20150211093209.GA21128@glandium.org","threadId":"38120","inReplyTo":"20141205205335.GA28935@glandium.org","subject":"Announcing git-cinnabar 0.1.0 (Was: Announcing a new (prototype) git-remote-hg tool)","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2015-02-11T09:32:09Z","receivedAt":"2015-02-11T09:32:09Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nCinnabar is the common natural form in which mercury can be found on\nEarth. It contains mercury sulfide and its powder is used to make the\nvermillion pigment.\n\nWhat does that have to do with git?\n\nHint: mercury.\n\nGit-cinnabar is a git remote helper to interact with mercurial\nrepositories. It allows to clone, pull and push from/to mercurial remote\nrepositories, using git.\n\nIf you've already seen the original thread this message is spawned from,\nthis is the same tool, with a new name, and two months worth of\nadditional work and testing.\n\nWhere git-cinnabar stands out compared to other similar tools is that it\ndoesn't use a local mercurial clone under the hood (unlike all the\nexisting other such tools), and is close to an order of magnitude faster\nto clone a repository like http://hg.mozilla.org/mozilla-central than\nthe git-remote-hg tool that used to be shipped in contrib/.\n\nI won't claim it is exempt of problems and limitations, which is why it's\nnot a 1.0. I'm however confident enough with its state to make the\nfirst \"official\" release.\n\nCode on https://github.com/glandium/git-cinnabar\n\nCheers,\n\nMike\n"}]}