{"thread":{"id":"36329","subject":"fast-import deltas","startedAt":"2014-04-01T10:25:54Z","lastAt":"2014-04-09T17:44:52Z","messageCount":13,"participants":["Mike Hommey","Jeff King","Junio C Hamano","Jonathan Nieder","Max Horn","Felipe Contreras"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"238195","messageId":"20140401102554.GA32231@glandium.org","threadId":"36329","inReplyTo":null,"subject":"fast-import deltas","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-04-01T10:25:54Z","receivedAt":"2014-04-01T10:25:54Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nI am currently prototyping a \"native\" mercurial remote handler for git,\nand it seems silly for git to compute deltas itself when I'm getting\ndeltas from the mercurial remote itself, albeit in a different form.\n\nWould adding a fast-import command to handle deltas be considered useful\nfor git? If so, what kind of format would be suitable?\n\nCheers,\n\nMike\n"},{"id":"238197","messageId":"20140401114502.GA15549@sigill.intra.peff.net","threadId":"36329","inReplyTo":"20140401102554.GA32231@glandium.org","subject":"Re: fast-import deltas","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-04-01T11:45:03Z","receivedAt":"2014-04-01T11:45:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 01, 2014 at 07:25:54PM +0900, Mike Hommey wrote:\n\n> I am currently prototyping a \"native\" mercurial remote handler for git,\n\nFor my own curiosity, how does this differ from what is in\ncontrib/remote-helpers/git-remote-hg?\n\n> Would adding a fast-import command to handle deltas be considered useful\n> for git? If so, what kind of format would be suitable?\n\nIt breaks fast-import's \"lowest common denominator\" data model that is\njust passing commits and their contents over the stream. But we already\ndo that in other cases for the sake of performance. I think the\nimportant thing is that the alternate formats are optional and enabled\nby the caller with command-line options.\n\nThat being said, I foresee a few complications:\n\n  1. Git needs to know the sha1 of the full object. So unless the\n     generating script knows that ahead of time, git has to expand the\n     delta immediately anyway (this could still be a win if we end up\n     using a good delta from elsewhere rather than doing our own delta\n     search, but I suspect it's not so big a win as if we can just blit\n     the delta straight to disk).\n\n  2. Git does not store on-disk deltas between objects that are not in\n     the same packfile. So you'd only be able to delta against an object\n     that came in the same stream (or you'd have to \"fix\" the packfile\n     on disk by adding an extra copy of the delta base, but that\n     probably eliminates any savings).\n\nAs for format, I believe that git is basically xdelta under the hood, so\nyou'd want either that or something that can be trivially converted to\nit.\n\n-Peff\n"},{"id":"238201","messageId":"20140401130703.GA1479@glandium.org","threadId":"36329","inReplyTo":"20140401114502.GA15549@sigill.intra.peff.net","subject":"Re: fast-import deltas","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-04-01T13:07:03Z","receivedAt":"2014-04-01T13:07:03Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Apr 01, 2014 at 07:45:03AM -0400, Jeff King wrote:\n> On Tue, Apr 01, 2014 at 07:25:54PM +0900, Mike Hommey wrote:\n> \n> > I am currently prototyping a \"native\" mercurial remote handler for git,\n> \n> For my own curiosity, how does this differ from what is in\n> contrib/remote-helpers/git-remote-hg?\n\ncontrib/remote-helpers/git-remote-hg does a local mercurial clone before\ndoing the git conversion. While this is not really a problem for most\nmercurial projects, it tends to be slow with big ones, like the firefox\nsource code. What I'm aiming at is something that can talk directly to a\nremote mercurial server.\n\n> > Would adding a fast-import command to handle deltas be considered useful\n> > for git? If so, what kind of format would be suitable?\n> \n> It breaks fast-import's \"lowest common denominator\" data model that is\n> just passing commits and their contents over the stream. But we already\n> do that in other cases for the sake of performance. I think the\n> important thing is that the alternate formats are optional and enabled\n> by the caller with command-line options.\n> \n> That being said, I foresee a few complications:\n> \n>   1. Git needs to know the sha1 of the full object. So unless the\n>      generating script knows that ahead of time, git has to expand the\n>      delta immediately anyway (this could still be a win if we end up\n>      using a good delta from elsewhere rather than doing our own delta\n>      search, but I suspect it's not so big a win as if we can just blit\n>      the delta straight to disk).\n\nGood point. That could quickly become a problem with long delta chains.\n\n>   2. Git does not store on-disk deltas between objects that are not in\n>      the same packfile. So you'd only be able to delta against an object\n>      that came in the same stream (or you'd have to \"fix\" the packfile\n>      on disk by adding an extra copy of the delta base, but that\n>      probably eliminates any savings).\n\nArguably, this would make the most difference on initial clone of big\nprojects, or large incremental updates (like, after a few weeks), which\nwould use a single pack anyways.\n\n> As for format, I believe that git is basically xdelta under the hood, so\n> you'd want either that or something that can be trivially converted to\n> it.\n\nIt seems to me fast-import keeps a kind of human readable format for its\nprotocol, i wonder if xdelta format would fit the bill. That being said,\nI also wonder if i shouldn't just try to write a pack on my own...\n\nCheers,\n\nMike\n"},{"id":"238202","messageId":"20140401131512.GA19321@sigill.intra.peff.net","threadId":"36329","inReplyTo":"20140401130703.GA1479@glandium.org","subject":"Re: fast-import deltas","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-04-01T13:15:12Z","receivedAt":"2014-04-01T13:15:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 01, 2014 at 10:07:03PM +0900, Mike Hommey wrote:\n\n> > For my own curiosity, how does this differ from what is in\n> > contrib/remote-helpers/git-remote-hg?\n> \n> contrib/remote-helpers/git-remote-hg does a local mercurial clone before\n> doing the git conversion. While this is not really a problem for most\n> mercurial projects, it tends to be slow with big ones, like the firefox\n> source code. What I'm aiming at is something that can talk directly to a\n> remote mercurial server.\n\nAh, that makes sense. Thanks for explaining.\n\n> >   2. Git does not store on-disk deltas between objects that are not in\n> >      the same packfile. So you'd only be able to delta against an object\n> >      that came in the same stream (or you'd have to \"fix\" the packfile\n> >      on disk by adding an extra copy of the delta base, but that\n> >      probably eliminates any savings).\n> \n> Arguably, this would make the most difference on initial clone of big\n> projects, or large incremental updates (like, after a few weeks), which\n> would use a single pack anyways.\n\nYeah. The nice thing is that this can be an opportunistic optimization.\nIf the delta base is part of the same output stream, then send the\ndelta. Otherwise, you can always fall back to reconstructing and sending\nthe full object yourself.\n\n> It seems to me fast-import keeps a kind of human readable format for its\n> protocol, i wonder if xdelta format would fit the bill. That being said,\n> I also wonder if i shouldn't just try to write a pack on my own...\n\nThe fast-import commands are human readable, but the blob contents are\nincluded inline. I don't see how sending a binary delta is any worse\nthan sending a literal binary blob over the stream.\n\n-Peff\n"},{"id":"238205","messageId":"20140401141856.GA2497@glandium.org","threadId":"36329","inReplyTo":"20140401131512.GA19321@sigill.intra.peff.net","subject":"Re: fast-import deltas","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-04-01T14:18:56Z","receivedAt":"2014-04-01T14:18:56Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Apr 01, 2014 at 09:15:12AM -0400, Jeff King wrote:\n> > It seems to me fast-import keeps a kind of human readable format for its\n> > protocol, i wonder if xdelta format would fit the bill. That being said,\n> > I also wonder if i shouldn't just try to write a pack on my own...\n> \n> The fast-import commands are human readable, but the blob contents are\n> included inline. I don't see how sending a binary delta is any worse\n> than sending a literal binary blob over the stream.\n\nOTOH, the xdelta format is not exactly straightforward to produce, with\nthe variable length encoding of integers. Not exactly hard, but when\neverything else in fast-import is straightforward, one has to wonder.\n\nMike\n"},{"id":"238230","messageId":"xmqqk3b90y79.fsf@gitster.dls.corp.google.com","threadId":"36329","inReplyTo":"20140401141856.GA2497@glandium.org","subject":"Re: fast-import deltas","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-01T17:14:02Z","receivedAt":"2014-04-01T17:14:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mike Hommey <mh@glandium.org> writes:\n\n> On Tue, Apr 01, 2014 at 09:15:12AM -0400, Jeff King wrote:\n>> > It seems to me fast-import keeps a kind of human readable format for its\n>> > protocol, i wonder if xdelta format would fit the bill. That being said,\n>> > I also wonder if i shouldn't just try to write a pack on my own...\n>> \n>> The fast-import commands are human readable, but the blob contents are\n>> included inline. I don't see how sending a binary delta is any worse\n>> than sending a literal binary blob over the stream.\n>\n> OTOH, the xdelta format is not exactly straightforward to produce, with\n> the variable length encoding of integers. Not exactly hard, but when\n> everything else in fast-import is straightforward, one has to wonder.\n\nUnless you already have your change in the xdelta on hand, or the\nformat your foreign change is in gives sufficient information to\nproduce a corresponding xdelta without looking at the content that\nyour foreign change applies to, it is silly to try to convert your\nforeign change into xdelta and feed it to fast-import.\n\nWhat constitutes \"sufficient\" information?  The xdelta format is a\nseries of instructions that lets you:\n\n - copy N bytes from offset in the source material to the\n   destination; or\n - copy these N literal bytes to the destination.\n\nto an existing piece of content, identified by the object name of\nthe \"source material\", to produce a result of \"applying delta\".\n\nAs an example, think about the case where you have *,v files used by\nRCS (and CVS).  The \"foreign changes\" given to you by that format is\na series of instructions that roughly corresponds to an \"ed\" script.\nInsert these lines at the line number L, delete N lines from line\nnumber K, etc.  In order to convert such a change into xdelta, you\nwould need to know what these line numbers correspond to byte offset\nin the original file.  You also may want to know what the Git object\nname for the original is, although in the fast-import stream you\nmight be able to get away by using the object mark facility.\n\nAssuming that you do have and are willing to read the original file,\nyou have three possible (and one impractical) approaches:\n\n - Apply the foreign changes to the original file yourself (as that\n   is the foreign system you are interested in, you know how to do\n   that much better than Git does), and produce xdelta between the\n   original and the result using only the original and the result.\n\n - Apply the foreign changes to the original file yourself, and feed\n   the resulting content to fast-import in full, letting fast-import\n   convert into the format Git understands.\n\n - Interpret the foreign changes, using the original file as a\n   reference, to convert it into xdelta.\n\n - Teach fast-import how to interpret various formats that are used\n   to express foreign changes, and feed that.\n\nIn the first approach, this \"given the original and the result,\nproduce xdelta between them\" can be reused by other people's\nsystem.  You may be able to borrow diff-delta.c from us under our\nlicensing terms.\n\nThe second is the most straightforward; eventual deltification will\nhappen when the resulting repository is repacked and uses the same\ncode from diff-delta.c.\n\nThe third would be \"*,v expresses the source location and length in\nterms of lines, so look at the original to convert these into byte\noffset and byte length xdelta wants\", which I would think is silly.\n\nAnd the last one is a maintenance nightmare I do not think we would\nwant to touch with a ten-foot pole.\n\nIn short, the most practical solution would be to reconstitute a\nfull object and feed that to fast-import, unless you already have\nxdelta or you can turn your foreign change into xdelta without ever\nlooking at the original.\n"},{"id":"238231","messageId":"20140401173856.GC6851@google.com","threadId":"36329","inReplyTo":"xmqqk3b90y79.fsf@gitster.dls.corp.google.com","subject":"Re: fast-import deltas","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-04-01T17:38:56Z","receivedAt":"2014-04-01T17:38:56Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n\n> Assuming that you do have and are willing to read the original file,\n> you have three possible (and one impractical) approaches:\n[...]\n>  - Apply the foreign changes to the original file yourself, and feed\n>    the resulting content to fast-import in full, letting fast-import\n>    convert into the format Git understands.\n\nThis (when importing from Subversion) was the motivation for\nintroducing fast-import's cat-blob command, for what it's worth.\n\n[...]\n> In short, the most practical solution would be to reconstitute a\n> full object and feed that to fast-import, unless you already have\n> xdelta or you can turn your foreign change into xdelta without ever\n> looking at the original.\n\nIf your delta format happens to be similar enough to xdelta, then\nstreaming in deltas in xdelta format does sound like a neat trick.\n\nMaybe it would be useful to provide a micro-library that creates and\nvalidates xdelta opcodes for fast-import frontends to use.\n\nJonathan\n"},{"id":"238278","messageId":"20140401221003.GA5923@glandium.org","threadId":"36329","inReplyTo":"xmqqk3b90y79.fsf@gitster.dls.corp.google.com","subject":"Re: fast-import deltas","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-04-01T22:10:03Z","receivedAt":"2014-04-01T22:10:03Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Apr 01, 2014 at 10:14:02AM -0700, Junio C Hamano wrote:\n> Mike Hommey <mh@glandium.org> writes:\n> \n> > On Tue, Apr 01, 2014 at 09:15:12AM -0400, Jeff King wrote:\n> >> > It seems to me fast-import keeps a kind of human readable format for its\n> >> > protocol, i wonder if xdelta format would fit the bill. That being said,\n> >> > I also wonder if i shouldn't just try to write a pack on my own...\n> >> \n> >> The fast-import commands are human readable, but the blob contents are\n> >> included inline. I don't see how sending a binary delta is any worse\n> >> than sending a literal binary blob over the stream.\n> >\n> > OTOH, the xdelta format is not exactly straightforward to produce, with\n> > the variable length encoding of integers. Not exactly hard, but when\n> > everything else in fast-import is straightforward, one has to wonder.\n> \n> Unless you already have your change in the xdelta on hand, or the\n> format your foreign change is in gives sufficient information to\n> produce a corresponding xdelta without looking at the content that\n> your foreign change applies to, it is silly to try to convert your\n> foreign change into xdelta and feed it to fast-import.\n> \n> What constitutes \"sufficient\" information?  The xdelta format is a\n> series of instructions that lets you:\n> \n>  - copy N bytes from offset in the source material to the\n>    destination; or\n>  - copy these N literal bytes to the destination.\n> \n> to an existing piece of content, identified by the object name of\n> the \"source material\", to produce a result of \"applying delta\".\n\nThe patch format I'm getting from mercurial boils down to:\n  - replace N bytes from offset in the source material with the given\n    M bytes.\nWhich can easily be converted to xdelta without looking at the original.\n\nMike\n"},{"id":"238283","messageId":"xmqqlhvoy92m.fsf@gitster.dls.corp.google.com","threadId":"36329","inReplyTo":"20140401221003.GA5923@glandium.org","subject":"Re: fast-import deltas","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-01T22:32:49Z","receivedAt":"2014-04-01T22:32:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mike Hommey <mh@glandium.org> writes:\n\n> On Tue, Apr 01, 2014 at 10:14:02AM -0700, Junio C Hamano wrote:\n> ...\n>> Unless you already have your change in the xdelta on hand, or the\n>> format your foreign change is in gives sufficient information to\n>> produce a corresponding xdelta without looking at the content that\n>> your foreign change applies to, it is silly to try to convert your\n>> foreign change into xdelta and feed it to fast-import.\n>\n> The patch format I'm getting from mercurial boils down to:\n>   - replace N bytes from offset in the source material with the given\n>     M bytes.\n> Which can easily be converted to xdelta without looking at the original.\n\nIf that is the case, and if you can identify the original in a way\nfast-import can understand, it might be interesting [*1*] to add support\nfor accepting <base object, xdelta> pair in place of blob data, as\nJonathan already hinted earlier.\n\nIt would not be sufficient for the receiving fast-import to store\nthe delta data to its output---it needs to make sure that the base\nobject is stored in the same output file as well, but that should\nnot be too complicated.\n\n\n[Footnote]\n\n*1* I am still not sure how useful the feature would be outside\nslurping from Hg and Git---for obvious reasons, though.  As long as\nthe change is to a cleanly isolated codepath, it would be OK, I\nguess.\n"},{"id":"238285","messageId":"20140401231235.GA8422@glandium.org","threadId":"36329","inReplyTo":"xmqqlhvoy92m.fsf@gitster.dls.corp.google.com","subject":"Re: fast-import deltas","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-04-01T23:12:35Z","receivedAt":"2014-04-01T23:12:35Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Apr 01, 2014 at 03:32:49PM -0700, Junio C Hamano wrote:\n> [Footnote]\n> \n> *1* I am still not sure how useful the feature would be outside\n> slurping from Hg and Git---for obvious reasons, though.  As long as\n> the change is to a cleanly isolated codepath, it would be OK, I\n> guess.\n\nThat's why I started the thread by asking if there would be some\ninterest for this feature. I'm not even sure it would be entirely\nbeneficial to my usecase, just a hunch.\n\nMike\n"},{"id":"238284","messageId":"A0BF7D05-E351-4A5B-8F0F-DD0FAD391656@quendi.de","threadId":"36329","inReplyTo":"20140401131512.GA19321@sigill.intra.peff.net","subject":"Re: fast-import deltas","fromName":"Max Horn","fromEmail":"max@quendi.de","sentAt":"2014-04-01T23:29:13Z","receivedAt":"2014-04-01T23:29:13Z","isPatch":false,"sender":{"key":"max@quendi.de","avatar":"https://avatars.githubusercontent.com/u/241512?v=4"},"body":"\nOn 01.04.2014, at 15:15, Jeff King <peff@peff.net> wrote:\n\n> On Tue, Apr 01, 2014 at 10:07:03PM +0900, Mike Hommey wrote:\n> \n>>> For my own curiosity, how does this differ from what is in\n>>> contrib/remote-helpers/git-remote-hg?\n>> \n>> contrib/remote-helpers/git-remote-hg does a local mercurial clone before\n>> doing the git conversion. While this is not really a problem for most\n>> mercurial projects, it tends to be slow with big ones, like the firefox\n>> source code. What I'm aiming at is something that can talk directly to a\n>> remote mercurial server.\n> \n> Ah, that makes sense. Thanks for explaining.\n\n\nHm, myself, I am not quite convinced. Yes, there is an overhead, but it is one-time (well, the space overhead is not, but Mike only mentioned time, not space). I wonder if it is really worth the effort to start yet another project on this... Moreover, I don't see a fundamental reason why one could not modify git-remote-hg to work this way. At least optionally - myself, I would strongly prefer the current way, as translating between git and hg 100% round trip clean is provably impossible [1].\n\nThing is, there are by now more than half a dozen projects of this kind. In my impression, all do the low hanging fruit, some go slightly beyond that, but *none* solves all the tough parts and itty-gritty details...\n\nJust to mention a few of the problems that are usually ignored, even though they have real world impact:\n\n- the concept of Mercurial branches has no counterpart in git, making all kinds of translations hard. As a consequence, many translators ignore hg branches completely (e.g. hg-git -- at least it used to do that, not sure whether that changed) or handle them only partially (e.g. contrib/remote-helpers/git-remote-hg: It does not deal with multiple heads or with closed branches)\n(this can cause sever issues with git-remote-hg, by the way, with dangling refs, which, when pruned by an auto-gc, can wipe your fast-import marks file, causing major pain...)\n\n- in the other direction, git branches most closely correspond to hg bookmarks. But what if a hg repository has both a branch \"foo\" and a bookmark \"foo\"? git-remote-hg partially deals with that (by mapping the hg bookmark \"foo\" to the git branch \"foo\", and mapping the hg branch \"foo\" to the git branch \"branches/foo\"), but this still has issues (besides being annoying for users, it clearly still not avoids ref name conflicts)\n\n- git and hg also allow different characters sets in branch and bookmark names\n\n- in hg you can simultaneously have things called \"foo\" and \"foo/bar\". In git, you can't.\n\nThere is plenty more. Of course, some of this might just be impossible [1] to handle nicely. But I find it kind of sad that everybody seems to prefer to start yet another solution, then leave it as 80%, instead of trying to improve upon existing work :-(.\n\nBy the way, to get back to the speed bottleneck: We found that by far the slowest part in importing large repositories like the Mozilla one was not the initial cloning of the hg repository (althoug that could sometimes take ages) but rather an unfortunate mismatch between the hg and git storage approach. When creating a fast-import stream, the normal way to go about that is to import things commit by commit. But if you do that, then extracting file data from Mercurial and its revlog data format easily can degenerate into the worst case quadratic runtime :-/. Now, if one know that one is going to import the whole repository anyway, one could do better by first exporting all file revisions, generating many blobs and their marks, and keeping these in memory, *then* exporting the commits, reverting to these blob marks. \n\nHowever, this stops being a great idea once you are working in incremental mode. That said, it certainly would make sense to investigate this possibility (regardless of whether one uses a local hg clone or directly talks to the remote repository); at least in theory, even if one only uses this approach during the initial import, it should be a strict improvement over the current situation.\n\nIn closing, I should mention that the problems caused by translating between hg and git concepts are by far not the only ones; the fast-import interface itself still has limitations that make some things annoying. E.g. when a remote is renamed, the remote handler does not know that, which can lead to awkward situations that right now may require some trickery to resolve correctly, if it is possible at all. Or if a user manually removed a commit that a remote-helper previously referenced in a marks file, and that remote helper than uses that marks file, fast-import just dies, complaining about the invalid mark. As a result, every proper remote helper basically would need to fully parse and verify those marks files, detect \"broken\" marks, and deal with that -- there is no way to benefit from the existing mark verification code in fast-import right now.\n\nPlease don't get me wrong. I don't want to whine, and I hope I can contribute to solving some of these issues at some point (though lack of time is a nasty issue). In the meantime, I'd love if other people were interested in improving one of the existing solutions to the problem (such as git-remote-hg, gitifyhg or hg-git), instead of creating yet another half-way solution... :-)\n\n\nCheers,\nMax\n\n[1] That is, unless you are willing to use a custom server, such as Kiln Harmony <http://blog.fogcreek.com/kiln-harmony-internals-the-basics/>. But that is cheating, as this is not a real round-trip conversion; rather, you keep a git and a hg repository in perfect sync all the time and present them as a single entity to the outside world.\n"},{"id":"238299","messageId":"20140402041346.GB15690@glandium.org","threadId":"36329","inReplyTo":"A0BF7D05-E351-4A5B-8F0F-DD0FAD391656@quendi.de","subject":"Re: fast-import deltas","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-04-02T04:13:46Z","receivedAt":"2014-04-02T04:13:46Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Wed, Apr 02, 2014 at 01:29:13AM +0200, Max Horn wrote:\n> \n> On 01.04.2014, at 15:15, Jeff King <peff@peff.net> wrote:\n> \n> > On Tue, Apr 01, 2014 at 10:07:03PM +0900, Mike Hommey wrote:\n> > \n> >>> For my own curiosity, how does this differ from what is in\n> >>> contrib/remote-helpers/git-remote-hg?\n> >> \n> >> contrib/remote-helpers/git-remote-hg does a local mercurial clone\n> >> before doing the git conversion. While this is not really a problem\n> >> for most mercurial projects, it tends to be slow with big ones,\n> >> like the firefox source code. What I'm aiming at is something that\n> >> can talk directly to a remote mercurial server.\n> > \n> > Ah, that makes sense. Thanks for explaining.\n> \n> \n> Hm, myself, I am not quite convinced. Yes, there is an overhead, but\n> it is one-time (well, the space overhead is not, but Mike only\n> mentioned time, not space).\n\nI didn't mention it, but it definitely is a factor. As for the\nperformance factor, it certainly is more than a one-time overhead.\n\n> I wonder if it is really worth the effort\n> to start yet another project on this... Moreover, I don't see a\n> fundamental reason why one could not modify git-remote-hg to work this\n> way.\n\nThe way git-remote-hg works is fundamentally different to how one can\ndirectly get and push stuff to a remote mercurial server. As such, there\nis not much value in the current git-remote-hg code for that purpose.\nAlso, I'm currently prototyping something to see whether what I think\nshould work really works. Starting from the current git-remote-hg code\nwould add useless constraints to the prototype.\n\nMike\n"},{"id":"238589","messageId":"53458714720a6_af197d30861@nysa.notmuch","threadId":"36329","inReplyTo":"20140402041346.GB15690@glandium.org","subject":"Re: fast-import deltas","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2014-04-09T17:44:52Z","receivedAt":"2014-04-09T17:44:52Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Mike Hommey wrote:\n> On Wed, Apr 02, 2014 at 01:29:13AM +0200, Max Horn wrote:\n> > I wonder if it is really worth the effort to start yet another project on\n> > this... Moreover, I don't see a fundamental reason why one could not modify\n> > git-remote-hg to work this way.\n> \n> The way git-remote-hg works is fundamentally different to how one can\n> directly get and push stuff to a remote mercurial server.\n\nThat is why you would modify it; so it does what you want, and instead of using\nremote-helper's import/export, it would use fetch/push.\n\nEither way, I say you should hack Git and do as many changes as you want, and\nonce you have some numbers, it would be clearer what approach should be the\nideal one, how much is the benefit, and then we could discuss if it's worth the\nmodifications in Git needed.\n\n-- \nFelipe Contreras\n"}]}