{"thread":{"id":"33552","subject":"Is there a way to speed up remote-hg?","startedAt":"2013-04-20T11:07:54Z","lastAt":"2013-04-21T12:59:06Z","messageCount":3,"participants":["John Szakmeister","Felipe Contreras"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"214912","messageId":"CAEBDL5XO4oU9QL1=kQ_f8_MM9jHAKQojMQr_6VSZsEYNY7PLpA@mail.gmail.com","threadId":"33552","inReplyTo":null,"subject":"Is there a way to speed up remote-hg?","fromName":"John Szakmeister","fromEmail":"john@szakmeister.net","sentAt":"2013-04-20T11:07:54Z","receivedAt":"2013-04-20T11:07:54Z","isPatch":false,"sender":{"key":"john@szakmeister.net","avatar":"https://avatars.githubusercontent.com/u/448087?v=4"},"body":"I really like the idea of remote-hg, but it appears to be awfully slow\non the clone step:\n\n    ...\n    progress revision 81499 'master' (81500/81664)\n    progress revision 81599 'master' (81600/81664)\n    Checking out files: 100% (3744/3744), done.\n    git clone hg::https://bitbucket.org/python_mirrors/cpython\n4484.61s user 41510.05s system 102% cpu 12:29:45.73 total\n\nThat seems like an awfully high price to pay.  It there a way to speed\nthis up at all?  I realize the Python hg repo has more history than\nothers, but even a smaller project like Sphinx takes a while:\n\n    git clone hg::https://bitbucket.org/birkenfeld/sphinx  56.41s user\n90.86s system 98% cpu 2:28.87 total\n\nI was just curious if something more could be done here.  I don't go\naround cloning Python all the time, so it's not a big issue, but it'd\nbe nice if it was more performant.\n\nThanks!\n\n-John\n"},{"id":"214948","messageId":"CAMP44s3Rh6Ef8aSif39UXgr5tqBCCMj92vo_MFoDOupu3Xj8Hw@mail.gmail.com","threadId":"33552","inReplyTo":"CAEBDL5XO4oU9QL1=kQ_f8_MM9jHAKQojMQr_6VSZsEYNY7PLpA@mail.gmail.com","subject":"Re: Is there a way to speed up remote-hg?","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-04-20T23:07:41Z","receivedAt":"2013-04-20T23:07:41Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Sat, Apr 20, 2013 at 6:07 AM, John Szakmeister <john@szakmeister.net> wrote:\n> I really like the idea of remote-hg, but it appears to be awfully slow\n> on the clone step:\n\nThe short answer is no. I do have a couple of patches that improve\nperformance, but not by a huge factor.\n\nI have profiled the code, and there are two significant places where\nperformance is wasted:\n\n1) Fetching the file contents\n\nExtracting, decompressing, transferring, and then compressing and\nstoring the file contents is mostly unavoidable, unless we already\nhave the contents of such file, which in Git, it would be easy to\ncheck by analyzing the checksum (SHA-1). Unfortunately Mercurial\ndoesn't have that information. The SHA-1 that is stored is not of the\ncontents, but the contents and the parent checksum, which means that\nif you revert a modification you made to a file, or move a file, any\noperation that ends up in the same contents, but from a different\npath, the SHA-1 is different. This means the only way to know if the\ncontents are the same, is by extracting, and calculating the SHA-1\nyourself, which defeats the purpose of what you want the calculation\nfor.\n\nI've tried, calculating the SHA-1 and use a previous reference to\navoid the transfer, or do the transfer, and let Git check for existing\nobjects doesn't make a difference.\n\nThis is by Mercurial's stupid design, and there's nothing we, or\nanybody could do about it until they change it.\n\n2) Checking for file changes\n\nFor each commit (or revision), we need to figure out which files were\nmodified, and for that, Mercurial has a neat shortcut that stores such\nmodifications in the commit context itself, so it's easy to retrieve.\nUnfortunately, it's sometimes wrong.\n\nSince the Mercurial tools never use this information for any real\nwork, simply to show the changes to the users, Mercurial folks never\nnoticed the contents they were storing were wrong. Which means if you\nhave a repository that started with old versions of mercurial, chances\nare this information would be wrong, and there's no real guarantee\nthat future versions won't have this problem, since to this day this\ninformation continues to be used only display stuff to the user.\n\nSo, since we cannot rely on this, we need to manually check for\ndifferences the way Mercurial does, which blows performance away,\nbecause you need to get the contents of the two parent revisions, and\ncompare them away. My content I mean the the manifest, or list of\nfiles, which takes considerable amount of time.\n\nFor 1) there's nothing we can do, and for 2) we could trust the files\nMercurial thinks were modified, and that gives us a very significant\nboost, but the repository will sometimes end up wrong. Most of the\ntime is spent on 2).\n\nSo unfortunately there's nothing we can do, that's just Mercurial\ndesign, and it really has nothing to do with Git. Any other tool would\nhave the same problems, even a tool that converts a Mercurial\nrepository to Mercurial (without using tricks).\n\nIt seems Bazaar is more sensible in this regard; 1) the checksums are\ntry of the file contents, and 2) each revision does store the file\nmodifications correctly. So a clone in Bazaar is much faster. In my\nopinion Mercurial just screwed up their design.\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"215011","messageId":"CAEBDL5UD9Bedxbtp3H+m5J_0Hq0ddVa1OLxXjyX1CX-Dhd+jzA@mail.gmail.com","threadId":"33552","inReplyTo":"CAMP44s3Rh6Ef8aSif39UXgr5tqBCCMj92vo_MFoDOupu3Xj8Hw@mail.gmail.com","subject":"Re: Is there a way to speed up remote-hg?","fromName":"John Szakmeister","fromEmail":"john@szakmeister.net","sentAt":"2013-04-21T12:59:06Z","receivedAt":"2013-04-21T12:59:06Z","isPatch":false,"sender":{"key":"john@szakmeister.net","avatar":"https://avatars.githubusercontent.com/u/448087?v=4"},"body":"On Sat, Apr 20, 2013 at 7:07 PM, Felipe Contreras\n<felipe.contreras@gmail.com> wrote:\n> On Sat, Apr 20, 2013 at 6:07 AM, John Szakmeister <john@szakmeister.net> wrote:\n>> I really like the idea of remote-hg, but it appears to be awfully slow\n>> on the clone step:\n>\n> The short answer is no. I do have a couple of patches that improve\n> performance, but not by a huge factor.\n>\n> I have profiled the code, and there are two significant places where\n> performance is wasted:\n>\n> 1) Fetching the file contents\n>\n> Extracting, decompressing, transferring, and then compressing and\n> storing the file contents is mostly unavoidable, unless we already\n> have the contents of such file, which in Git, it would be easy to\n> check by analyzing the checksum (SHA-1). Unfortunately Mercurial\n> doesn't have that information. The SHA-1 that is stored is not of the\n> contents, but the contents and the parent checksum, which means that\n> if you revert a modification you made to a file, or move a file, any\n> operation that ends up in the same contents, but from a different\n> path, the SHA-1 is different. This means the only way to know if the\n> contents are the same, is by extracting, and calculating the SHA-1\n> yourself, which defeats the purpose of what you want the calculation\n> for.\n>\n> I've tried, calculating the SHA-1 and use a previous reference to\n> avoid the transfer, or do the transfer, and let Git check for existing\n> objects doesn't make a difference.\n>\n> This is by Mercurial's stupid design, and there's nothing we, or\n> anybody could do about it until they change it.\n\nThat's a bummer. :-(\n\n> 2) Checking for file changes\n>\n> For each commit (or revision), we need to figure out which files were\n> modified, and for that, Mercurial has a neat shortcut that stores such\n> modifications in the commit context itself, so it's easy to retrieve.\n> Unfortunately, it's sometimes wrong.\n>\n> Since the Mercurial tools never use this information for any real\n> work, simply to show the changes to the users, Mercurial folks never\n> noticed the contents they were storing were wrong. Which means if you\n> have a repository that started with old versions of mercurial, chances\n> are this information would be wrong, and there's no real guarantee\n> that future versions won't have this problem, since to this day this\n> information continues to be used only display stuff to the user.\n>\n> So, since we cannot rely on this, we need to manually check for\n> differences the way Mercurial does, which blows performance away,\n> because you need to get the contents of the two parent revisions, and\n> compare them away. My content I mean the the manifest, or list of\n> files, which takes considerable amount of time.\n\nEek!\n\n> For 1) there's nothing we can do, and for 2) we could trust the files\n> Mercurial thinks were modified, and that gives us a very significant\n> boost, but the repository will sometimes end up wrong. Most of the\n> time is spent on 2).\n>\n> So unfortunately there's nothing we can do, that's just Mercurial\n> design, and it really has nothing to do with Git. Any other tool would\n> have the same problems, even a tool that converts a Mercurial\n> repository to Mercurial (without using tricks).\n[snip]\n\nThat's unfortunate, but thank you for taking the time to explain!\n\n-John\n"}]}