{"thread":{"id":"31065","subject":"GSOC remote-svn","startedAt":"2012-07-22T21:03:27Z","lastAt":"2012-07-25T06:20:10Z","messageCount":21,"participants":["Florian Achleitner","Jonathan Nieder","Matthieu Moy","Junio C Hamano","Jeff King","Erik Faye-Lund"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"195458","messageId":"4514544.Xip1OCQ7Uj@flomedio","threadId":"31065","inReplyTo":null,"subject":"GSOC remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-07-22T21:03:27Z","receivedAt":"2012-07-22T21:03:27Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi!\n\nRefering to Jonathan's concerns in Saturday night's IRC log:\n\n> [22:59:34] barrbrain, flyingflo: I'm worried about the remote \nhelper project\n> [23:00:05] someone needs to review remote-svn.c to catch things \nlike that refspec issue which should be straightforward to an experienced eye\n\nLet me explain the refspec issue:\n\nIn the the exisiting code in contrib/svn-fe commits are always imported to \nrefs/heads/master, that was hardcoded. So I thought that couldn't be it.\nI made the name of the branch to import variable, depending on the name of \nthe remote.\nBut my remote-helper didn't advertise the refspec capability, so transport-\nhelper assumed it imports to refs/heads/master, which is the default import \nrefspec. The subsequent update of references in store_updated_refs lead to \nwrong values after the fetch, which I considered a bug and tried to fix.\n\nIn fact I didn't realize that the actual updating of references is not done by \nthe remote-helper. I thought the remote-helper would have to evaluate the \nfetch refspec and tell fast-import the correct target branch.\nFurthermore I confused 'private namespace' with refs/remotes//, \nwhich I considered somehow private too.\n\nAfter several mailing iterations, showing me that I was wrong, I found what \nthe right point is, namely that the remote helper writes references to a \nreally private dir in refs//, it doesn't touch anything else, and \nby advertising the 'refspec' capability, git-fetch knows where the private \nrefs are and updates non-private references according to the fetch refspec in \nsome post-processing in store_updated_refs. (Ok, you will say \"of course!\", \nbut I didn't know that I was wrong and it's hidden in some 1000 lines of \ncode).\n\nFor me that was not very easy to figure out, and it took a lot of time, but I \nthink now remote-svn does it right.\n\n> [23:00:38] (also, remote-svn.c should be at the toplevel so it \ncan be tested more easily with tests in t/\n> [23:01:10] and it should not be named remote-svn, since we \nhaven't pinned down details about the svn:: conversion yet. That's why \nDmitry's was called git-remote-svn-alpha)\n\nOk. Why is that important? I think if it's not called remote-svn git doesn't \nfind it as a helper for the 'svn' protocol. Actually in my local git tree, I \nhave a symlink in the toplevel (to simpify PATH).\n\n> [23:01:45] I'm happy to review patches but I don't have a lot of \ntime for it, which has been a problem:\n> [23:02:11] * I think I wasn't cc-ed on earlier discussion so \nthey seem to come out of the blue. That's fine, but\n> [23:03:05] * I really rely on patches that do one logical thing \nwith a commit message describing the context and what the patch is trying to \naccomplish. That makes review way, way easier when it is happening.\n\nProbably I should stop sending proposals or incomplete stuff to the list/you.\nThe current state may probably be viewed easier in my github repo.\n\nI think for creating patches that are acceptable I will need to squash and \nsplit a lot of my development commits after the code is somehow finished and \nno longer experimental.\n\n> [23:04:42] Also it seems very chaotic: there are basic things \nabout remote-svn.c that need fixing, and then patches for other things are \nappearing on top of that.\n> [23:04:49] Help?\n> [23:05:26] thanks, and hope that helps\n\nAbout the current state:\n\nTester:\nI wrote a small simulation script in python that mimics svnrdumps behaviour by \nreplaying an existing svn dump file from a start rev up to an end rev to test \nincremental imports. I use it together with a little testrepo shell script.\nWill need to bring that into t/ later, after figuring out how the test \nframework works. As it's not finished it's not published.\n\nIncremental import:\nBy reading the latest svn revision number from a note attached to the private \nmaster ref, it starts future imports from the next svn revision. That \nbasically works well.\nIt doesn't reuse mark files. What's the point of reusing them? Dmitry's svn-\nalpha did that.\nAll I need to know is the revision to start from and the branch i want to add \ncommits to, right? It now simply reads that from the note.\n\nThis got stuck on another problem:\nIncremental update of the note tree doesn't work. fast-import refuses to \nupdate the notes tree: ' doesn't contain '.\nI don't yet know what's the reason for this.\nI'm digging into the internals of notes to find out why..\n(no problem with the file tree).\n\nThis state hasn't hit the list of course, as it's in no way useful nor \ncomplete.\n\nI often get caught in the traingle of those three processes (git transport-\nhelper, fast-import, remote-svn) needing to understand a lot about the \nexisting two to understand why things don't work and why they need to work \nlike they do.\n\n--\nFlorian\n"},{"id":"195464","messageId":"20120722214333.GB680@burratino","threadId":"31065","inReplyTo":"4514544.Xip1OCQ7Uj@flomedio","subject":"Re: GSOC remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-07-22T21:43:33Z","receivedAt":"2012-07-22T21:43:33Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nFlorian Achleitner wrote:\n\n> After several mailing iterations, showing me that I was wrong, I found what \n> the right point is, namely that the remote helper writes references to a \n> really private dir in refs//, it doesn't touch anything else, and \n> by advertising the 'refspec' capability, git-fetch knows where the private \n> refs are and updates non-private references according to the fetch refspec in \n> some post-processing in store_updated_refs.\n\nRight, that's fine. And you did a fine job of navigating the existing\ndocumentation (which could be improved, hint hint).\n\nWhat I am more concerned about is that you had code you sent a while\nago for review, that this would have been obvious to David, Ram,\nDmitry, or me if we had seen it, and yet none of us gave you that\nhelp. We are failing at _our_ job of giving you prompt advice and\ninstead you have had to work on your own.\n\nIsn't it likely that there are multiple other bugs like that which\nstill haven't been fixed?\n\nThat's why I think we need to get into a habit of giving and getting\nfeedback quickly and incrementally improving work. Soon, before the\nsummer ends.\n\n[...]\n>> [23:01:10] and it should not be named remote-svn, since we \n> haven't pinned down details about the svn:: conversion yet. That's why \n> Dmitry's was called git-remote-svn-alpha)\n>\n> Ok. Why is that important? I think if it's not called remote-svn git doesn't \n> fid it as a helper for the 'svn' protocol.\n\nIt finds it as a helper for the 'svn-alpha' protocol instead.\n\nThe point is that when I perform the following steps:\n\n\tgit clone svn://path/to/remote/repo\n\n\t... wait a day, update git\n\n\tcd repo\n\tgit pull\n\nnobody would expect the result to be a non-fast-forward update caused\nby the details of svn-to-git conversion changing. Using a name like\ntestsvn or svn-alpha would help in managing expectations --- the\nremote helper is meant for experimentation for now and not meant to be\nsomething people can rely on for collaboration.\n\n[...]\n>> [23:03:05] * I really rely on patches that do one logical thing \n> with a commit message describing the context and what the patch is trying to \n> accomplish. That makes review way, way easier when it is happening.\n>\n> Probably I should stop sending proposals or incomplete stuff to the list/you.\n> The current state may probably be viewed easier in my github repo.\n\nNo, incomplete stuff is nice. Just please do explain the _purpose_ of\nthe code you are sending out. The best possible outcome is if someone\nrealizes that something that would have taken hours doesn't need to be\ndone at all.\n\n[...]\n> I wrote a small simulation script in python that mimics svnrdumps behaviour by \n> replaying an existing svn dump file from a start rev up to an end rev to test \n> incremental imports. I use it together with a little testrepo shell script.\n> Will need to bring that into t/ later, after figuring out how the test \n> framework works. As it's not finished it's not published.\n\nSounds neat --- how can one try it out?\n\n> Incremental import:\n> By reading the latest svn revision number from a note attached to the private \n> master ref, it starts future imports from the next svn revision. That \n> basically works well.\n> It doesn't reuse mark files. What's the point of reusing them? Dmitry's svn-\n> alpha did that.\n> All I need to know is the revision to start from and the branch i want to add \n> commits to, right? It now simply reads that from the note.\n\nThe marks are used to handle copyfrom operations referring to older\nrevisions. Are you sure you want to abandon them? Can you explain a\nlittle more about your plan?\n\n[...]\n> This state hasn't hit the list of course, as it's in no way useful nor \n> complete.\n\nA good habit to get into is to make sure your partial progress toward\na goal is in a usable state periodically, even if it is not complete.\nThat makes it a lot easier to test and to get other people to look it\nover.\n\nA rule of thumb is that unless you are adding a new and complicated\nfeature, it should be possible to fit each change in a patch of around\n250 lines including context (not including documentation and tests).\n\n> I often get caught in the traingle of those three processes (git transport-\n> helper, fast-import, remote-svn) needing to understand a lot about the \n> existing two to understand why things don't work and why they need to work \n> like they do.\n\nProbably you are finding documentation bugs right and left (yes,\ninformation not being easy to find is a bug) that don't get fixed\nbecause no one has reported them. Questions are welcome and very\nuseful.\n\nHope that helps,\nJonathan\n"},{"id":"195516","messageId":"vpqr4s2nc0j.fsf@bauges.imag.fr","threadId":"31065","inReplyTo":"4514544.Xip1OCQ7Uj@flomedio","subject":"Re: GSOC remote-svn","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-07-23T07:59:40Z","receivedAt":"2012-07-23T07:59:40Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Florian Achleitner writes:\n\n> But my remote-helper didn't advertise the refspec capability, so transport-\n> helper assumed it imports to refs/heads/master, which is the default import \n> refspec.\n\nThe man page for git-remote-helpers says:\n\nrefspec \n [...] It is recommended that all importers providing the import\n capability use this.\n\nI'm not sure I fully understand the rationale, but one difference\nbetween refs//* and refs/remotes/ is that\nrefs/remotes/ is automatically updated by Git on push, while the private\nnamespace isn't (it only receives updates when importing).\n\nI played a bit with that it git-remote-mediawiki. There's a\nconfiguration variable mediawiki.dumbPush that controls what \"push\"\ndoes. It can either export the local history without touching local\nmetadata (and then, it is expected that the user does a \"git pull\n--rebase\" right after to re-import the history), or update the local\nmetadata (private ref, and notes that keep the <->\n map), so that the next \"git pull\" says\n\"Already up to date.\".\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"195519","messageId":"2948040.5ceLh0WG3L@flomedio","threadId":"31065","inReplyTo":"20120722214333.GB680@burratino","subject":"Re: GSOC remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-07-23T09:42:17Z","receivedAt":"2012-07-23T09:42:17Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Sunday 22 July 2012 16:43:33 Jonathan Nieder wrote:\n> Hi,\n> \n> Florian Achleitner wrote:\n> > After several mailing iterations, showing me that I was wrong, I found\n> > what\n> > the right point is, namely that the remote helper writes references to a\n> > really private dir in refs//, it doesn't touch anything else,\n> > and by advertising the 'refspec' capability, git-fetch knows where the\n> > private refs are and updates non-private references according to the\n> > fetch refspec in some post-processing in store_updated_refs.\n> \n> Right, that's fine. And you did a fine job of navigating the existing\n> documentation (which could be improved, hint hint).\n> \n> What I am more concerned about is that you had code you sent a while\n> ago for review, that this would have been obvious to David, Ram,\n> Dmitry, or me if we had seen it, and yet none of us gave you that\n> help. We are failing at _our_ job of giving you prompt advice and\n> instead you have had to work on your own.\n> \n> Isn't it likely that there are multiple other bugs like that which\n> still haven't been fixed?\n> \n> That's why I think we need to get into a habit of giving and getting\n> feedback quickly and incrementally improving work. Soon, before the\n> summer ends.\n> \n> [...]\n> \n> >> [23:01:10] and it should not be named remote-svn, since we\n> > \n> > haven't pinned down details about the svn:: conversion yet. That's why\n> > Dmitry's was called git-remote-svn-alpha)\n> > \n> > Ok. Why is that important? I think if it's not called remote-svn git\n> > doesn't fid it as a helper for the 'svn' protocol.\n> \n> It finds it as a helper for the 'svn-alpha' protocol instead.\n> \n> The point is that when I perform the following steps:\n> \n> \tgit clone svn://path/to/remote/repo\n> \n> \t... wait a day, update git\n> \n> \tcd repo\n> \tgit pull\n> \n> nobody would expect the result to be a non-fast-forward update caused\n> by the details of svn-to-git conversion changing. Using a name like\n> testsvn or svn-alpha would help in managing expectations --- the\n> remote helper is meant for experimentation for now and not meant to be\n> something people can rely on for collaboration.\n\nOk, that makes sense. Renaming is easily done!\n\n> \n> > I wrote a small simulation script in python that mimics svnrdumps\n> > behaviour by replaying an existing svn dump file from a start rev up to\n> > an end rev to test incremental imports. I use it together with a little\n> > testrepo shell script. Will need to bring that into t/ later, after\n> > figuring out how the test framework works. As it's not finished it's not\n> > published.\n> \n> Sounds neat --- how can one try it out?\n\nI'll send a patch ...\n\n> \n> > Incremental import:\n> > By reading the latest svn revision number from a note attached to the\n> > private master ref, it starts future imports from the next svn revision.\n> > That basically works well.\n> > It doesn't reuse mark files. What's the point of reusing them? Dmitry's\n> > svn- alpha did that.\n> > All I need to know is the revision to start from and the branch i want to\n> > add commits to, right? It now simply reads that from the note.\n> \n> The marks are used to handle copyfrom operations referring to older\n> revisions. Are you sure you want to abandon them? Can you explain a\n> little more about your plan?\n\nOk, that makes sense. I didn't need the marks for incremental import. But to \nevaluate the copyfrom props we need some revision->sha1 mapping.\nI just added the options to save and import marks to fast-import's command \nline. \nIf the file is missing, it will need to be generated from the notes, or the \nwhole history will be reimported.\n\nBut when I fetch from a git repo that imported from svn, the notes are not \nfetched automatically. In this case I currently loose marks and notes.\nWhat can I do?\n\nFlorian\n"},{"id":"195520","messageId":"1571752.59fsElNHEJ@flomedio","threadId":"31065","inReplyTo":"4514544.Xip1OCQ7Uj@flomedio","subject":"Re: GSOC remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-07-23T09:42:56Z","receivedAt":"2012-07-23T09:42:56Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Sunday 22 July 2012 23:03:27 Florian Achleitner wrote:\n> This got stuck on another problem:\n> Incremental update of the note tree doesn't work. fast-import refuses to \n> update the notes tree: ' doesn't contain '.\n> I don't yet know what's the reason for this.\n> I'm digging into the internals of notes to find out why..\n> (no problem with the file tree).\n\nbtw, this one was rather simple, just a syntax error in the fast-import \nstream..\n"},{"id":"195521","messageId":"20120723115902.GD14294@burratino","threadId":"31065","inReplyTo":"vpqr4s2nc0j.fsf@bauges.imag.fr","subject":"Re: GSOC remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-07-23T11:59:03Z","receivedAt":"2012-07-23T11:59:03Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Matthieu Moy wrote:\n\n> The man page for git-remote-helpers says:\n>\n> refspec \n> [...] It is recommended that all importers providing the import\n> capability use this.\n>\n> I'm not sure I fully understand the rationale, but one difference\n> between refs//* and refs/remotes/ is that\n> refs/remotes/ is automatically updated by Git on push, while the private\n> namespace isn't (it only receives updates when importing).\n\nIt's mostly to allow \"git fetch\" to avoid non fast-forward updates\nunless -f was used or the refspec starts with +.\n\nI always liked the idea of tweaking the fast-import stream format to\nallow the import to happen on no branch at all since it would avoid\nall these questions. Maybe another day.\n\nThanks,\nJonathan\n"},{"id":"195522","messageId":"20120723120428.GE14294@burratino","threadId":"31065","inReplyTo":"2948040.5ceLh0WG3L@flomedio","subject":"Re: GSOC remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-07-23T12:04:28Z","receivedAt":"2012-07-23T12:04:28Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Florian Achleitner wrote:\n\n> But when I fetch from a git repo that imported from svn, the notes are not \n> fetched automatically. In this case I currently loose marks and notes.\n> What can I do?\n\nIn the long term, git will need to be tweaked to automatically fetch\nnotes along with branches by default. There are other reasons not\nrelated to remote helpers to want this, too.\n\nIn the short term, we can document which ref the notes are expected to\nbe fetched to. Maybe someone interested would provide commands like\n\"git remote-testsvn --clone \" and \"git remote-testsvn\n--add-remote \" as a stopgap to set up the appropriate\nfetch refspec automatically so the user doesn't have to worry about\nit.\n\nThanks,\nJonathan\n"},{"id":"195523","messageId":"1448476.VR1Gla8Cvg@flomedio","threadId":"31065","inReplyTo":"2948040.5ceLh0WG3L@flomedio","subject":"[PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-07-23T12:44:14Z","receivedAt":"2012-07-23T12:44:14Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"To ease testing without depending on a reachable svn server, this\ncompact python script mimics parts of svnrdumps behaviour.\nIt requires the remote url to start with sim://.\nStart and end revisions are evaluated.\nIf the requested revision doesn't exist, as it is the case with\nincremental imports, if no new commit was added, it returns 1\n(like svnrdump).\nTo allow using the same dump file for simulating multiple\nincremental imports the highest revision can be limited by setting\nthe environment variable SVNRMAX to that value. This simulates the\nsituation where higher revs don't exist yet.\n---\n contrib/svn-fe/svnrdump_sim.py | 53 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n create mode 100755 contrib/svn-fe/svnrdump_sim.py\n\ndiff --git a/contrib/svn-fe/svnrdump_sim.py b/contrib/svn-fe/svnrdump_sim.py\nnew file mode 100755\nindex 0000000..4701d76\n--- /dev/null\n+++ b/contrib/svn-fe/svnrdump_sim.py\n@@ -0,0 +1,53 @@\n+#!/usr/bin/python\n+\"\"\"\n+Simulates svnrdump by replaying an existing dump from a file, taking care\n+of the specified revision range.\n+To simulate incremental imports the environment variable SVNRMAX can be set\n+to the highest revision that should be available.\n+\"\"\"\n+import sys, os\n+\n+\n+def getrevlimit():\n+\tvar = 'SVNRMAX'\n+\tif os.environ.has_key(var):\n+\t\treturn os.environ[var]\n+\treturn None\n+\t\n+def writedump(url, lower, upper):\n+\tif url.startswith('sim://'):\n+\t\tfilename = url[6:]\n+\t\tif filename[-1] == '/': filename = filename[:-1] #remove terminating slash\n+\telse:\n+\t\traise ValueError('sim:// url required')\n+\tf = open(filename, 'r');\n+\tstate = 'header'\n+\twroterev = False\n+\twhile(True):\n+\t\tl = f.readline()\n+\t\tif l == '': break\n+\t\tif state == 'header' and l.startswith('Revision-number: '):\n+\t\t\tstate = 'prefix'\n+\t\tif state == 'prefix' and l == 'Revision-number: %s\\n' % lower:\n+\t\t\tstate = 'selection'\n+\t\tif not upper == 'HEAD' and state == 'selection' and l == 'Revision-number: %s\\n' % upper:\n+\t\t\tbreak;\n+\n+\t\tif state == 'header' or state == 'selection':\n+\t\t\tif state == 'selection': wroterev = True\n+\t\t\tsys.stdout.write(l)\n+\treturn wroterev\n+\n+if __name__ == \"__main__\":\n+\tif not (len(sys.argv) in (3, 4, 5)):\n+\t\tprint \"usage: %s dump URL -rLOWER:UPPER\"\n+\t\tsys.exit(1)\n+\tif not sys.argv[1] == 'dump': raise NotImplementedError('only \"dump\" is suppported.')\n+\turl = sys.argv[2]\n+\tr = ('0', 'HEAD')\n+\tif len(sys.argv) == 4 and sys.argv[3][0:2] == '-r':\n+\t\tr = sys.argv[3][2:].lstrip().split(':')\n+\tif not getrevlimit() is None: r[1] = getrevlimit()\n+\tif writedump(url, r[0], r[1]): ret = 0\n+\telse: ret = 1\n+\tsys.exit(ret)\n\\ No newline at end of file\n-- \n1.7.9.5\n"},{"id":"195524","messageId":"20120723125921.GA16768@burratino","threadId":"31065","inReplyTo":"1448476.VR1Gla8Cvg@flomedio","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-07-23T12:59:21Z","receivedAt":"2012-07-23T12:59:21Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Florian Achleitner wrote:\n\n> To ease testing without depending on a reachable svn server, this\n> compact python script mimics parts of svnrdumps behaviour.\n\nThanks. Mind if I forge your sign-off?\n"},{"id":"195525","messageId":"14140147.99YetxyXPJ@flomedio","threadId":"31065","inReplyTo":"20120723125921.GA16768@burratino","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-07-23T13:16:50Z","receivedAt":"2012-07-23T13:16:50Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Monday 23 July 2012 07:59:21 Jonathan Nieder wrote:\n> Florian Achleitner wrote:\n> > To ease testing without depending on a reachable svn server, this\n> > compact python script mimics parts of svnrdumps behaviour.\n> \n> Thanks. Mind if I forge your sign-off?\n\nUps. No problem, anyways I've added it locally, so here's the new version ..\n"},{"id":"195526","messageId":"5998541.c9PWeIAsEV@flomedio","threadId":"31065","inReplyTo":"20120723125921.GA16768@burratino","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-07-23T13:16:57Z","receivedAt":"2012-07-23T13:16:57Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"To ease testing without depending on a reachable svn server, this\ncompact python script mimics parts of svnrdumps behaviour.\nIt requires the remote url to start with sim://.\nStart and end revisions are evaluated.\nIf the requested revision doesn't exist, as it is the case with\nincremental imports, if no new commit was added, it returns 1\n(like svnrdump).\nTo allow using the same dump file for simulating multiple\nincremental imports the highest revision can be limited by setting\nthe environment variable SVNRMAX to that value. This simulates the\nsituation where higher revs don't exist yet.\n\nSigned-off-by: Florian Achleitner \n---\n\nI had to fix the missing sign-off anyways..\n\n contrib/svn-fe/svnrdump_sim.py | 53 \n++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n create mode 100755 contrib/svn-fe/svnrdump_sim.py\n\ndiff --git a/contrib/svn-fe/svnrdump_sim.py b/contrib/svn-fe/svnrdump_sim.py\nnew file mode 100755\nindex 0000000..4701d76\n--- /dev/null\n+++ b/contrib/svn-fe/svnrdump_sim.py\n@@ -0,0 +1,53 @@\n+#!/usr/bin/python\n+\"\"\"\n+Simulates svnrdump by replaying an existing dump from a file, taking care\n+of the specified revision range.\n+To simulate incremental imports the environment variable SVNRMAX can be set\n+to the highest revision that should be available.\n+\"\"\"\n+import sys, os\n+\n+\n+def getrevlimit():\n+\tvar = 'SVNRMAX'\n+\tif os.environ.has_key(var):\n+\t\treturn os.environ[var]\n+\treturn None\n+\t\n+def writedump(url, lower, upper):\n+\tif url.startswith('sim://'):\n+\t\tfilename = url[6:]\n+\t\tif filename[-1] == '/': filename = filename[:-1] #remove terminating slash\n+\telse:\n+\t\traise ValueError('sim:// url required')\n+\tf = open(filename, 'r');\n+\tstate = 'header'\n+\twroterev = False\n+\twhile(True):\n+\t\tl = f.readline()\n+\t\tif l == '': break\n+\t\tif state == 'header' and l.startswith('Revision-number: '):\n+\t\t\tstate = 'prefix'\n+\t\tif state == 'prefix' and l == 'Revision-number: %s\\n' % lower:\n+\t\t\tstate = 'selection'\n+\t\tif not upper == 'HEAD' and state == 'selection' and l == 'Revision-\nnumber: %s\\n' % upper:\n+\t\t\tbreak;\n+\n+\t\tif state == 'header' or state == 'selection':\n+\t\t\tif state == 'selection': wroterev = True\n+\t\t\tsys.stdout.write(l)\n+\treturn wroterev\n+\n+if __name__ == \"__main__\":\n+\tif not (len(sys.argv) in (3, 4, 5)):\n+\t\tprint \"usage: %s dump URL -rLOWER:UPPER\"\n+\t\tsys.exit(1)\n+\tif not sys.argv[1] == 'dump': raise NotImplementedError('only \"dump\" is \nsuppported.')\n+\turl = sys.argv[2]\n+\tr = ('0', 'HEAD')\n+\tif len(sys.argv) == 4 and sys.argv[3][0:2] == '-r':\n+\t\tr = sys.argv[3][2:].lstrip().split(':')\n+\tif not getrevlimit() is None: r[1] = getrevlimit()\n+\tif writedump(url, r[0], r[1]): ret = 0\n+\telse: ret = 1\n+\tsys.exit(ret)\n\\ No newline at end of file\n-- \n1.7.9.5\n"},{"id":"195527","messageId":"7v4noyo6tc.fsf@alter.siamese.dyndns.org","threadId":"31065","inReplyTo":"1448476.VR1Gla8Cvg@flomedio","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-23T15:06:39Z","receivedAt":"2012-07-23T15:06:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> It requires the remote url to start with sim://.\n> Start and end revisions are evaluated.\n\nIt is a bit unclear where \"start\" and \"end\" comes from, and if\n\"evaluated\" is the most important aspect of the handling of these\ntwo values. Do you mean the tool takes start and end revisions as\narguments? If so, describe \"how\". E.g. as two arguments (-rSTART\n-rEND)? As an argument that shows a range (-rSTART-END? -rSTART,END)?\n\nDo not answer with \"It is in the code\" (I cheated and peeked to find\nout it is -rSTART:END, but the reader should not have to peek).\n\n> If the requested revision doesn't exist, as it is the case with\n> incremental imports, if no new commit was added, it returns 1\n> (like svnrdump).\n\nThis sentence does not parse for me. What is it trying to say?\nRequested revision does not exist _where_? It is unclear how\n\"incremental import\" and \"revision doesn't exist\" are related. \"no\nnew commit was added\" to _what_ by _whom_? I presume that nobody is\nadding new commit _to_ an existing dump file, and the only thing\nthis script does is to read and selectively write part of a dump file,\nso that would not add any new commit either.\n\nPuzzled.\n"},{"id":"195528","messageId":"vpqr4s2jvhz.fsf@bauges.imag.fr","threadId":"31065","inReplyTo":"5998541.c9PWeIAsEV@flomedio","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-07-23T16:24:40Z","receivedAt":"2012-07-23T16:24:40Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Florian Achleitner writes:\n\n> I had to fix the missing sign-off anyways..\n>\n> contrib/svn-fe/svnrdump_sim.py | 53 \n> ++++++++++++++++++++++++++++++++++++++++\n\nYou also have whitespace damages (i.e. line wrapping introduced by your\nmailer). Using git-send-email avoids this kind of problem (there are\nalso some advices for some mailers in Documentation/SubmittingPatches).\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"195557","messageId":"1406203.FoHCDc6FfD@flomedio","threadId":"31065","inReplyTo":"vpqr4s2jvhz.fsf@bauges.imag.fr","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-07-23T19:28:45Z","receivedAt":"2012-07-23T19:28:45Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Monday 23 July 2012 18:24:40 Matthieu Moy wrote:\n> You also have whitespace damages (i.e. line wrapping introduced by your\n> mailer). Using git-send-email avoids this kind of problem (there are\n> also some advices for some mailers in Documentation/SubmittingPatches).\n\nDamn. That's usually no problem with kmail either, if the config is right.\nI've already used git-send-email several times.\nBut for replying to threads and adding several Cc: addresses it's a little \ncumbersome.\nHow do you do that in a nice way?\n\n--\nFlorian\n"},{"id":"195559","messageId":"vpqwr1ugt06.fsf@bauges.imag.fr","threadId":"31065","inReplyTo":"1406203.FoHCDc6FfD@flomedio","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-07-23T19:46:49Z","receivedAt":"2012-07-23T19:46:49Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Florian Achleitner writes:\n\n> On Monday 23 July 2012 18:24:40 Matthieu Moy wrote:\n>> You also have whitespace damages (i.e. line wrapping introduced by your\n>> mailer). Using git-send-email avoids this kind of problem (there are\n>> also some advices for some mailers in Documentation/SubmittingPatches).\n>\n> Damn. That's usually no problem with kmail either, if the config is right.\n> I've already used git-send-email several times.\n> But for replying to threads and adding several Cc: addresses it's a little \n> cumbersome.\n> How do you do that in a nice way?\n\nFor the threading itself, I usually find the message-id, and use\n\"git send-email --in-reply-to=''\". The painful part\nis when you want to reproduce a Cc: list, but I have no magic trick for\nthat ;-).\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"195561","messageId":"20120723200214.GA6277@sigill.intra.peff.net","threadId":"31065","inReplyTo":"vpqwr1ugt06.fsf@bauges.imag.fr","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-07-23T20:02:14Z","receivedAt":"2012-07-23T20:02:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 23, 2012 at 09:46:49PM +0200, Matthieu Moy wrote:\n\n> > Damn. That's usually no problem with kmail either, if the config is right.\n> > I've already used git-send-email several times.\n> > But for replying to threads and adding several Cc: addresses it's a little \n> > cumbersome.\n> > How do you do that in a nice way?\n> \n> For the threading itself, I usually find the message-id, and use\n> \"git send-email --in-reply-to=''\". The painful part\n> is when you want to reproduce a Cc: list, but I have no magic trick for\n> that ;-).\n\nI save a copy of the message I am replying to (usually my cover letter,\nwhich I generated by just replying in my MUA) into a well-known location\n(I use a mutt hot-key to do this), and then run it through this\nmonstrosity:\n\n get_reply_headers() {\n perl -ne '\n if (defined $opt && /^\\s+(.*)/) {\n $val .= \" $1\";\n next;\n }\n if (defined $opt) {\n print \"--$opt=\", quotemeta($val), \" \";\n $opt = $val = undef;\n }\n if (/^(cc|to):\\s*(.*)/i) {\n $opt = lc($1);\n $val = $2;\n }\n elsif (/^message-id:\\s*(.*)/i) {\n $opt = \"in-reply-to\";\n $val = $1;\n }\n '\n }\n\nwhich can be used on the command line of format-patch or send-email, like:\n\n eval \"git format-patch $(get_reply_headers HEAD, i.e. the\nrequested start revision doesn't exist on the server.\n\nTo allow using the same dump file for simulating multiple\nincremental imports the highest visible revision can be limited by\nsetting the environment variable SVNRMAX to that value. This\neffectively limits HEAD to simulate the situation where higher\nrevs don't exist yet.\n\nSigned-off-by: Florian Achleitner \n---\n contrib/svn-fe/svnrdump_sim.py | 53 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n create mode 100755 contrib/svn-fe/svnrdump_sim.py\n\ndiff --git a/contrib/svn-fe/svnrdump_sim.py b/contrib/svn-fe/svnrdump_sim.py\nnew file mode 100755\nindex 0000000..4701d76\n--- /dev/null\n+++ b/contrib/svn-fe/svnrdump_sim.py\n@@ -0,0 +1,53 @@\n+#!/usr/bin/python\n+\"\"\"\n+Simulates svnrdump by replaying an existing dump from a file, taking care\n+of the specified revision range.\n+To simulate incremental imports the environment variable SVNRMAX can be set\n+to the highest revision that should be available.\n+\"\"\"\n+import sys, os\n+\n+\n+def getrevlimit():\n+\tvar = 'SVNRMAX'\n+\tif os.environ.has_key(var):\n+\t\treturn os.environ[var]\n+\treturn None\n+\t\n+def writedump(url, lower, upper):\n+\tif url.startswith('sim://'):\n+\t\tfilename = url[6:]\n+\t\tif filename[-1] == '/': filename = filename[:-1] #remove terminating slash\n+\telse:\n+\t\traise ValueError('sim:// url required')\n+\tf = open(filename, 'r');\n+\tstate = 'header'\n+\twroterev = False\n+\twhile(True):\n+\t\tl = f.readline()\n+\t\tif l == '': break\n+\t\tif state == 'header' and l.startswith('Revision-number: '):\n+\t\t\tstate = 'prefix'\n+\t\tif state == 'prefix' and l == 'Revision-number: %s\\n' % lower:\n+\t\t\tstate = 'selection'\n+\t\tif not upper == 'HEAD' and state == 'selection' and l == 'Revision-number: %s\\n' % upper:\n+\t\t\tbreak;\n+\n+\t\tif state == 'header' or state == 'selection':\n+\t\t\tif state == 'selection': wroterev = True\n+\t\t\tsys.stdout.write(l)\n+\treturn wroterev\n+\n+if __name__ == \"__main__\":\n+\tif not (len(sys.argv) in (3, 4, 5)):\n+\t\tprint \"usage: %s dump URL -rLOWER:UPPER\"\n+\t\tsys.exit(1)\n+\tif not sys.argv[1] == 'dump': raise NotImplementedError('only \"dump\" is suppported.')\n+\turl = sys.argv[2]\n+\tr = ('0', 'HEAD')\n+\tif len(sys.argv) == 4 and sys.argv[3][0:2] == '-r':\n+\t\tr = sys.argv[3][2:].lstrip().split(':')\n+\tif not getrevlimit() is None: r[1] = getrevlimit()\n+\tif writedump(url, r[0], r[1]): ret = 0\n+\telse: ret = 1\n+\tsys.exit(ret)\n\\ No newline at end of file\n-- \n1.7.9.5\n"},{"id":"195566","messageId":"7v8veakyar.fsf@alter.siamese.dyndns.org","threadId":"31065","inReplyTo":"1343074107-23262-1-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-23T20:38:52Z","receivedAt":"2012-07-23T20:38:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> To ease testing without depending on a reachable svn server, this\n> compact python script mimics parts of svnrdumps behaviour.\n> It requires the remote url to start with sim://.\n> Eventual slashes at the end of the url are stripped.\n\ns/ventual/xcess/ perhaps?\n\n> The url specifies the path of the svn dump file (as created by\n> svnrdump). Selectable parts of it, or the whole file, are written\n> to stdout. The part is selectable by giving start and end revision\n> on the command line.\n>\n> Start and end revisions can be specified on the command line\n> (-rSTART:END, like for svnrdump).\n> Only revisions between START and excluding END are replayed from\n> the dumpfile specified by the url. END can also be HEAD.\n>\n> If the start revision specified on the command line doesn't exist\n> in the dump file, it returns 1.\n> This emulates the behaviour of svnrdump when START>HEAD, i.e. the\n> requested start revision doesn't exist on the server.\n\nMuch more understandable than before.\n\n> To allow using the same dump file for simulating multiple\n> incremental imports the highest visible revision can be limited by\n> setting the environment variable SVNRMAX to that value. This\n> effectively limits HEAD to simulate the situation where higher\n> revs don't exist yet.\n\nIt is unclear how this is different from giving the ceiling by\nspecifying it as the \"END\" in -rSTART:END command line. Is this\nfeature really needed?\n"},{"id":"195622","messageId":"CABPQNSauwwQgM-48NBJVWhS5dRHhhoXpjrOxoXFiN5ymY2Rvzg@mail.gmail.com","threadId":"31065","inReplyTo":"1448476.VR1Gla8Cvg@flomedio","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2012-07-24T12:06:01Z","receivedAt":"2012-07-24T12:06:01Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Mon, Jul 23, 2012 at 2:44 PM, Florian Achleitner\n wrote:\n> + sys.exit(ret)\n> \\ No newline at end of file\n\nNit: add a \\n after \"sys.exit(ret)\", perhaps?\n"},{"id":"195652","messageId":"20120724195049.GD5210@burratino","threadId":"31065","inReplyTo":"7v8veakyar.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-07-24T19:50:49Z","receivedAt":"2012-07-24T19:50:49Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJunio C Hamano wrote:\n> Florian Achleitner writes:\n\n>> To ease testing without depending on a reachable svn server, this\n>> compact python script mimics parts of svnrdumps behaviour.\n>> It requires the remote url to start with sim://.\n[...]\n>> To allow using the same dump file for simulating multiple\n>> incremental imports the highest visible revision can be limited by\n>> setting the environment variable SVNRMAX to that value. This\n>> effectively limits HEAD to simulate the situation where higher\n>> revs don't exist yet.\n>\n> It is unclear how this is different from giving the ceiling by\n> specifying it as the \"END\" in -rSTART:END command line. Is this\n> feature really needed?\n\nI think the idea is that you put this script (or a symlink to it) on\nyour $PATH with higher precedence than svnrdump and run a command\nthat expected to be able to use svnrdump. Then instead of going to\nthe network, the command you run magically uses your test data\ninstead.\n\nIf the command you are testing wanted to run \"svnrdump\" without the\nupper endpoint set, we need to handle that request, either by emitting\nall the revs we have, or by stopping somewhere. The revlimit feature\nprovides the \"stopping somewhere\" behavior which is not strictly\nneeded but is presumably very useful when testing incremental fetch.\n\nFlorian, do you mind if I make the revlimit feature a separate patch\nwhen applying this?\n\nAnyway, it looks good and reasonable to me, so will apply.\n\nThanks.\nJonathan\n"},{"id":"195719","messageId":"4800743.K7ssi2Nkyl@flomedio","threadId":"31065","inReplyTo":"20120724195049.GD5210@burratino","subject":"Re: [PATCH] Add a svnrdump-simulator replaying a dump file for testing.","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-07-25T06:20:10Z","receivedAt":"2012-07-25T06:20:10Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Tuesday 24 July 2012 14:50:49 Jonathan Nieder wrote:\n> > It is unclear how this is different from giving the ceiling by\n> > specifying it as the \"END\" in -rSTART:END command line. Is this\n> > feature really needed?\n> \n> I think the idea is that you put this script (or a symlink to it) on\n> your $PATH with higher precedence than svnrdump and run a command\n> that expected to be able to use svnrdump. Then instead of going to\n> the network, the command you run magically uses your test data\n> instead.\n> \n> If the command you are testing wanted to run \"svnrdump\" without the\n> upper endpoint set, we need to handle that request, either by emitting\n> all the revs we have, or by stopping somewhere. The revlimit feature\n> provides the \"stopping somewhere\" behavior which is not strictly\n> needed but is presumably very useful when testing incremental fetch.\n\nExactly, the purpose is to transparently replace svnrdump.\nCallers of svnrdump usually will specify -rSTART:HEAD, because they want to \nfetch everything they don't yet have.\nThis feature allows to limit HEAD and to simulate incremental fetches using \nthe same dump file.\nFor me it proved very useful.\n\n> Florian, do you mind if I make the revlimit feature a separate patch\n> when applying this?\n\nNo problem.\n\n> \n> Anyway, it looks good and reasonable to me, so will apply.\n> \n> Thanks.\n> Jonathan\n\n--\nFlorian\n"}]}