{"thread":{"id":"23439","subject":"Update on SoC proposal: git-remote-svn","startedAt":"2010-04-13T05:29:56Z","lastAt":"2010-04-14T17:15:58Z","messageCount":6,"participants":["Ramkumar Ramachandra","Sam Vilain","Sverre Rabbelier","Steven Michalske","David Michael Barr"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"139394","messageId":"1271136573-sup-5613@kytes","threadId":"23439","inReplyTo":null,"subject":"Update on SoC proposal: git-remote-svn","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-04-13T05:29:56Z","receivedAt":"2010-04-13T05:29:56Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"\nHi,\n\nSam Vilain commented on my SoC proposal using Google's SoC interface,\nand requested me to CC my response to the mailing list. His original\ncomment is also quoted below.\n\n----------------------->8----------------------<8-------------------\nHi Sam,\n\n> Hi Ramkumar, I've looked at this proposal and seen that it differs a\n> bit from the version on the list, and I can't see the relevant\n> discussion, so I'll just throw my bit in here - though note this is a\n> technical comment and not a critique of your proposal.\n\nThere's been a lot of discussion, some on the list, and some more off\nthe list.\n\n> First, consider not using the SVN API at all while prototyping the\n> import part of the chain.  Instead, parse the 'svnadmin dump' stream\n> from a local mirror.  This will allow you to tackle the actual\n> problems involved and importing the data effectively, without\n> suffering from the brain-damage that is the SVN API.  After all, the\n> SVN API should be returning you all of the same information that the\n> dump stream does, so you can treat making it work using the remote\n> access API (eg svn_ra_replay, which is faster for mirroring AIUI) as a\n> separate task.  You will also more easily spot information which you\n> should be extracting from the API, but aren't - it's definitely all in\n> the dump format; it has to be.  I received similar advice to this\n> before building a perforce importer and let's just say it was\n> invaluable.\n\nYes. I've studied the SVN API, and I agree with you- it's quite\nhorrible. Instead of providing a API that's transparently\nbackward-compatible, they've provided different methods for different\nversions. There are also several variations of certain methods, and\nthis is quite confusing.\n\n`svnadmin dump` is exactly how I plan to start out- I've already\ndiscussed this with my to-be mentor, Sverre Rabbelier and David\nMichael-Barr, who's building a new SVN exporter in his own time.\n\n> Second, consider making the mirror phase emit directly to a tracking\n> branch via git-fast-export, that is not intended to be checked out.\n> Instead, it contains trees which correspond to revisions in the\n> mirrored SVN repository.  Directory and file properties can be saved\n> in the tree using specially named dotfiles, and revision properties\n> can be saved in the commit log.  Perhaps I misread your intentions\n> with the \"stripped down svnsync\" part, but syncing to a local SVN\n> repository seems to me like a waste of time; people can just do that\n> themselves if they choose anyway.  An SVN repository can easily be 10\n> times the size of the corresponding git store, and it just seems like\n> double-handling of the data and will make the whole process slower and\n> more cumbersome than it needs to be.\n\n> With all the blobs already in the git store, and the information\n> needed to perform the data mining operation which is the extraction of\n> git-style branch histories from the svn data, you will be working with\n> data which is all in git-land, and exporting referencing blobs which\n> are already in the store.  This will save you a LOT of time, as it\n> means in this stage you are not handling the actual file images; just\n> constructing branch histories in the git-fast-import stream.  Your\n> branch miner will potentially be able to process thousands of\n> revisions per second this way, even from python.\n\nAgreed. Sverre, David and I discussed exactly this- The final version\nof mirror-client will dump all the SVN information to a Git store\nfirst, so we can do the mapping painlessly in Git. There are some\nconcerns about information loss though, which we'll have to deal with\nas we go on.\n\n> Also bear in mind\n> that people might use SVN in a way that violates the expectations of\n> this branch miner.  An example is putting a README file in the\n> top-level projects directory, a heuristic approach might consider that\n> the start of a new project and then mess up later stages.  Another\n> example is people accidentally deleting trunk and re-adding it; the\n> nice thing about this two-stage approach is that it allows advanced\n> users to muck with the \"raw\" data (ie, this whole repository tracking\n> branch) using git to do things like graft away the bad revisions, and\n> then the second stage will use the corrected data.  Of course\n> eventually, this detail will be hidden by the remote helper.\n\nExcellent suggestion! I'll attempt to build the plumbing for the\nmapping in a manner that exposes a sane interface.\n\n> As a general comment - you must be careful in trying to assume that\n> what you are attempting is even possible.  Sure, you want 'git clone\n> svn://example.com/myrepo' to work, but what does that mean?  A\n> repository in SVN is a filesystem, which can contain multiple\n> projects.  In git, a repo is a single project.  People might expect to\n> be able to clone the trunk URL for instance.  My advice there is to\n> not support that use case at all, it's a complete can of worms which\n> you will discover as you tackle the conversion algorithms.  Just focus\n> on making the case where the complete repository is mirrored work for\n> this project.  Mining a single branch out of SVN without all data\n> available is the domain of git-svn and really you don't want to go\n> there.\n\nHm, this is something that I hadn't thought about earlier. Thanks for\nthe suggestion- I will not attempt to go into complicated cases,\natleast in my summer term.\n\n> Anyway like I say, please follow-up on the mailing list, and this\n> advice can receive wider scrutiny.\n\nThank you for your valuable comment!\n\n-- Ram\n"},{"id":"139398","messageId":"4BC42C98.6010006@vilain.net","threadId":"23439","inReplyTo":"1271136573-sup-5613@kytes","subject":"Re: Update on SoC proposal: git-remote-svn","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2010-04-13T08:34:32Z","receivedAt":"2010-04-13T08:34:32Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Ramkumar Ramachandra wrote:\n>> Hi Ramkumar, I've looked at this proposal and seen that it differs a\n>> bit from the version on the list, and I can't see the relevant\n>> discussion, so I'll just throw my bit in here - though note this is a\n>> technical comment and not a critique of your proposal.\n>>     \n>\n> There's been a lot of discussion, some on the list, and some more off\n> the list.\n>   \n\nRight, that will be why :-).  No problem, apologies for not catching the\nearlier discussion.  It's good to hear that you have collectively\nreached a very similar design to the one indicated from my perforce\nimporter experiment (SVN is of course, a poor imitation of Perforce in\nmany ways).\n\n>> Anyway like I say, please follow-up on the mailing list, and this\n>> advice can receive wider scrutiny.\n>>     \n>\n> Thank you for your valuable comment!\n>   \n\nNo problem at all.  I will watch the development of this project with\ninterest.\n\nAll the best,\nSam\n"},{"id":"139423","messageId":"l2yfabb9a1e1004130901z19db3392kbbd4d975a5d439f7@mail.gmail.com","threadId":"23439","inReplyTo":"1271136573-sup-5613@kytes","subject":"Re: Update on SoC proposal: git-remote-svn","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-04-13T16:01:55Z","receivedAt":"2010-04-13T16:01:55Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Tue, Apr 13, 2010 at 07:29, Ramkumar Ramachandra <artagnon@gmail.com> wrote:\n> Hm, this is something that I hadn't thought about earlier. Thanks for\n> the suggestion- I will not attempt to go into complicated cases,\n> atleast in my summer term.\n\nI think we can safely say that \"git clone svn://example.com/myrepo\"\nonly has to work for svn repositories that have a 'sane' layout. If\nyou want a non-standard layout you can do 'git init && git configure\n.... && git fetch', where you configure everything the way you want\nbefore fetching. We could perhaps even provide a tool to aid with\nconfiguring things; or perhaps a mode to not do any rewriting (just\nimport the data into git as-is) and then have a tool that examines the\nhistory and interactively helps you set up an appropriate rewrite\nconfig? Interesting times are ahead :).\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"139456","messageId":"F0CD5B83-2D28-47E1-A336-5C88E2803CBE@gmail.com","threadId":"23439","inReplyTo":"1271136573-sup-5613@kytes","subject":"Re: Update on SoC proposal: git-remote-svn","fromName":"Steven Michalske","fromEmail":"smichalske@gmail.com","sentAt":"2010-04-14T06:33:23Z","receivedAt":"2010-04-14T06:33:23Z","isPatch":false,"sender":{"key":"smichalske@gmail.com","avatar":"https://gravatar.com/avatar/721f27456adc9ac84f3bb235f021a70015abb9e09222ae8622fc5579c6a203c1?d=mp&s=160"},"body":"Ramkumar,\n\nIn reading this I wondered how a svn dump of one of the repositories  \nmonitor would size.  If I were to check out the svn root of that  \nrepository, I would use well over 3TB of disk space to have that  \nchecked out, I filled my 750GB drive with about a third of it checked  \nout.  About 256MB of code with thousands of tags and hundreds of  \nbranches.\n\nIt looks like svnadmin dump defaults to dumping all data.  Fortunately  \nit has a delta option, which looks like it would be needed to dump  \nthis repository I am speaking of without filling up many hard drives.\n\nThis might also be helped if the dumps are chunked into ranges for  \nmany thousands of commits as well, this would keep the files more  \nmanageable\n\nJust food for thought.\n\nSteve\n\nOn Apr 12, 2010, at 10:29 PM, Ramkumar Ramachandra wrote:\n\n>\n> Hi,\n>\n> Sam Vilain commented on my SoC proposal using Google's SoC interface,\n> and requested me to CC my response to the mailing list. His original\n> comment is also quoted below.\n>\n> ----------------------->8----------------------<8-------------------\n> Hi Sam,\n>\n>> Hi Ramkumar, I've looked at this proposal and seen that it differs a\n>> bit from the version on the list, and I can't see the relevant\n>> discussion, so I'll just throw my bit in here - though note this is a\n>> technical comment and not a critique of your proposal.\n>\n> There's been a lot of discussion, some on the list, and some more off\n> the list.\n>\n>> First, consider not using the SVN API at all while prototyping the\n>> import part of the chain.  Instead, parse the 'svnadmin dump' stream\n>> from a local mirror.  This will allow you to tackle the actual\n>> problems involved and importing the data effectively, without\n>> suffering from the brain-damage that is the SVN API.  After all, the\n>> SVN API should be returning you all of the same information that the\n>> dump stream does, so you can treat making it work using the remote\n>> access API (eg svn_ra_replay, which is faster for mirroring AIUI)  \n>> as a\n>> separate task.  You will also more easily spot information which you\n>> should be extracting from the API, but aren't - it's definitely all  \n>> in\n>> the dump format; it has to be.  I received similar advice to this\n>> before building a perforce importer and let's just say it was\n>> invaluable.\n>\n> Yes. I've studied the SVN API, and I agree with you- it's quite\n> horrible. Instead of providing a API that's transparently\n> backward-compatible, they've provided different methods for different\n> versions. There are also several variations of certain methods, and\n> this is quite confusing.\n>\n> `svnadmin dump` is exactly how I plan to start out- I've already\n> discussed this with my to-be mentor, Sverre Rabbelier and David\n> Michael-Barr, who's building a new SVN exporter in his own time.\n>\n>> Second, consider making the mirror phase emit directly to a tracking\n>> branch via git-fast-export, that is not intended to be checked out.\n>> Instead, it contains trees which correspond to revisions in the\n>> mirrored SVN repository.  Directory and file properties can be saved\n>> in the tree using specially named dotfiles, and revision properties\n>> can be saved in the commit log.  Perhaps I misread your intentions\n>> with the \"stripped down svnsync\" part, but syncing to a local SVN\n>> repository seems to me like a waste of time; people can just do that\n>> themselves if they choose anyway.  An SVN repository can easily be 10\n>> times the size of the corresponding git store, and it just seems like\n>> double-handling of the data and will make the whole process slower  \n>> and\n>> more cumbersome than it needs to be.\n>\n>> With all the blobs already in the git store, and the information\n>> needed to perform the data mining operation which is the extraction  \n>> of\n>> git-style branch histories from the svn data, you will be working  \n>> with\n>> data which is all in git-land, and exporting referencing blobs which\n>> are already in the store.  This will save you a LOT of time, as it\n>> means in this stage you are not handling the actual file images; just\n>> constructing branch histories in the git-fast-import stream.  Your\n>> branch miner will potentially be able to process thousands of\n>> revisions per second this way, even from python.\n>\n> Agreed. Sverre, David and I discussed exactly this- The final version\n> of mirror-client will dump all the SVN information to a Git store\n> first, so we can do the mapping painlessly in Git. There are some\n> concerns about information loss though, which we'll have to deal with\n> as we go on.\n>\n>> Also bear in mind\n>> that people might use SVN in a way that violates the expectations of\n>> this branch miner.  An example is putting a README file in the\n>> top-level projects directory, a heuristic approach might consider  \n>> that\n>> the start of a new project and then mess up later stages.  Another\n>> example is people accidentally deleting trunk and re-adding it; the\n>> nice thing about this two-stage approach is that it allows advanced\n>> users to muck with the \"raw\" data (ie, this whole repository tracking\n>> branch) using git to do things like graft away the bad revisions, and\n>> then the second stage will use the corrected data.  Of course\n>> eventually, this detail will be hidden by the remote helper.\n>\n> Excellent suggestion! I'll attempt to build the plumbing for the\n> mapping in a manner that exposes a sane interface.\n>\n>> As a general comment - you must be careful in trying to assume that\n>> what you are attempting is even possible.  Sure, you want 'git clone\n>> svn://example.com/myrepo' to work, but what does that mean?  A\n>> repository in SVN is a filesystem, which can contain multiple\n>> projects.  In git, a repo is a single project.  People might expect  \n>> to\n>> be able to clone the trunk URL for instance.  My advice there is to\n>> not support that use case at all, it's a complete can of worms which\n>> you will discover as you tackle the conversion algorithms.  Just  \n>> focus\n>> on making the case where the complete repository is mirrored work for\n>> this project.  Mining a single branch out of SVN without all data\n>> available is the domain of git-svn and really you don't want to go\n>> there.\n>\n> Hm, this is something that I hadn't thought about earlier. Thanks for\n> the suggestion- I will not attempt to go into complicated cases,\n> atleast in my summer term.\n>\n>> Anyway like I say, please follow-up on the mailing list, and this\n>> advice can receive wider scrutiny.\n>\n> Thank you for your valuable comment!\n>\n> -- Ram\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"139471","messageId":"E7EBCCD4-418B-4573-A70A-B7E06D6C6D50@cordelta.com","threadId":"23439","inReplyTo":"F0CD5B83-2D28-47E1-A336-5C88E2803CBE@gmail.com","subject":"Re: Update on SoC proposal: git-remote-svn","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-04-14T12:52:02Z","receivedAt":"2010-04-14T12:52:02Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi Steve,\n\n> In reading this I wondered how a svn dump of one of the\n> repositories monitor would size.  If I were to check out the svn\n> root of that repository, I would use well over 3TB of disk space\n> to have that checked out, I filled my 750GB drive with about a\n> third of it checked out.  About 256MB of code with thousands\n> of tags and hundreds of branches.\n\nI encountered this issue with my first attempt to validate the output of \nmy dump conversion tool. My case wasn't as dire, 350GB would\nhave sufficed but I was working in a 160GB partition.\nChecking out tags side by side is a sure way to fill your disk.\n\n> It looks like svnadmin dump defaults to dumping all data.\n> Fortunately it has a delta option, which looks like it would be\n> needed to dump this repository I am speaking of without filling\n> up many hard drives.\n\nThe svn dump format is not quite that silly, even without deltification\nit doesn't output blobs that are just an unaltered copy from a\nprevious revision.\nHandling deltified dumps will greatly increase the complexity of the\nimport process. Blob content would have be computed from existing\nblobs rather than simply passed through.\n\n> This might also be helped if the dumps are chunked into ranges\n> for many thousands of commits as well, this would keep the files\n> more manageable\n\nBeing able to handle a dump stream reassembled from such\npiecewise dumps is an important feature which I haven't finished\nimplementing yet.\n\n> Just food for thought.\n\nThanks for the feed.\n\n--\nDavid Barr\n"},{"id":"139492","messageId":"21A95727-F438-48FB-9A16-132B4DE85B6E@cordelta.com","threadId":"23439","inReplyTo":"F0CD5B83-2D28-47E1-A336-5C88E2803CBE@gmail.com","subject":"Re: Update on SoC proposal: git-remote-svn","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-04-14T17:15:58Z","receivedAt":"2010-04-14T17:15:58Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi Steve,\n\n> If I were to check out the svn root of that repository, I would use well\n> over 3TB of disk space to have that checked out ...\n\nThis stirred my thoughts and I whipped up a bash script that uses SVK,\nfind, shasum and ln to build a filesystem view of the root of an svn\nrepository that consumes moderate storage:\n\nSVK_DEPOT=\"\"\nMAX_REV=12345\n\nCO_DIR=validation\nHASH_DIR=hashes\n\nsvk co -r1 /$SVK_DEPOT/ $CO_DIR\nmkdir -p $HASH_DIR\nfor (( REV=1 ; REV<=MAX_REV ; ++REV )) do\n  svk up -r$REV $CO_DIR\n  # Hashify working copy\n  find $CO_DIR -type d -cmin -5 -prune -o \\\n    -type f -links 1 -exec shasum '{}' + | (\n    while read HASH FILE ; do\n      [ -x \"$FILE\" ] && HASH=\"$HASH\"x\n      ln \"$FILE\" $HASH_DIR/$HASH 2>/dev/null || \\\n        ln -f $HASH_DIR/$HASH \"$FILE\"\n    done\n  )\ndone\n\nImportant assumptions are that each update will take less\nthan 5 minutes and that SVK uses writes to a temporary file\nand then renames to perform a modification.\nI've used this to build a simple validation script for my project.\nI estimate it will use about 20GB to represent my 1GB repo\nand that it will take about 3 hours.\n\n--\nDavid Barr\n"}]}