{"thread":{"id":"29992","subject":"GSoC intro","startedAt":"2012-03-19T14:42:15Z","lastAt":"2012-04-19T12:26:03Z","messageCount":46,"participants":["Florian Achleitner","Andrew Sayers","David Barr","Ramkumar Ramachandra","Miles Bader","Dmitry Ivankov","Jonathan Nieder","Tomas Carnecky","Stephen Bash","Jakub Narebski","Thomas Rast"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"187253","messageId":"11292500.AVmZFUUvNi@flobuntu","threadId":"29992","inReplyTo":null,"subject":"GSoC intro","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-03-19T14:42:15Z","receivedAt":"2012-03-19T14:42:15Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi fellow git developers!\n\nI'm curious about applying for GSoC 2012 considering the idea \"Remote helper \nfor Subversion\". \nI'm using git since years and have converted my svn repos to git years ago, \nbut I'm not yet familiar with the pre-work on this topic. Is there a branch in \ngit's git?\nDoes a \"full-featured bi-directional git-remote-svn\" mean, that it should work \nlike any remote git repository where you can push to and fetch from?\n\nBelow I briefly introduce myself, for those who are interested.\n\nAbout me\nMy name is Florian Achleitner (IRC: FlyingFlo). I'm \nfrom Austria and I study Telematics (a blend of computer science and \nelectric engineering) at the Graz University of Technology. I'm currently in \nthe first year of the master program. Before starting my studies I worked for \nfour years as a developer of embedded systems in industry.\n\nMy programming experience grew since I started writing programs on TI \ncalculators in school probably 15 years ago.\nI'm open-source enthusiast, exclusively using Linux since years.\nI currently work as teaching assistant for an exercise about programming \noperating systems. In this course we also teach the students to use git.\n\nAbout me and GSoC\nIn summer 2010 I participated in GSoC for hugin writing a Makefile-creation \nlibrary in C++, which is used to drive the panorama creation [1]. It was a \ngreat experience and a cool, successful summer job! ( and it was merged in \nhugin's master branch :-) )\n\nWhy git?\n- I use git daily. It's always good to work on things you use and a chance to \ncontribute something.\n- I like C\n- I used svn. Nowadays I only use it if i have to ;)\n- The community interaction aspect of open source development is very \ninteresting.. as the ideas page says \".. and get it merged into upstream Git.\"\n\n\n[1] http://hugin.hg.sourceforge.net/hgweb/hugin/hugin/branches branch: \ngsoc2010_makefilelib (unfortunately the web fronted doesn't display a specific \nbranch)\n\nRegards,\nFlo\n-- \nFlorian Achleitner, BSc\n\n\"In a world without walls and fences, \nwho needs windows and gates?\" ;-)\n"},{"id":"187282","messageId":"4F67A5B6.7090508@pileofstuff.org","threadId":"29992","inReplyTo":"11292500.AVmZFUUvNi@flobuntu","subject":"Re: GSoC intro","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-03-19T21:31:34Z","receivedAt":"2012-03-19T21:31:34Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"Hi Florian,\n\nI've become interested in Git<->SVN issues lately, so I'll tell you what\nI know. Hopefully more knowledgeable people will correct me if I'm wrong.\n\nThe main thrust of SVN development is done in the \"svn-fe\" project. You\ncan see the work so far in the \"vcs-svn\"[1] and \"contrib/svn-fe\"[2]\nsubdirectories of the main git repo. My experience as a user has been\nthat it does a great job of the things it does, but so far it only does\na subset of the things I want. For example, it can't write to SVN and I\nthink I'm right in saying it can't yet update from SVN after the initial\ndownload. David Barr is the main contact for svn-fe - he's an\nexperienced mentor and will be able to tell you all about the juicy\nlow-hanging fruit.\n\nOne limitation of svn-fe is that it downloads the whole SVN repository\ninto a single git branch, without separating out trunk, branches and\ntags. I've been working on this problem over the past few months, and\nhave split it into three parts (a language to describe which directories\nare branches etc., export from SVN to that language, and import from\nthat language to git). I'd be very flattered if you wanted to work on\nthis, but I couldn't honestly recommend it over svn-fe. The language\nitself is a one man job that doesn't have much creative work left; SVN\nexport is all about exposing yourself to weird little abuses of version\ncontrol that don't teach you much beyond bad habits; and while git\nimport would be a fun little project, I don't know enough about git's C\nimplementation to provide any useful mentoring.\n\nGood luck with the summer, and as an svn-fe user I hope you're very\nproductive :)\n\n\t- Andrew\n\n[1] https://github.com/git/git/tree/master/vcs-svn\n[2] https://github.com/git/git/tree/master/contrib/svn-fe\n"},{"id":"187322","messageId":"6438739.8F5iWPIgU8@flobuntu","threadId":"29992","inReplyTo":"11292500.AVmZFUUvNi@flobuntu","subject":"Re: GSoC intro","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-03-20T12:25:28Z","receivedAt":"2012-03-20T12:25:28Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Monday 19 March 2012 21:31:34 you wrote:\n> [...]\n> Good luck with the summer, and as an svn-fe user I hope you're very\n> productive\n> \n> - Andrew\n> \n> [1] https://github.com/git/git/tree/master/vcs-svn\n> [2] https://github.com/git/git/tree/master/contrib/svn-fe\n\nThanks for the starting points Andrew!\n\n-- Florian\n"},{"id":"187328","messageId":"CAFfmPPPabm_H9f2Zr8eWc7Wxo6UDz-km_Vg8cc-O38XhGCrj7Q@mail.gmail.com","threadId":"29992","inReplyTo":"11292500.AVmZFUUvNi@flobuntu","subject":"Re: GSoC intro","fromName":"David Barr","fromEmail":"davidbarr@google.com","sentAt":"2012-03-20T13:19:41Z","receivedAt":"2012-03-20T13:19:41Z","isPatch":false,"sender":{"key":"davidbarr@google.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi Florian,\n\n> I'm curious about applying for GSoC 2012 considering the idea \"Remote helper\n> for Subversion\".\n> I'm using git since years and have converted my svn repos to git years ago,\n> but I'm not yet familiar with the pre-work on this topic. Is there a branch in\n> git's git?\n\nMuch of the progress so far has been merged into master.\nStill outstanding are some of Dmitry's patches:\nremote-svn-alpha_v2 [1]\nsvn-fe-options_v7 [2]\n\n> Does a \"full-featured bi-directional git-remote-svn\" mean, that it should work\n> like any remote git repository where you can push to and fetch from?\n\nYes, that's the plan. To be fair, it is a stretch goal. Two GSoC\nstudents have brought us as far as a read-only remote helper. So I\nthink there's at least two summers' worth of work remaining.\n\n> Below I briefly introduce myself, for those who are interested.\n>\n> About me\n> My name is Florian Achleitner (IRC: FlyingFlo). I'm\n> from Austria and I study Telematics (a blend of computer science and\n> electric engineering) at the Graz University of Technology. I'm currently in\n> the first year of the master program. Before starting my studies I worked for\n> four years as a developer of embedded systems in industry.\n>\n> My programming experience grew since I started writing programs on TI\n> calculators in school probably 15 years ago.\n> I'm open-source enthusiast, exclusively using Linux since years.\n> I currently work as teaching assistant for an exercise about programming\n> operating systems. In this course we also teach the students to use git.\n\nThanks for the introduction. When I first got involved with this\nsub-project, I gave a quick self introduction [3]. As a potential\nmentor, it would be prudent to let you know what my commitments are.\nMy day job is primarily to contribute to chromium.org and webkit.org.\nI also have a 20% commitment to git-core and related projects.\n\n> About me and GSoC\n> In summer 2010 I participated in GSoC for hugin writing a Makefile-creation\n> library in C++, which is used to drive the panorama creation [1]. It was a\n> great experience and a cool, successful summer job! ( and it was merged in\n> hugin's master branch :-) )\n\n> [1] http://hugin.hg.sourceforge.net/hgweb/hugin/hugin/branches branch:\n> gsoc2010_makefilelib (unfortunately the web fronted doesn't display a specific\n> branch)\n\nA track record is a plus.\n\n> Why git?\n> - I use git daily. It's always good to work on things you use and a chance to\n> contribute something.\n\nI'm sure this is the reason most git contributors are here.\n\n> - I like C\n> - I used svn. Nowadays I only use it if i have to ;)\n\nYou're in good company.\n\n> - The community interaction aspect of open source development is very\n> interesting.. as the ideas page says \".. and get it merged into upstream Git.\"\n\nThe git contributors are mostly a pleasure to work with. The volume\nand quality of feedback to contribution, especially from newcomers,\nsets it apart from the other communities I participate in.\n\nSome extra reading:\n\nTo catch up on the current state of the art with respect to\ntranslating Subversion history read:\nAnother bite of the reposturgeon, Eric S. Raymond [4].\nUnfortunately, he hasn't published the code quite yet.\nHowever, he did what we have been lax to do and contacted the\nSubversion developers to assist updating protocol documentation [5].\nI think the corner cases for the Subversion delta format are still\nundocumented [6].\n\n[1] https://github.com/divanorama/git/tree/remote-svn-alpha_v2\n[2] https://github.com/divanorama/git/tree/svn-fe-options_v7\n[3] http://thread.gmane.org/gmane.comp.version-control.git/143187/focus=143201\n[4] http://esr.ibiblio.org/?p=4071\n[5] http://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\n[6] http://svn.apache.org/repos/asf/subversion/trunk/notes/svndiff\n\n> Regards,\n> Flo\n\n--\nDavid Barr\n"},{"id":"187435","messageId":"11105663.xsXc2sBkNH@flobuntu","threadId":"29992","inReplyTo":"CAFfmPPPabm_H9f2Zr8eWc7Wxo6UDz-km_Vg8cc-O38XhGCrj7Q@mail.gmail.com","subject":"Re: GSoC intro","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-03-21T21:16:27Z","receivedAt":"2012-03-21T21:16:27Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi!\n\nAfter the exam today, I started to dig into the topic a little.\nSo I accumulated some questions ..\n\nOn Wednesday 21 March 2012 00:19:41 David Barr wrote:\n> Much of the progress so far has been merged into master.\n> Still outstanding are some of Dmitry's patches:\n> remote-svn-alpha_v2 [1]\n> svn-fe-options_v7 [2]\n\nI tried to find svn-related parts in gits sources. I found:\n - the huge ./git-svn.perl, which seems to be the git-svn implementation.\n - ./contrib/svn-fe/ and ./vcs-svn/, \nthose you pointed me at.\nDid I miss something?\nIs there any seperate source documentation? The source files I looked at \ncontain only very few comments. And nothing about the big picture.\nI built make doc, but it seems it's mostly user documentation.\n\n> Yes, that's the plan. To be fair, it is a stretch goal. Two GSoC\n> students have brought us as far as a read-only remote helper. So I\n> think there's at least two summers' worth of work remaining.\n\nWhat is the remote helper? How can I use/try it?\n> [1] https://github.com/divanorama/git/tree/remote-svn-alpha_v2\nIs it in here? Should my project continue on this work?\nUntil now, I've never used any remote that was not git.\n\n> > About me and GSoC\n> > In summer 2010 I participated in GSoC for hugin writing a\n> > Makefile-creation library in C++, which is used to drive the panorama\n> > creation [1]. It was a great experience and a cool, successful summer\n> > job! ( and it was merged in hugin's master branch :-) )\n> > \n> > [1] http://hugin.hg.sourceforge.net/hgweb/hugin/hugin/branches branch:\n> > gsoc2010_makefilelib (unfortunately the web fronted doesn't display a\n> > specific branch)\n> \n> A track record is a plus.\n\nIf you like, I could provide more references, e.g. a university course project \nin C using git.\n\n> Some extra reading:\n> [...]\nHaven't yet read it.\n\nHm, and there are still some general questions:\nWhat about git-svn? Whats wrong with it? (I haven't used it) I saw the huge \nperl script, this looks a little extreme ;). But it provides bi-directional \naccess?!\n\nsvn-fe reads a dump of the svn repo. How can this approach ever be \nbidirectional? Probably I've to do the extra reading first .. \n\n-- \nFlorian \n"},{"id":"187710","messageId":"CALkWK0nW91PE2810qrZUbL0x-_YTTA_2tLFVhvXBJ2NFGvVxog@mail.gmail.com","threadId":"29992","inReplyTo":"11105663.xsXc2sBkNH@flobuntu","subject":"Re: GSoC intro","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2012-03-26T11:06:46Z","receivedAt":"2012-03-26T11:06:46Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Florian,\n\nFlorian Achleitner wrote:\n> On Wednesday 21 March 2012 00:19:41 David Barr wrote:\n>> Much of the progress so far has been merged into master.\n>> Still outstanding are some of Dmitry's patches:\n>> remote-svn-alpha_v2 [1]\n>> svn-fe-options_v7 [2]\n>\n> I tried to find svn-related parts in gits sources. I found:\n>  - the huge ./git-svn.perl, which seems to be the git-svn implementation.\n>  - ./contrib/svn-fe/ and ./vcs-svn/,\n> those you pointed me at.\n> Did I miss something?\n> Is there any seperate source documentation? The source files I looked at\n> contain only very few comments. And nothing about the big picture.\n\nA lot of big-picture discussions can be found in mailing list\narchives. Let us know what you're looking for exactly.\n\n>> Yes, that's the plan. To be fair, it is a stretch goal. Two GSoC\n>> students have brought us as far as a read-only remote helper. So I\n>> think there's at least two summers' worth of work remaining.\n>\n> What is the remote helper? How can I use/try it?\n\nThe remote helper is an external program that git invokes to handle\nspecific protocols. See ./git_remote_helpers for example.\n\n>> [1] https://github.com/divanorama/git/tree/remote-svn-alpha_v2\n> Is it in here? Should my project continue on this work?\n> Until now, I've never used any remote that was not git.\n\nYou might also decide to build a brand new remote helper.\n\n>> A track record is a plus.\n>\n> If you like, I could provide more references, e.g. a university course project\n> in C using git.\n>\n>> Some extra reading:\n>> [...]\n> Haven't yet read it.\n>\n> Hm, and there are still some general questions:\n> What about git-svn? Whats wrong with it? (I haven't used it) I saw the huge\n> perl script, this looks a little extreme ;). But it provides bi-directional\n> access?!\n\nThe main problem with git-svn.perl is that it's hard to maintain or\nextend. See also: David Barr's LCA talk [1].\n\n> svn-fe reads a dump of the svn repo. How can this approach ever be\n> bidirectional? Probably I've to do the extra reading first ..\n\nIt can't. You'll have to write something to handle the Git -> SVN\nconversion. See also: one of my earlier attempts in this regard [2].\n\n[1]: http://www.youtube.com/watch?v=0hVuv-wv4Dw\n[2]: https://github.com/artagnon/git/tree/svn-fi\n\n Ram\n"},{"id":"187852","messageId":"2148933.pnpYo0xMAP@flomedio","threadId":"29992","inReplyTo":"CALkWK0nW91PE2810qrZUbL0x-_YTTA_2tLFVhvXBJ2NFGvVxog@mail.gmail.com","subject":"Re: GSoC intro","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-03-27T13:53:01Z","receivedAt":"2012-03-27T13:53:01Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi Ramkumar, and everybody!\n\nThanks for your tips!\nI'm currently a little quiet on that topic because I'm working hard on a \nuniversity project that we need to kick off this week, because easter holidays \nstart in a few days. I'll then switch back to git immediatly, it's proposal \ntime!\n\nOn Monday 26 March 2012 16:36:46 Ramkumar Ramachandra wrote:\n\n> \n> A lot of big-picture discussions can be found in mailing list\n> archives. Let us know what you're looking for exactly.\n> \n> The main problem with git-svn.perl is that it's hard to maintain or\n> extend. See also: David Barr's LCA talk [1].\n\nThe talk from David Barr is exactly what I was searching (about the big \npicture and so ..). It gives a good introduction to what you guys already know \nvery well, of course. Like what exists, which are the well-known problems..\nThx for the link!\n \n> Ram\n\nFlorian \n"},{"id":"187917","messageId":"buo8vilf8ox.fsf@dhlpc061.dev.necel.com","threadId":"29992","inReplyTo":"CALkWK0nW91PE2810qrZUbL0x-_YTTA_2tLFVhvXBJ2NFGvVxog@mail.gmail.com","subject":"Re: GSoC intro","fromName":"Miles Bader","fromEmail":"miles@gnu.org","sentAt":"2012-03-28T08:09:34Z","receivedAt":"2012-03-28T08:09:34Z","isPatch":false,"sender":{"key":"miles@gnu.org","avatar":"https://gravatar.com/avatar/01069b69593af7bff28e2f97afeb3644ae6fe2f5f56cb3a8cf34c5fb8c36efe5?d=mp&s=160"},"body":"Ramkumar Ramachandra writes:\n>> Hm, and there are still some general questions:\n>> What about git-svn? Whats wrong with it? (I haven't used it) I saw the huge\n>> perl script, this looks a little extreme ;). But it provides bi-directional\n>> access?!\n>\n> The main problem with git-svn.perl is that it's hard to maintain or\n> extend. See also: David Barr's LCA talk [1].\n\ngit-svn's also pretty annoying to use (e.g. the way dcommit rebases\nanything you push to svn, which makes juggling local git branches\nproblematic; ugh)... :/\n\n-miles\n\n-- \n.Numeric stability is probably not all that important when you're guessing.\n"},{"id":"187920","messageId":"loom.20120328T112337-150@post.gmane.org","threadId":"29992","inReplyTo":"buo8vilf8ox.fsf@dhlpc061.dev.necel.com","subject":"Re: GSoC intro","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2012-03-28T09:30:49Z","receivedAt":"2012-03-28T09:30:49Z","isPatch":false,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"Miles Bader gnu.org> writes:\n\n> git-svn's also pretty annoying to use (e.g. the way dcommit rebases\n> anything you push to svn, which makes juggling local git branches\n> problematic; ugh)... :/\n\nThere is git svn dcommit --no-rebase to disable the rebase step.\n\"upstream svn\" branch history will still diverge from your local branch though.\n\n> \n> -miles\n> \n"},{"id":"188301","messageId":"2487557.B8qfnaixh3@flomedio","threadId":"29992","inReplyTo":"2148933.pnpYo0xMAP@flomedio","subject":"GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner@student.tugraz.at","sentAt":"2012-04-02T08:30:58Z","receivedAt":"2012-04-02T08:30:58Z","isPatch":false,"sender":{"key":"florian.achleitner@student.tugraz.at","avatar":null},"body":"Hi everybody!\n\nHere is my draft of the proposal for the GSoC project. RFC!\nPlease comment and tell me what you think and if I understood it all right!\n\nI spent a lot of lines with wiriting about the current situation. This is \nmostly because, as a newbee, I spent a lot of time examining what we already \nhave and wrote it down finally.\n\nThe draft is inlined below. I hope it's not too long to read. I will put it on \na github wiki later, once i figure out how this works ;)\n\nFlorian\n\n\n==Remote helper for Subversion==\n\n==Introduction==\n{ for non-insiders }\nGit [1] is a powerful distributed version control system (DVCS). \"Distributed\" \nmeans that everybody works on a full featured repository. To collaborate with \nother user's repositories git can fetch and pull from remote repositories \nusing several transports (http://, ssh://, git://, ...). Git has a very \npowerful and useful concept of branches. They are lightweight pointers to \ncommits (heads).\n\nSubversion (svn) [2] was created as a successor of CVS, both follow a strict \nclient-server design, where the repository exclusively lives on the central \nserver and every client only checks out a copy of a single revision at a time. \nSVN doesn't truly have a concept of branches. SVN branches are a copy of a \ndirectory (so are tags).\n\n==What we want (the general goal)==\nshort: \ngit clone svn://\ngit push\ngit fetch\n\nA full-featured bi-directional remote helper for svn that allows git to use a \nsvn repository as a remote, mostly like a remote git repo.\nRemote helpers are separate programs invoked by git to communicate with \nforeign repositories. They are used by transceiving a command and data stream \nvia stdin and stdout.\n\nThe remote helper interface [2] supports commands that deliver a git-fast-\nimport stream from the remote repo.\n\ngit-fast-import [4] is a format to serialize a git repository into a text \nformat. It is used by the tools git-fast-import and git-fast-export.\n\nThe remote helper has to convert the foreign protocol and data (svn) to the \ngit-fast-import format.\n\n==What are the challenges? ==\nTo summarize: The way git tracks the state of the working tree and svn's way \nare different in several aspects. This makes a direct mapping impossible.\nThere are lots of discussions about these issues on the git mailing list [5].\n\nSome aspects: (I'm sure this is incomplete)\n- svn commit and file metadata, it's symlink and permission representation have \nto to be mapped to git.\n\n- svn history can only be extracted from the server (we have svnrdump for \nthat)\n\n- svn commits are only possible after updating the working copy first, i.e. \nfetching and merging new revisions on the server. This is like implicitly \nrebasing your local work on the remote head before pushing to an svn \nrepository.\nIn git there is of course no such restriction.\n\n- and the most challenging: mapping subversion branches to git branches. \nIn svn a branch is created by copying a directory with 'svn copy'. svn doesn't \nhave a concept of branches by itself. \n\nBranches exist due to the convention of having branches/, trunk/, and tags/ \ndirectories in a repository, so do tags. But this is not mandatory and \ntherefore there are many different layouts. It follows that in svn it is also \npossible to commit across branches. This means that a single commit can change \nfiles on more than one branch (accidentally or deliberately).\nTo convert svn branches to git we have to detect branch semantics by examining \nthe svn tree's structure and it's metadata (it has a 'copyfrom' property). \nPrevious efforts show that this will not be possible fully automatically \nwithout configuration and interaction with the user.\n\nThis brings us to:\n==What we have: (existing work)==\nAndrew Sayers is currently developing a language to describe svn to git branch \nmapping [6]. I plan to use the language as a configuration for the remote \nhelper that specifies unclear aspects.\n\n\"esr\" developed a tool to manipulate and export subversion repositories [7] \nthat should be able to detect branches, but it's sources are not available \nyet.\n\nIn git's tree there is git-svn, a huge Perl script used to convert svn to git. \nIt detects branches, but with problems. It also supports some kind of pushing \ncommits to svn using a separate command. It's problem: it's unmaintainable, \nbugs are hard to locate and to fix.\n\nThere are several other one-way conversion tools, e.g. svn-fast-export, \nsvn2git.py.\n\nIn git's source tree we have a vcs-svn/, a set of functions to convert svn \ndumps to git-fast-import streams. Those are used by svn-fe to one-way import \nsvn history to git. svn-fe doesn't do branch mapping yet.\n\nWe have Ramkumar Ramachandra's svnrdump [8] which now lives in the svn source \ntree. It can create dump files [9] from remote svn servers and load dump files \nup to svn server.\nIt practically provides read-write access to svn using a text format.\n\nThere is a prototype remote helper from Dmitry Ivankov. A bash script \nproviding one way fetching from svn via svnrdump and svn-fe.\n\n{ did I miss something important? }\n\n==Project outline==\nPlease look at the drawing on:\nhttp://filestore.mg34.vc-graz.ac.at/flo/drawing.svg\n\n1. Write a new bi-directional remote helper in C. \n - It uses vcs-svn utilities to convert svn dumps to git-fast-import and \nvice-versa.\n - It calls svnrdump as a backend to communicate with svn.\n - It reads a configuration file containing branch mappings according to [6]. \nThese mapping have to be pre-generated using tools developed along with the \nlanguage. The remote helper has no way of asking the user what to do. It will \nfail if a mapping is unclear.\n - Because generating the branch mapping configuration already requires that \nyou have a dump of the svn repo, the helper should probably be able to read \nfrom a file in place of svnrdump too.\n - Using the config the helper translates svn branches/tags to git \nbranches/tags and converts other metadata as applicable. It probably has to \nstore some information about the mapping in a file in .git to allow a \nreconstruction on subsequent invocations. I think this is especially important \nwhen pushing to branches (does it already exist in svn, and where? is it new).\n - It communicate with git via the fast-import format. The remote helper \ninterface (will have)|has commands for that.\n\n2. Extend the remote helper interface as necessary to read and write fast-\nimport streams to remote helpers\n\n3. Add output capabilities to vcs-svn. Currently the code in vcs-svn can only \nconvert svn to git. To push to svn we also need conversion and mapping from \ngit to svn. The actual mapping code for branches should also be placed here \n{??} and called by the remote helper.\n\n{ Hmm.. so it looks like thats a lot? what do you think? }\n\nTimeline\n{ Still to come !}\n\nAbout me\n{ I sent an introduction to the list already, so I'll not copy it here. But it \nwill be in the application on GSOC site.}\n\n[1] http://git-scm.com/\n[2] http://subversion.tigris.org/\n[3] git sources git/Documentation/git-remote-helpers.html\n[4] git sources git/Documentation/git-fast-import.html\n[5] http://thread.gmane.org/gmane.comp.version-control.git/192106\n[6] https://github.com/andrew-sayers/SVN-Branching-Language\n[7] http://esr.ibiblio.org/?p=4071\n[8] http://svnbook.red-bean.com/en/1.7/svn.ref.svnrdump.html\n[9] svn sources subversion/notes/dump-load-format.txt\n[10] https://github.com/divanorama/git/blob/remote-svn-alpha/contrib/svn-\nfe/git-remote-svn-alpha\n"},{"id":"188303","messageId":"CALkWK0nYhfPQrwrrnGP21za6CkryuzfV+ki+WKUdtA1BaLHTsA@mail.gmail.com","threadId":"29992","inReplyTo":"2487557.B8qfnaixh3@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2012-04-02T11:00:14Z","receivedAt":"2012-04-02T11:00:14Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Florian,\n\nFlorian Achleitner wrote:\n> - svn commits are only possible after updating the working copy first, i.e.\n> fetching and merging new revisions on the server. This is like implicitly\n> rebasing your local work on the remote head before pushing to an svn\n> repository.\n\nThis shouldn't worry you, because we don't have a Git -> SVN converter\nyet. However, I have written a prototype svn-fi. Unfortunately, due\nto the way marks work in fast-import, svn-fi is far from complete.\n\nSee: https://github.com/artagnon/git/tree/svn-fi\n\n> Branches exist due to the convention of having branches/, trunk/, and tags/\n> directories in a repository, so do tags. But this is not mandatory and\n> therefore there are many different layouts. It follows that in svn it is also\n> possible to commit across branches. This means that a single commit can change\n> files on more than one branch (accidentally or deliberately).\n> To convert svn branches to git we have to detect branch semantics by examining\n> the svn tree's structure and it's metadata (it has a 'copyfrom' property).\n> Previous efforts show that this will not be possible fully automatically\n> without configuration and interaction with the user.\n\nSee also: http://article.gmane.org/gmane.comp.version-control.git/150007\n\n> \"esr\" developed a tool to manipulate and export subversion repositories [7]\n> that should be able to detect branches, but it's sources are not available\n> yet.\n\nSources are available at git://gitorious.org/reposurgeon/reposurgeon.git\nDo let us know how SBL compares to reposurgeon. Personally, I like\nthe idea of a standard \"language\" to express the mapping.\n\n> In git's source tree we have a vcs-svn/, a set of functions to convert svn\n> dumps to git-fast-import streams. Those are used by svn-fe to one-way import\n> svn history to git. svn-fe doesn't do branch mapping yet.\n\nAre you planning to extend svn-fe to do the mapping, write it as a\nseparate program, or write it into the remote helper? I personally\ndon't mind if the mapping is done in Perl (like in git-svn or SBL) as\nopposed to C; mapping is just parse-intensive.\n\n> 1. Write a new bi-directional remote helper in C.\n> [...]\n>  - It reads a configuration file containing branch mappings according to [6].\n> These mapping have to be pre-generated using tools developed along with the\n> language. The remote helper has no way of asking the user what to do. It will\n> fail if a mapping is unclear.\n\nRight.\n\n>  - Because generating the branch mapping configuration already requires that\n> you have a dump of the svn repo, the helper should probably be able to read\n> from a file in place of svnrdump too.\n\nYou can clone the SVN dumpstream from svnrdump using tee (or similar),\nsending one copy to svn-fe and another to the SBL configuration\ngenerator.\n\n>  - Using the config the helper translates svn branches/tags to git\n> branches/tags and converts other metadata as applicable. It probably has to\n> store some information about the mapping in a file in .git to allow a\n> reconstruction on subsequent invocations. I think this is especially important\n> when pushing to branches (does it already exist in svn, and where? is it new).\n\nHow will the actual mapping be done? Using filter-branch's\nsubdirectory filter, or something else?\n\n> 3. Add output capabilities to vcs-svn. Currently the code in vcs-svn can only\n> convert svn to git. To push to svn we also need conversion and mapping from\n> git to svn. The actual mapping code for branches should also be placed here\n> {??} and called by the remote helper.\n\nI think this bit sounds overtly ambitious. I think if you can build a\nseamless one-way SVN -> Git bridge in one summer, it'll be quite an\nachievement in itself. Finishing and getting svn-fi merged should be\nlast priority; I'll try to work on it myself in summer.\n\n Ram\n"},{"id":"188351","messageId":"20120402205659.GA13725@burratino","threadId":"29992","inReplyTo":"2487557.B8qfnaixh3@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-02T20:57:00Z","receivedAt":"2012-04-02T20:57:00Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Florian,\n\nFlorian Achleitner wrote:\n\n> Here is my draft of the proposal for the GSoC project. RFC!\n> Please comment and tell me what you think and if I understood it all right!\n\nI like the rough idea. I also agree with Ram that the scope seems too\nwide for one summer and think it would be useful to narrow the scope a\nlittle.\n\nSome tasks I can think of:\n\n - getting Dmitry's importer into contrib/ and making sure it works\n reliably. This might require some fixes to svnrdump, svn-fe,\n and the transport-helper. Some known problems that I suspect may\n be still unresolved:\n\n - files marked with both svn:special (symlink) and svn:executable\n\n - dealing with after-the-fact edits to the svn repository. For\n example, revprops including svn:log can be and often are changed\n after the fact.\n\n - what happens when the connection to the Subversion server is\n interrupted? The Subversion dump format does not have an\n \"end of commit\" marker so currently we can get confused and\n seem to succeed.\n\n - svn-fe does not correctly handle revs that change a text file to\n a symlink or vice versa without changing its text.\n\n - UI for importing only some revisions (e.g., \"all revisions after\n r1000\"). Dmitry has a patch for the svn-fe plumbing to handle\n this but I don't think the corresponding change for the remote\n helper has been written.\n\n - this would probably also require changes to svnrdump. What\n happens when r2000 involves copying a file from a version before\n r1000? If imports do not start at r0, normal dumps of r1000:\n are not self-contained.\n\n - UI for storing the mapping between Subversion revision numbers and\n git commit names in the git object db somewhere. Currently we\n store it in a marks file. There is a script floating around to\n convert that marks file into a set of commit notes and Dmitry also\n has a patch for svn-fe to make it write commit notes directly.\n What happens when the notes and marks file go out of sync --- which\n is authoritative?\n\n This also implies that repeated fetches would not have to start\n importing again at r1.\n\n - Storing empty directories and path-specific properties like\n svn:ignore that we don't currently handle.\n\n - Splitting history into branches.\n\n Somehow svn-fe has to communicate \"svn cp\" source and target\n information to the branch mapper so we can trace history to before\n the birth of the paths we are following. That is, the full history\n of branches/1.7.x/ includes the early history of trunk/ if the\n 1.7.x branch was originally created as a copy of the trunk.\n\n This might be able to use mechanism similar to storage of\n empty directories and path properties.\n\n - UI for importing only a subset of paths (e.g., \"just the trunk\").\n\n - this would probably also require changes to svnrdump. What\n happens when r2000 involves copying a file from a branch we\n have chosen not to import?\n\n - Mapping authorship information from Subversion (which usually\n amounts to a remote username) to something more idiomatic in git\n (usually a human's name and email address) in a way that makes\n round trips possible.\n\n - Sharing an imported repository with other users of the remote\n helper.\n\n - this might involve changes to the remote helper machinery to\n allow new clones to use some fetch/push ref specification\n different from refs/heads/*:refs/remotes/origin/*, or it might\n involve some change to core git to automatically push notes\n corresponding to some refs in some situations.\n\n - Importing pairs that have multiple parents. In the\n subversion model, path nodes have only one (copyfrom) parent,\n but repositories can use the svn:mergeinfo property to indicate\n that changes made in certain revs to another patch have been\n incorporated. Under what circumstances is that enough\n justification to add a second parent on the git side?\n\n - Because svn:mergeinfo is a normal path property, the branch\n mapper could have enough information to take care of this with\n the help of the previously mentioned facility for storing path\n properties.\n\nAll of the above is just for reasonable fetch support.\n\nFor push support, one early problem to solve would be that pushing\na commit so that the git commit id from re-importing it is the same\nrequires permission to set the svn:date property. Is our target\naudience one that already has that permission? Is that permission\nsomething reasonable for a committer to ask for from the repository\nadmin in order to use the remote helper?\n\nBecause of the above:\n\n> 1. Write a new bi-directional remote helper in C. \n\nThe word \"new\" makes me worried that you'd be throwing away whatever\nwork already exists. :)\n\n[...]\n> { Hmm.. so it looks like thats a lot? what do you think? }\n\nI agree --- what you've described is more than one summer's worth\nof work. Are there any aspects you're particularly interested in\nfocusing on? For example,\n\n (1) If we focus on repositories without any branching structure at\n all and where the user has full ability to write whatever she\n pleases to the repository, I think developing a bidirectional\n remote helper is feasible during the summer. Round-trip\n support (i.e., commit ids staying the same with a push followed\n by a fetch) is feasible with such a quick plan if we're willing\n to store some git-specific junk in the repo.\n\n (2) Regarding a tool that sits between svn-fe and the remote helper\n and implements the \"follow parent\" rule for tracing the full\n history of a single (linear) branch: I think developing that\n _and_ getting it merged could fit in the summer.\n\n (3) Regarding storing and sharing Subversion's path-specific\n and revision-specific properties: I think implementing a\n mechanism for that and getting it merged could fit in one\n summer.\n\n (4) Regarding getting git weirdness like distinct author and\n committer names, lack of rename information cooked at commit\n time, and timezones in author and committer dates handled during\n pushes to Subversion in a non-invasive way that is user-friendly\n for the pusher likely to be acceptable on the receiving side for\n normal projects: that could certainly fill a summer.\n\n (5) Subversion weirdness like revs that change the entire repository\n at once in a many-branch repo, non-standard file modes, and\n noticing and acting appropriately for svn:log messages that were\n changed after the fact could fill another summer.\n\nSo ideally I would like 5 students working on the remote helper\nproject. ;-)\n\nHope that helps,\nJonathan\n"},{"id":"188360","messageId":"4F7A258C.5000200@pileofstuff.org","threadId":"29992","inReplyTo":"2487557.B8qfnaixh3@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-04-02T22:17:48Z","receivedAt":"2012-04-02T22:17:48Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"Hey Florian,\n\nComments below. The nitpickier ones aren't so much there to help the\nproposal as for general information.\n\nOn 02/04/12 09:30, Florian Achleitner wrote:\n\n> \n> Subversion (svn) [2] was created as a successor of CVS, both follow a strict \n> client-server design, where the repository exclusively lives on the central \n> server and every client only checks out a copy of a single revision at a time. \n> SVN doesn't truly have a concept of branches. SVN branches are a copy of a \n> directory (so are tags).\n\nJust a little nitpick - SVN was primarily inspired by CVS, but there's\nno formal connection between the projects - both are developed by\ndifferent development teams even to this day.\n\n\n> git-fast-import [4] is a format to serialize a git repository into a text \n> format. It is used by the tools git-fast-import and git-fast-export.\n> \n> The remote helper has to convert the foreign protocol and data (svn) to the \n> git-fast-import format.\n\nAs discussed on IRC, I'd like to see some discussion of solutions that\nuse plumbing directly (e.g. git-commit-tree) if you choose to focus on\nbranch import.\n\n\n> Branches exist due to the convention of having branches/, trunk/, and tags/ \n> directories in a repository, so do tags. But this is not mandatory and \n> therefore there are many different layouts. It follows that in svn it is also \n> possible to commit across branches. This means that a single commit can change \n> files on more than one branch (accidentally or deliberately).\n\nThis is basically accurate, but a contrived example might help explain\nwhy fully automatic branch export is impossible in the general case:\n\nImagine a repository that consists of a single revision with a single\nfile, \"scratchpad/libfoo/foo.c\" - how would we decide which directory is\nthe branch? Has the author has even decided yet? For example, he might\nbe learning version control and not understand what branches are.\n\nHaving said that, automatic branch export might be possible in some\nimportant special cases (like repositories that use the standard\nlayout). I haven't really looked into this yet.\n\n\n> - Because generating the branch mapping configuration already requires that \n> you have a dump of the svn repo, the helper should probably be able to read \n> from a file in place of svnrdump too.\n\nIt might help if I explain how the SVN branch exporter will work:\n\nFirst, it will read an SVN dump and create a file containing JSON blobs\nsummarising each revision - e.g. it specifies which files were changed,\nbut not the contents of the changes. As Ram mentioned, downloading the\ndump and tee'ing it to both this process and svn-fe makes a lot of sense.\n\nNext, it will read the JSON file and detect trunks. This turns out to\nbe extremely fast now it's been freed from the SVN dump format.\n\nNext, the user will have the opportunity to review the detected trunks.\n For example, if somebody put a \"README.txt\" in the root directory, the\nprevious step will need to be rerun with that file ignored.\n\nNext, the main branch detection stage will be run using the JSON file\nand the previous branch information.\n\nNext, the user has another chance to make changes. Some users will blow\nstraight past this stage, but sufficiently fussy users with sufficiently\nlarge repositories could spend several days looping through this and the\nprevious stage until their branches and merges are just right.\n\nThe SBL file is finally complete whenever the user decides - you'll need\nto tell them how to restart the import process, in case they restarted\ntheir computer while they were refining the file.\n\n\n> 3. Add output capabilities to vcs-svn. Currently the code in vcs-svn can only \n> convert svn to git. To push to svn we also need conversion and mapping from \n> git to svn. The actual mapping code for branches should also be placed here \n> {??} and called by the remote helper.\n\nI agree with Jonathan and Ram that we're not ready for this yet. Even\nmapping git branches back to a branchless representation won't be\npractical until branch import is fairly mature.\n\n\t- Andrew\n\n[1]https://github.com/andrew-sayers/Proof-of-concept-History-Converter/blob/master/git-branch-import.pl\n[2]git sources git/Documentation/git-commit-tree.txt\n"},{"id":"188362","messageId":"20120402222958.GD13969@burratino","threadId":"29992","inReplyTo":"4F7A258C.5000200@pileofstuff.org","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-02T22:29:59Z","receivedAt":"2012-04-02T22:29:59Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Andrew,\n\nAndrew Sayers wrote:\n> On 02/04/12 09:30, Florian Achleitner wrote:\n\n>> The remote helper has to convert the foreign protocol and data (svn) to the \n>> git-fast-import format.\n>\n> As discussed on IRC, I'd like to see some discussion of solutions that\n> use plumbing directly (e.g. git-commit-tree) if you choose to focus on\n> branch import.\n\nDo you mean that fast-import is not a plumbing command?\n\n>From the IRC log[1]:\n\n> andrew_sayers\tFrom my reading of the protocol, you'd have to pass\n> \tall the files in for each branch.\n> andrew_sayers\tFor each commit.\n\nI'm a little confused by this. Do you mean that a fast-import stream\nis not allowed to use multiple branches, or that when a fast-import\nstream represents a commit that changes one file, it needs to list\nall files rather than the one that changed? Neither is true.\n\nThe fast-import tool started as a tool to write objects to pack\ndirectly, or in other words to save time by avoiding the step of\nwriting loose objects. That is still one of its main benefits.\n\n[...]\n>> 3. Add output capabilities to vcs-svn. Currently the code in vcs-svn can only \n>> convert svn to git. To push to svn we also need conversion and mapping from \n>> git to svn. The actual mapping code for branches should also be placed here \n>> {??} and called by the remote helper.\n>\n> I agree with Jonathan and Ram that we're not ready for this yet.\n\nJust to be clear, I never said such a thing. :)\n\nThanks for some useful clarifications.\nJonathan\n\n[1] http://colabti.org/irclogger/irclogger_log/git-devel?date=2012-04-02#l153\n"},{"id":"188365","messageId":"20120402230452.GE13969@burratino","threadId":"29992","inReplyTo":"20120402205659.GA13725@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-02T23:04:53Z","receivedAt":"2012-04-02T23:04:53Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n\n> - Importing pairs that have multiple parents. In the\n> subversion model, path nodes have only one (copyfrom) parent,\n> but repositories can use the svn:mergeinfo property to indicate\n> that changes made in certain revs to another patch have been\n\nThe above should say \"... changes made in certain revs to another\n_path_ ...\".\n\n> incorporated. Under what circumstances is that enough\n> justification to add a second parent on the git side?\n\nOne subtlety here is that sometimes people merge almost everything\nfrom some branch but leave a few revisions out. Imagine the following\nhistory:\n\n o --- B1 --- B2 --- B3 ---- B4 -- F' ---- B6 --- B7 --- B8 [branch]\n \\ \\ \\\n \\ \\ \\\n T1 --- F ------------------------ M1 ------------------ M2 [trunk]\n\nThe bugfix F was applied to the trunk first and then applied to the\nbranch as rev F'. Then the maintainer merged the remaining changes\nB1, B2, B3, B4 from the branch to trunk. In git this operation would\nbe carried out by running \"git merge branch\". Finally some more\nchanges were made on the branch and the maintainer merged those to\ntrunk, too.\n\nIn subversion, this could be done like so:\n\n 1. Make commit T1 on trunk.\n 2. Make commit F on trunk.\n 3. Make commits B1, B2, B3, B4 on branch.\n 4. Make commit F' on branch, either using \"svn merge\" or by hand.\n 5. Merge changes B1, B2, B3, B4 from branch to trunk using\n \"svn merge -r o:B4 \" and commit.\n 6. Make commits B6, B7, B8 on branch.\n 7. Merge changes B6, B7, B8 from branch to trunk using\n \"svn merge -r F':B8 \" and commit.\n\nThe resulting svn:mergeinfo property on trunk in revision M1 would\nlook like this:\n\n\t/branches/branch:B1-B4\n\nTo a naive importer, this looks like a merge of B4. The svn:mergeinfo\nproperty on trunk in revision M2 would look like this:\n\n\t/branches/branch:B1-B4,B6-B8\n\nwhich looks like a bunch of cherry-picks rather than a merge, since\nit looks like this almost-merge leaves out F'.\n\nIf the maintainer used \"svn merge --reintegrate\" instead, the\nsvn:mergeinfo properties are a little simpler, so maybe I am worrying\nfor no good reason. Anyway, hopefully that makes the setup a little\nclearer.\n"},{"id":"188369","messageId":"4F7A3450.7000302@pileofstuff.org","threadId":"29992","inReplyTo":"20120402222958.GD13969@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-04-02T23:20:48Z","receivedAt":"2012-04-02T23:20:48Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"On 02/04/12 23:29, Jonathan Nieder wrote:\n> Hi Andrew,\n> \n> Andrew Sayers wrote:\n>> On 02/04/12 09:30, Florian Achleitner wrote:\n> \n>>> The remote helper has to convert the foreign protocol and data (svn) to the \n>>> git-fast-import format.\n>>\n>> As discussed on IRC, I'd like to see some discussion of solutions that\n>> use plumbing directly (e.g. git-commit-tree) if you choose to focus on\n>> branch import.\n> \n> Do you mean that fast-import is not a plumbing command?\n\nSorry, that wasn't clear. I meant commands that just expose a single\nprimitive bit of functionality (like git-commit-tree) instead of those\nthat present an abstract interface to the whole git machinery (like\ngit-fast-import). I'm not sure what the right word for that would be?\n\nI agree it's possible to use fast-import for this problem, but it seems\nlike it's redundant after svn-fe has already loaded everything into git.\n For example, if svn-fe loaded three revisions into the master branch,\nyou could create a trunk branch by doing something like:\n\nCOMMIT=$( git show -s --pretty=%b master^^ | \\\n git commit-tree master^^:trunk )\nCOMMIT=$( git show -s --pretty=%b master^ | \\\n git commit-tree master^:trunk -p $COMMIT )\nCOMMIT=$( git show -s --pretty=%b master | \\\n git commit-tree master:trunk -p $COMMIT )\necho $COMMIT > .git/refs/heads/foo\n\nThe point I was making in IRC was that (so far as I understand)\nfast-import doesn't let you pass trees around in this way, but instead\nrequires you to transmit the contents of all the changed files.\n\nThe code above could of course be achieved more easily with\ngit-filter-branch, or achieved more efficiently with a custom bit of C.\n I suggested discussing the problem in terms of single-purpose commands\nbecause it strikes me as about the right level to expose the\narchitectural questions without getting bogged down in detail.\n\n\t- Andrew\n"},{"id":"188371","messageId":"20120403000945.GA15075@burratino","threadId":"29992","inReplyTo":"4F7A3450.7000302@pileofstuff.org","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-03T00:09:45Z","receivedAt":"2012-04-03T00:09:45Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Andrew Sayers wrote:\n\n> Sorry, that wasn't clear. I meant commands that just expose a single\n> primitive bit of functionality (like git-commit-tree) instead of those\n> that present an abstract interface to the whole git machinery (like\n> git-fast-import).\n\nOk. I think you are misunderstanding the purpose of fast-import[1] but\nit doesn't take away from what you're saying.\n\n> I agree it's possible to use fast-import for this problem, but it seems\n> like it's redundant after svn-fe has already loaded everything into git.\n\nRight, I missed your point here before. The fundamental question is\nnot about what commands to use but about the order of operations.\n\n1. In one scheme, first you import the whole tree without splitting it\n into branches, with a tool like svn-fe. Afterwards, you\n postprocess the resulting repository with tools like \"git\n filter-branch --subdirectory-filter\". The result of the import can\n depend on all revisions --- you can say, in rev 1, \"I'm not sure\n whether this new directory is a branch; let me see how it develops\n by rev 1000 to decide how to process it\".\n\n2. In another scheme, you only import the subset of the repository\n you are interested in. This is what git-svn does, for example.\n This requires the branch discovery to happen at the same time as\n the import, because otherwise there is no way to tell what subset\n of the repository you are actually interested in.\n\n3. Lastly, in yet another scheme, you import the whole tree and it is\n split into branches on the fly. The advantages relative to (1) are:\n\n - impatient people can peek at the partial result of the import as\n it happens\n\n - the result of importing rev n is guaranteed to depend only on\n revs <= n, so different people importing at different times will\n get the same commits (assuming nobody is rewriting early history\n behind the scenes) and it is obvious how to support incremental\n importants to expand a repository with all revs <= n to a\n repository with all revs <= 2n\n\n However, if splitting branches only can happen during the initial\n import, that makes it harder to tweak the configuration and try\n again to see what changes.\n\nThe relevant technical difference is that in the naive implementation\nof scheme (2) you can make use of arbitrary information available over\nsvn protocol, in naive scheme (3) you can only use information that\nmakes it into the fast-import stream, and in naive scheme (1) you can\nonly use information that makes it into the actual git repository. So\nto use scheme (1) you need to make sure svn-fe stores all interesting\ndata in a visible way, including copyfrom info (which is not a bad\nidea anyway).\n\n[...]\n> The point I was making in IRC was that (so far as I understand)\n> fast-import doesn't let you pass trees around in this way, but instead\n> requires you to transmit the contents of all the changed files.\n\nfast-import's \"ls\" command allows exactly what you are talking about,\nand svn-fe uses it to copy subtrees from earlier revs into later ones\nwhen it receives an \"svn cp\" command.\n\nSee [2] for some work that preexists that.\n\nDid I understand correctly?\nJonathan\n\n[1] By acting as a single process that takes a stream of commands it\nreally is able to do something that no other plumbing command can do.\n[2] http://thread.gmane.org/gmane.comp.version-control.git/158375\n"},{"id":"188393","messageId":"2576556.3d3popQR3z@flomedio","threadId":"29992","inReplyTo":"20120402205659.GA13725@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-03T07:49:40Z","receivedAt":"2012-04-03T07:49:40Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi!\n\nI'm curiously watching the discussion I kicked off with my proposal. Before \nrefining the proposal I think I will let the discussion continue at the moment.\nBut just to clarify some things:\nYou know I'm rather new to this topic. I've used svn and git, I know what git \nplumbing is about, but I haven't used plumbing commands to write something \ninto git yet. So I can't tell from experience if it would be good or not, \ncompared to fast-import.\nSo please explain what's the advantage/disadvantage of which design decision.\nThat makes it easier to get the point.\n\nI'm also not yet familiar with svn's internals and what properties they use \nfor what. \nSo there are several questions I simply don't have an answer for.\nI know that you have discussed several issues in a huge lot of mails on this \nlist. I'm watching and learning currently.\n\nJonathan wrote about a script \"floating around\". What's that?\nIs it somewhere in a tree in some repo, is at a patch somewhere in a mail on \nthe list, is it in git.git in some branch?? \nHow does one find catch floating scripts?\n\nAnd two clarifications about what I meant in the proposal:\n\nOn Monday 02 April 2012 16:30:14 Ramkumar Ramachandra wrote:\n> Are you planning to extend svn-fe to do the mapping, write it as a\n> separate program, or write it into the remote helper? I personally\n> don't mind if the mapping is done in Perl (like in git-svn or SBL) as\n> opposed to C; mapping is just parse-intensive.\n\nI personally don't like Perl. :p (I would use python if i need a scripting \nlanguage).\nAs far as I've seen, svn-fe is a 5-liner calling functions in vcs-svn/. So I \nthought there is no point of piping something through svn-fe in the remote-\nhelper. I thought I would use those functions like svn-fe does.\nI thought about vcs-svn/ being a library for svn interaction that the remote-\nhelper, and svn-fe, and svn-fi (?) are using.\n\nOn Monday 02 April 2012 15:57:00 Jonathan Nieder wrote:\n> Florian Achleitner wrote:\n> Because of the above:\n> > 1. Write a new bi-directional remote helper in C.\n> \n> The word \"new\" makes me worried that you'd be throwing away whatever\n> work already exists. :)\n\nProbably I missed something. \nBut all I've seen that is directly a remote-helper is a bash script which \nbasically calls a pipeline from svnrdump | svn-fe | fast-import [2]. \nI'm not planning to write a longer program in bash. (I personally use bash \nonly for things that fit on one terminal height).\n\nBash and Perl are not my favourites ;)\n\n[1] https://github.com/divanorama/git/blob/remote-svn-alpha_v2/contrib/svn-\nfe/git-remote-svn-alpha\n\nCheers,\nFlorian\n"},{"id":"188426","messageId":"20120403184803.GH15589@burratino","threadId":"29992","inReplyTo":"2576556.3d3popQR3z@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-03T18:48:03Z","receivedAt":"2012-04-03T18:48:03Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nFlorian Achleitner wrote:\n\n> You know I'm rather new to this topic. I've used svn and git, I know what git \n> plumbing is about, but I haven't used plumbing commands to write something \n> into git yet. So I can't tell from experience if it would be good or not, \n> compared to fast-import.\n\nYes, no problem. I think the question of using fast-import or other\ncommands is not a fundamental one.\n\n> So please explain what's the advantage/disadvantage of which design decision.\n> That makes it easier to get the point.\n\nThe main advantages of using fast-import are:\n\n - it's faster (assuming it works correctly) :)\n - there are backends for version control systems other than git\n - remote helpers can declare the export/import capabilities to support other\n version control systems, instead of declaring fetch/push and supporting\n only git\n\nHowever, whatever tools you use, the immediate idea is to transfer\ndata between a Subversion repository and a Git repository, and the\nproblems to be solved are the same.\n\n[...]\n> I'm also not yet familiar with svn's internals and what properties they use\n> for what.\n> So there are several questions I simply don't have an answer for.\n> I know that you have discussed several issues in a huge lot of mails on this\n> list. I'm watching and learning currently.\n\nThe svnbook at http://svnbook.red-bean.com/, the Subversion lists at\n, and the #svn-dev\nIRC channel on freenode\n are the best\nresources I know for questions in that vein.\n\nI also learned a lot from looking at the dump format that \"svnadmin\ndump\" spits out, since it matches Subversion concepts pretty well. It\nis documented at\n\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\n\nSome basic design questions are covered in the thread starting at\n\n http://thread.gmane.org/gmane.comp.version-control.git/159054\n\n> Jonathan wrote about a script \"floating around\". What's that?\n\nI think you mean the marks-to-notes converter. One version is at\n\n http://thread.gmane.org/gmane.comp.version-control.git/163395/focus=168514\n\n[...]\n> On Monday 02 April 2012 16:30:14 Ramkumar Ramachandra wrote:\n\n>> Are you planning to extend svn-fe to do the mapping, write it as a\n>> separate program, or write it into the remote helper? I personally\n>> don't mind if the mapping is done in Perl (like in git-svn or SBL) as\n>> opposed to C; mapping is just parse-intensive.\n>\n> I personally don't like Perl. :p (I would use python if i need a scripting \n> language).\n> As far as I've seen, svn-fe is a 5-liner calling functions in vcs-svn/. So I \n> thought there is no point of piping something through svn-fe in the remote-\n> helper. I thought I would use those functions like svn-fe does.\n> I thought about vcs-svn/ being a library for svn interaction that the remote-\n> helper, and svn-fe, and svn-fi (?) are using.\n\nYes, I think when Ram added vcs-svn/ to the main git repository, the\nintent was to make it a library that some git-remote-svn.c could use\ndirectly.\n\n[...]\n> On Monday 02 April 2012 15:57:00 Jonathan Nieder wrote:\n\n>> The word \"new\" makes me worried that you'd be throwing away whatever\n>> work already exists. :)\n>\n> Probably I missed something. \n> But all I've seen that is directly a remote-helper is a bash script which \n> basically calls a pipeline from svnrdump | svn-fe | fast-import [2]. \n> I'm not planning to write a longer program in bash. (I personally use bash \n> only for things that fit on one terminal height).\n>\n> Bash and Perl are not my favourites ;)\n\nI think that's fine. It's a prototype, and it has -alpha in its name\nto make sure people understand there are no compatibility guarantees\nwhich avoids constraining us. What I was more worried about is\nthrowing away discoveries made in the previous design and starting\nover.\n\nJonathan\n"},{"id":"188441","messageId":"4F7B7169.4050507@pileofstuff.org","threadId":"29992","inReplyTo":"20120403000945.GA15075@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-04-03T21:53:45Z","receivedAt":"2012-04-03T21:53:45Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"On 03/04/12 01:09, Jonathan Nieder wrote:\n> Andrew Sayers wrote:\n> \n>> Sorry, that wasn't clear. I meant commands that just expose a single\n>> primitive bit of functionality (like git-commit-tree) instead of those\n>> that present an abstract interface to the whole git machinery (like\n>> git-fast-import).\n> \n> Ok. I think you are misunderstanding the purpose of fast-import[1] but\n> it doesn't take away from what you're saying.\n\nI had certainly missed the \"ls\" command - having seen that, I agree\nfast-import is the best solution to this problem. I'm still a bit\nconcerned about fast-import as a learning tool, although this is a bit\nof a meta-conversation as far as GSoC is concerned.\n\nPersonally, I like to learn things by understanding the basic building\nblocks, then seeing how to construct things from them. I found git easy\nto learn because I could start with the basic data structures and\nalgorithms, then layer an approximation of a patches-and-tarballs\nworkflow on top of it. I would expect a discussion of the problem in\nterms of primitive commands like git-commit-tree to help that learning\nstyle, although I am committing a logical fallacy by assuming that\neveryone thinks like me until proven otherwise :)\n\nI think a lot of learners want to play a bit, make some informative\nmistakes, then flesh out their understanding with something a bit more\ntechnical. People that want to \"look under the hood\" are well-served by\ngit, because they can use the ordinary interface\n(status/commit/branch/etc.) then use the source when they're ready. It\nseems like people that want to \"peek behind the curtain\" at a\ncommunication stream would be well-served by fast-import if only there\nwas a curtain for them to peek behind. I'd be intereseted to know what\ngit learners think, but I'd feel more comfortable pointing students at\nfast-import if there was a FUSE module, or a shell, or some other\ninterface on top of it whose failure mode was a puzzling mess instead of\na safely inert repository.\n\nIncidentally Florian, some of the above probably spoke to you, other\nbits probably less so. It took me several years after leaving\nuniversity to see my own learning style, so if you find it hard to learn\ngit one way, try some different approaches before assuming it's a\npersonal problem :)\n\n>> I agree it's possible to use fast-import for this problem, but it seems\n>> like it's redundant after svn-fe has already loaded everything into git.\n> \n> Right, I missed your point here before. The fundamental question is\n> not about what commands to use but about the order of operations.\n> \n> 1. In one scheme, first you import the whole tree without splitting it\n> into branches, with a tool like svn-fe. Afterwards, you\n> postprocess the resulting repository with tools like \"git\n> filter-branch --subdirectory-filter\". The result of the import can\n> depend on all revisions --- you can say, in rev 1, \"I'm not sure\n> whether this new directory is a branch; let me see how it develops\n> by rev 1000 to decide how to process it\".\n> \n> 2. In another scheme, you only import the subset of the repository\n> you are interested in. This is what git-svn does, for example.\n> This requires the branch discovery to happen at the same time as\n> the import, because otherwise there is no way to tell what subset\n> of the repository you are actually interested in.\n> \n> 3. Lastly, in yet another scheme, you import the whole tree and it is\n> split into branches on the fly. The advantages relative to (1) are:\n> \n> - impatient people can peek at the partial result of the import as\n> it happens\n> \n> - the result of importing rev n is guaranteed to depend only on\n> revs <= n, so different people importing at different times will\n> get the same commits (assuming nobody is rewriting early history\n> behind the scenes) and it is obvious how to support incremental\n> importants to expand a repository with all revs <= n to a\n> repository with all revs <= 2n\n> \n> However, if splitting branches only can happen during the initial\n> import, that makes it harder to tweak the configuration and try\n> again to see what changes.\n> \n\nThat's a good way of putting the question, but for SVN it's useful to\ndistinguish between trunk and non-trunk branches. I previously[1]\nsuggested this algorithm for deciding if a directory is a branch:\n\nA directory is a branch if...\n1. it is not a subdirectory of an existing branch; and\n2. either:\n2a. it is in a list of branches specified by the user, or\n2b. it is copied from a (subdirectory of a) branch\n\nThis is a pretty solid heuristic for detecting branches copied from an\nexisting branch even in scheme (2) or (3), but does absolutely nothing\nfor trunk detection. Although trunk detection is trivial in the sane\ncase (the \"trunk\" directory is the one and only trunk, end of story),\nhere's a contrived example for why it's hard in the general case:\n\nOur SVN newbie created \"scratchpad/libfoo/foo.c\" in revision 1. He\nspends the next 1,000 revisions working in scratchpad/libfoo, creating\nthe fooiest foo that ever did foo. After that, he creates\n\"scratchpad/libbar/bar.c\" and continues for another thousand revisions.\n This cycle repeats until he's finally ready to tie all his libraries\ntogether. It's only now that he finally decides whether to create\n\"scratchpad/main.c\" (if he thinks \"scratchpad\" is the trunk), or\n\"trunk/main.c\" (if he thinks all the subdirectories of scratchpad were\ntrunks) or \"scratchpad/main/main.c\" (if he wants to give me an aneurysm\nworrying how to cope when he does `svn cp scratchpad/main scratchpad`).\n\nI paused after writing the paragraph above, because the last part got me\nthinking. Copying a subdirectory to its parent directory isn't actually\npossible in SVN, but the concept of \"branch absorption\" is an\ninteresting one. In theory, we could say that \"scratchpad/libfoo\" and\n\"scratchpad/libbar\" were trunk branches at first, but were deleted when\nthe \"scratchpad\" branch was created. I'll have to check whether this\nleads to undesirable results in the real world, but this might make it\npossible to do on-the-fly trunk detection as described in scheme (3).\n\n> The relevant technical difference is that in the naive implementation\n> of scheme (2) you can make use of arbitrary information available over\n> svn protocol, in naive scheme (3) you can only use information that\n> makes it into the fast-import stream, and in naive scheme (1) you can\n> only use information that makes it into the actual git repository. So\n> to use scheme (1) you need to make sure svn-fe stores all interesting\n> data in a visible way, including copyfrom info (which is not a bad\n> idea anyway).\n\nThe approach I'm looking at is to extract information from an SVN dump\nat an early stage, then use the extracted information when the user\ntidies up the SBL file. This was originally a simple optimisation\n(reading a small gzipped JSON file is much faster than reading an SVN\ndump that's 99% file bodies you don't care about) but it wouldn't be too\nhard to teach svn-fe how to produce the file if you were so inclined.\n\n\t- Andrew\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/192286\n"},{"id":"188446","messageId":"20120403222100.GA20252@burratino","threadId":"29992","inReplyTo":"4F7B7169.4050507@pileofstuff.org","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-03T22:21:00Z","receivedAt":"2012-04-03T22:21:00Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Andrew Sayers wrote:\n\n> This is a pretty solid heuristic for detecting branches copied from an\n> existing branch even in scheme (2) or (3), but does absolutely nothing\n> for trunk detection. Although trunk detection is trivial in the sane\n> case (the \"trunk\" directory is the one and only trunk, end of story),\n> here's a contrived example for why it's hard in the general case:\n\nFor the remote helper in its default configuration, I think it's ok to\nassume the standard layout (trunk/, branches/*, tags/*).\n\nThanks for some useful examples.\n\nSincerely,\nJonathan\n"},{"id":"188562","messageId":"1421035.yALBSXSHGd@flomedio","threadId":"29992","inReplyTo":"2487557.B8qfnaixh3@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-05T13:36:40Z","receivedAt":"2012-04-05T13:36:40Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi everybody!\n\nThanks for your inputs. I've now submitted a slightly updated version of my \nproposal to google. Additionally it's on github [1].\n\nSummary of diffs:\nI'll concentrate on the fetching from svn, writing a remote helper without \nbranch detection (like svn-fe) first, and then creating the branch mapper.\n\n[1] https://github.com/flyingflo/git/wiki/\n\n-- Florian\n\nOn Monday 02 April 2012 10:30:58 Florian Achleitner wrote:\n\n> \n> ==Remote helper for Subversion==\n> \n"},{"id":"188566","messageId":"CA+gfSn8_x7OPUmQ96we3YFhCG8EybKENBY0f+D+dAYQjb8Skzg@mail.gmail.com","threadId":"29992","inReplyTo":"1421035.yALBSXSHGd@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2012-04-05T15:47:22Z","receivedAt":"2012-04-05T15:47:22Z","isPatch":false,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"Hi!\n\nOn Thu, Apr 5, 2012 at 7:36 PM, Florian Achleitner\n wrote:\n> Hi everybody!\n>\n> Thanks for your inputs. I've now submitted a slightly updated version of my\n> proposal to google. Additionally it's on github [1].\n>\n> Summary of diffs:\n> I'll concentrate on the fetching from svn, writing a remote helper without\n> branch detection (like svn-fe) first, and then creating the branch mapper.\nI think that the general goal should include a possibility to clone\n\"svn:// clone\" (not necessarily exactly \"clone\", special easy to use\ncommand/script looks fine too) so that this new clone is able to fetch\nand push too. This is a new feature compared to git-svn.perl and\nallows to share svn->git conversion result. Not completely trivial,\nbut cool to have. At least I recommend to keep it in mind during\ndesign phase(s).\n\nThough it is not a must as there are many many other cool things to\nimplement in git-svn area :)\n\n>\n> [1] https://github.com/flyingflo/git/wiki/\n>\n> -- Florian\n>\n> On Monday 02 April 2012 10:30:58 Florian Achleitner wrote:\n>\n>>\n>> ==Remote helper for Subversion==\n>>\n>\n>\n"},{"id":"188568","messageId":"1333642733-ner-3749@calvin","threadId":"29992","inReplyTo":"20120402205659.GA13725@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Tomas Carnecky","fromEmail":"tomas.carnecky@gmail.com","sentAt":"2012-04-05T16:18:53Z","receivedAt":"2012-04-05T16:18:53Z","isPatch":false,"sender":{"key":"tomas.carnecky@gmail.com","avatar":null},"body":"On Mon, 02 Apr 2012 15:57:00 -0500, Jonathan Nieder wrote:\n> - UI for storing the mapping between Subversion revision numbers and\n> git commit names in the git object db somewhere. Currently we\n\nI wrote a proof-of-concept importer which stored this mapping in notes. Worked\nfairly well. Maybe I can dig up the code again.\n\ntom\n"},{"id":"188809","messageId":"c1cc5fc7-ba1b-447a-9676-53956c5e9dae@mail","threadId":"29992","inReplyTo":"1421035.yALBSXSHGd@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2012-04-09T18:59:41Z","receivedAt":"2012-04-09T18:59:41Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"----- Original Message -----\n> From: \"Florian Achleitner\" \n> Sent: Thursday, April 5, 2012 9:36:40 AM\n> Subject: Re: GSOC Proposal draft: git-remote-svn\n> \n> Thanks for your inputs. I've now submitted a slightly updated version\n> of my proposal to google. Additionally it's on github [1].\n> \n> Summary of diffs:\n> I'll concentrate on the fetching from svn, writing a remote helper\n> without branch detection (like svn-fe) first, and then creating the\n> branch mapper.\n> \n> [1] https://github.com/flyingflo/git/wiki/\n\nFlorian - I just skimmed the github page since I've been away for a week. Not to toot my own horn to much, there's a lot of good discussion about svn-isms in my thread from 2010 (starts at [1], but most of the good stuff is the discussion that follows). I didn't see it in the references, and it probably doesn't need to be there, but if you haven't seen it yet, take a look at it (and cringe at my horrible abuse of git in my early days... ugh!).\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/159054\n\nThanks,\nStephen\n"},{"id":"188886","messageId":"20120410171707.GA3869@burratino","threadId":"29992","inReplyTo":"1421035.yALBSXSHGd@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-10T17:17:07Z","receivedAt":"2012-04-10T17:17:07Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nFlorian Achleitner wrote:\n\n> Thanks for your inputs. I've now submitted a slightly updated version of my \n> proposal to google. Additionally it's on github [1].\n>\n> Summary of diffs:\n> I'll concentrate on the fetching from svn, writing a remote helper without \n> branch detection (like svn-fe) first, and then creating the branch mapper.\n\nThanks for the update.\n\nIf I understand correctly, the remote helper from the first half would\ndo essentially the same thing as Dmitry's remote-svn-alpha script.\nSince in shell script form it is very simple, I don't think it should\ntake more than a couple of days to write such a thing in C.\n\n> Timeline\n>\n> GSoC timeline and summer holidays\n> Summer holidays in Austria at 9th of July. So until the mid-term\n> evaluations my git project will have co-exist with my regular\n> university work and projects. But holidays extend until the beginning\n> of October, so there’s some time left to catch up after the official\n> end of GSoC.\n\nAnother possibility that some people in similar situations have\nfollowed is to start early. That works a little better since it means\nthat by the time midterm evaluations come around we can have a\nreasonable idea of whether a change in strategy is needed for the\nproject to finished on time.\n\n> I plan to split the project in two parts:\n>\n> Writing the remote helper using existing functions in vcs-svn to\n> import svn history without detecting branches, like svn-fe does.\n> Milestone: 9th of July, GSoC mid-term\n>\n> Writing a branch mapper for the remote helper that reads the config\n> language (SBL) and imports branches trying to deal as good as possible\n> with all the little pitfalls that will occur. Milestone: 20th of\n> August, GSoC end\n\nCould you flesh out this timeline more? Ideally it would be nice to\nhave a definite plan here, even to the point of listing what patches\nwould need to be written, so during the summer all that would need to\nhappen is to execute and deal with bugs as they come.\n\nGiven the goal described here of an import with support for\nautomatically detecting branches, here are some rough steps I imagine\nwould be involved:\n\n . baseline: remote helper in C\n\n . option to import starting with a particular numbered revision.\n This would be good practice for seeing how options passed to\n \"git clone -c\" can be read from the config file.\n\n . option or URL schema to import a single project from a large\n Subversion repository that houses several projects. This would\n already be useful in practice since importing the entire Apache\n Software Foundation repository takes a while which is a waste\n when one only wants the history of the Subversion project.\n\n How should the importer handle Subversion copy commands that\n refer to other projects in this case?\n\n . automatically detecting trunk when importing a project with the\n standard layout. The trunk usually is not branched from elsewhere\n so this does not require copyfrom info. Some design questions\n come up here: should the remote helper import the entire project\n tree, too? (I think \"yes\", since copy commands that copy from\n other branches are very common and that would ensure the relevant\n info is available to git.) What should the mapping of git commit\n names to Subversion revision numbers that is stored in notes say\n in this case?\n\n . detecting trunk and branches and exposing them as different remote\n branches. This is a small step that just involves understanding\n how remote helpers expose branches.\n\n . storing path properties and copyfrom information in the commits\n produced by the vcs-svn/ library. How should these be stored?\n For example, there could be a parallel directory structure\n in the tree:\n\n\tfoo/\n\t\tbar.c\n\tbaz/\n\t\tqux.c\n\t.properties/\n\t\tfoo.properties\n\t\tfoo/\n\t\t\tbar.c.properties\n\t\tbaz/\n\t\t\tqux.c.properties\n\n with properites for stored at .properties/.properties.\n This strawman scheme doesn't work if the repository being imported\n has any paths ending with \".properties\", though. Ideas?\n\n . tracing history past branch creation events, using the now-saved\n copyfrom information.\n\n . tracing second-parent history using svn:mergeinfo properties.\n\nIn other words, in the above list the strategy is:\n\n 1. First convert the remote helper to C so it doesn't have to be\n translated again later.\n\n 2. Teach the remote helper to import a single project from a\n repository that houses multiple projects (i.e., path limiting).\n\n 3. Teach the remote helper to split an imported project that uses\n the standard layout into branches (an application of the code\n from (2)). This complicates the scheme for mapping between\n Subversion revision numbers and git commit ids.\n\n 4. Teach the SVN dumpfile to fast-import stream converter not to\n lose the information that is needed in order to get parenthood\n information.\n\n 5. Use the information from step (4) to get parenthood right for a\n project split into branches.\n\n 6. Getting the second parent right (i.e., merges). I mentioned\n this for fun but I don't expect there to be time for it.\n\nDoes that seem right, or does it need tweaks? How long would each\nstep take? Can the steps be subdivided into smaller steps?\n\nAnother question is: what is the design for this? With the existing\nremote-svn-alpha script, there are a few different components with\nwell defined interfaces:\n\n\tcommands like \"git fetch\"\n\t |\n\t | (1)\n\t |\n\ttransport-helper --- (2) --- git fast-import\n\t | |\n\t | (2, 3) |\n\t | |\n\tremote-svn-alpha | (3)\n\t | ''.. |\n\t | (2) ''(2).. |\n\t | ''.. |\n\tsvnrdump --------- (3) -------- svn-fe\n\n (1) communicates using function calls and shared data\n (2) launches\n (3) communicates over pipe\n\nOnce remote-svn-alpha is rewritten in C, the same structure is still\npresent, though it might be less obvious because some of the (2)\nand (3) can change into (1).\n\nWhere does the functionality you are adding fit into this picture?\nAre there any new components being added, and if so what do they take\nas input and output?\n\nHope that helps,\nJonathan\n\n> [1] https://github.com/flyingflo/git/wiki/\n"},{"id":"188913","messageId":"4F84B47D.1080301@pileofstuff.org","threadId":"29992","inReplyTo":"20120410171707.GA3869@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-04-10T22:30:21Z","receivedAt":"2012-04-10T22:30:21Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"On 10/04/12 18:17, Jonathan Nieder wrote:\n\n> Given the goal described here of an import with support for\n> automatically detecting branches, here are some rough steps I imagine\n> would be involved:\n\nJust to be clear, my understanding is that this project will take SBL\ncreated by another program (that I'm writing) and create branches as\nspecified. This frees Florian from having to deal with the maze of edge\ncases involved in that part of the problem.\n\n> \n> . baseline: remote helper in C\n> \n> . option to import starting with a particular numbered revision.\n> This would be good practice for seeing how options passed to\n> \"git clone -c\" can be read from the config file.\n> \n> . option or URL schema to import a single project from a large\n> Subversion repository that houses several projects. This would\n> already be useful in practice since importing the entire Apache\n> Software Foundation repository takes a while which is a waste\n> when one only wants the history of the Subversion project.\n> \n> How should the importer handle Subversion copy commands that\n> refer to other projects in this case?\n\nThis is a good point. I've just svnadmin and svnrdump, and it turns out\nsvnadmin doesn't allow you to dump a subtree while svnrdump strips out\nthe offending copy commands, so either way there's nothing to be done.\n\n> . automatically detecting trunk when importing a project with the\n> standard layout. The trunk usually is not branched from elsewhere\n> so this does not require copyfrom info. Some design questions\n> come up here: should the remote helper import the entire project\n> tree, too? (I think \"yes\", since copy commands that copy from\n> other branches are very common and that would ensure the relevant\n> info is available to git.) What should the mapping of git commit\n> names to Subversion revision numbers that is stored in notes say\n> in this case?\n> \n> . detecting trunk and branches and exposing them as different remote\n> branches. This is a small step that just involves understanding\n> how remote helpers expose branches.\n\nAfter last week's discussion about branch absorption, I tried writing\nanother algorithm over the weekend. I plan to test it during the week,\nbut online detection of branches and trunks looks fairly practical in\nmost real world cases (even those that are sanitily challenged).\n\n> . storing path properties and copyfrom information in the commits\n> produced by the vcs-svn/ library. How should these be stored?\n> For example, there could be a parallel directory structure\n> in the tree:\n\nYes, this is an important problem. It became apparent over the weekend\nthat my code was I/O bound, so I started caching the metadata I need\n(without e.g. file contents) in a gzipped file containing a list of JSON\nblobs (one blob per revision). That immediately caused the script to\njump from about a hundred revisions/second to a few thousand(!), and\neach further size optimisation caused it to jump by another few thousand\nper second.\n\nThis sort of speed is useful for the initial SVN->git conversion,\nbecause it means even people with very large repositories can have a\nquick edit/compile/test loop when they're looking for mis-detected branches.\n\nHaving said all that, a git directory is easier to examine and update\nthan a gzipped file. I have no idea what the performance would be like,\nbut even if a directory was slower we could use gzipped JSON as a cache\nlayer during the initial import, then throw it away and read straight\nfrom a git directory on update.\n\n> . tracing history past branch creation events, using the now-saved\n> copyfrom information.\n\nI'm not sure if I understand correctly, but I think you're referring to\nthis edge case:\n\nmkdir tronk brunches\nsvn add tronk brunches\nsvn ci -m \"Initial commit, with typos to evade stdlayout detection\"\n\nmkdir tronk/libfoo\ntouch tronk/libfoo/main.c\nsvn add tronk/libfoo\nsvn ci -m \"Created libfoo - no way to know this isn't a branch\"\n\nsvn up # so the 'svn cp' works correctly below\n\nsvn cp tronk brunches/copy_of_tronk\ntouch brunches/copy_of_tronk/main.c\nsvn add brunches/copy_of_tronk/main.c\nsvn ci -m \"Marking the copy as a branch, but what about the original?\"\n\n\nI'm not actually sure what the right behaviour is here. You could argue\nthat once we know \"copy_of_tronk\" is a branch, it follows that \"tronk\"\nitself is a branch. On the other hand, these directories have diverged,\nand who's to say it wasn't because of a disagreement about which\ndirectory was the branch?\n\nBranch absorption makes this problem less important - the \"tronk/libfoo\"\nbranch will be deleted and merged into the new \"tronk\" branch the moment\nsomeone creates \"tronk/main.c\", which tends to happen pretty quickly in\nthe real world.\n\nI'm open to suggestions, but my instinct right now is to say that\ncommunicating branchiness back through a copyfrom should at least\nrequire confirmation by the user.\n\n> . tracing second-parent history using svn:mergeinfo properties.\n\nMy old POC code did this, and I plan to include it in the work I'm doing\nnow. I expect this to be the hardest single part of the project to\nsolve in the general case, because of SVN's troubled approach to merge\nhandling.\n\n\n> Another question is: what is the design for this?\n\nHere's my part of the equation:\n\nRight now I have a script that first takes an SVN dump and produces\ngzipped JSON as output, then takes the gzipped JSON as input and\nproduces an SBL file as output. The first round will generally only\nneed to be run once (and is comparable to svn-fe in speed), whereas the\nsecond round might need to be run an arbitrary number of times (but is\nvery fast).\n\nIncidentally, the initial cache generation is the only part that's still\ntied to the SVN dump format, and I doubt it would be that hard for\nsomeone to rewrite it inside svn-fe or to make it read from git metadata\nin future.\n\nI'm currently focussing on bringing all the modules up to release\nquality, so that I can have something for Florian to play with in the\nnear future. This should have an interface that is mature but flexible,\nso I can change the interface to make his life easier but won't need to\nchange the interface because I missed something. After that, I'll\nconcentrate on improving the quality of the SBL output.\n\n\t- Andrew\n"},{"id":"188917","messageId":"20120410234622.GB11506@burratino","threadId":"29992","inReplyTo":"4F84B47D.1080301@pileofstuff.org","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-10T23:46:22Z","receivedAt":"2012-04-10T23:46:22Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Andrew Sayers wrote:\n\n> Just to be clear, my understanding is that this project will take SBL\n> created by another program (that I'm writing) and create branches as\n> specified.\n\nIf that seems like the right thing to do for the people involved\n(Florian and mentor, list consensus) and if that's easy. I'm happy as\nlong as the default configuration works well with sane repositories.\n\n[...]\n> On 10/04/12 18:17, Jonathan Nieder wrote:\n\n>> How should the importer handle Subversion copy commands that\n>> refer to other projects in this case?\n>\n> This is a good point. I've just svnadmin and svnrdump, and it turns out\n> svnadmin doesn't allow you to dump a subtree while svnrdump strips out\n> the offending copy commands, so either way there's nothing to be done.\n\n>From a quick test, it looks like svnrdump converts a directory copy\ninto the addition of its contents. Good.\n\nsvndumpfilter produces\n\n\tsvndumpfilter: E200003: Invalid copy source path '/branches/foo/subdir'\n\nand exits with status 1 so it seems like we're ok.\n\n[...]\n>> . tracing history past branch creation events, using the now-saved\n>> copyfrom information.\n>\n> I'm not sure if I understand correctly, but I think you're referring to\n> this edge case:\n\nNope, I'm talking about the most typical and boring case there is:\n\n\tsvn cp /trunk /branches/topic\n\nWhen cloning , it seems reasonable to expect that the ancestry\nof the trunk and branch would not be shown as disjoint linear histories,\nbut that the revision in which the branch was introduced would be\nshown as a child of the previous revision of the trunk, like so:\n\n\n\t o --- o --- o [topic]\n\t /\n\to --- o --- o --- o --- o --- o [trunk]\n\nThis requires paying attention to copyfrom information.\n\n[...]\n> Right now I have a script that first takes an SVN dump and produces\n> gzipped JSON as output, then takes the gzipped JSON as input and\n> produces an SBL file as output. The first round will generally only\n> need to be run once (and is comparable to svn-fe in speed), whereas the\n> second round might need to be run an arbitrary number of times (but is\n> very fast).\n\nFor what it's worth, for importing from repositories that use a\nnonstandard layout I do think this \"start with a quick pass to figure\nthe layout out\" approach is a sane one.\n\n[...]\n> I'm currently focussing on bringing all the modules up to release\n> quality, so that I can have something for Florian to play with in the\n> near future. This should have an interface that is mature but flexible,\n> so I can change the interface to make his life easier but won't need to\n> change the interface because I missed something. After that, I'll\n> concentrate on improving the quality of the SBL output.\n\nNeat.\n\nThanks for some useful clarifications.\nJonathan\n"},{"id":"188972","messageId":"m3y5q29si7.fsf@localhost.localdomain","threadId":"29992","inReplyTo":"20120410171707.GA3869@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2012-04-11T15:51:16Z","receivedAt":"2012-04-11T15:51:16Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jonathan Nieder writes:\n\n> 2. Teach the remote helper to import a single project from a\n> repository that houses multiple projects (i.e., path limiting).\n> \n> 3. Teach the remote helper to split an imported project that uses\n> the standard layout into branches (an application of the code\n> from (2)). This complicates the scheme for mapping between\n> Subversion revision numbers and git commit ids.\n\nCan't we use the either peg rev notation of externals, or the notation\nthat Subversion itself uses for svn:mergeinfo?\n\n-- \nJakub Narebski\n"},{"id":"188974","messageId":"20120411155537.GA4248@burratino","threadId":"29992","inReplyTo":"m3y5q29si7.fsf@localhost.localdomain","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-11T15:56:00Z","receivedAt":"2012-04-11T15:56:00Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jakub Narebski wrote:\n> Jonathan Nieder writes:\n\n>> 2. Teach the remote helper to import a single project from a\n>> repository that houses multiple projects (i.e., path limiting).\n>>\n>> 3. Teach the remote helper to split an imported project that uses\n>> the standard layout into branches (an application of the code\n>> from (2)). This complicates the scheme for mapping between\n>> Subversion revision numbers and git commit ids.\n>\n> Can't we use the either peg rev notation of externals, or the notation\n> that Subversion itself uses for svn:mergeinfo?\n\nMaybe. ;-) Could you give an example? Where would the text in this\nnotation be stored in the git repository? How are lookups performed?\n"},{"id":"189021","messageId":"1533147.bdVc1SQHSj@flomedio","threadId":"29992","inReplyTo":"4F84B47D.1080301@pileofstuff.org","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-11T19:09:00Z","receivedAt":"2012-04-11T19:09:00Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Tuesday 10 April 2012 23:30:21 you wrote:\n> On 10/04/12 18:17, Jonathan Nieder wrote:\n> \n> \n> > Given the goal described here of an import with support for\n> > automatically detecting branches, here are some rough steps I imagine\n> \n> > would be involved:\n> Just to be clear, my understanding is that this project will take SBL\n> created by another program (that I'm writing) and create branches as\n> specified. This frees Florian from having to deal with the maze of edge\n> cases involved in that part of the problem.\n\nFurthermore the remote-helper has no way of asking the user something, right?\nSo it can only fail if something is ambigous in the svn repository layout. So \nI thought the SBL is exactly to describe these cases, and that's what I need.\n\n> [..]\n"},{"id":"189020","messageId":"2866164.rI5svgrW1x@flomedio","threadId":"29992","inReplyTo":"20120410171707.GA3869@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-11T19:20:31Z","receivedAt":"2012-04-11T19:20:31Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Tuesday 10 April 2012 12:17:07 Jonathan Nieder wrote:\n> Hi,\n> \n> Florian Achleitner wrote:\n> > Thanks for your inputs. I've now submitted a slightly updated version of\n> > my proposal to google. Additionally it's on github [1].\n> > \n> > Summary of diffs:\n> > I'll concentrate on the fetching from svn, writing a remote helper\n> > without branch detection (like svn-fe) first, and then creating the\n> > branch mapper.\n> Thanks for the update.\n> \n> If I understand correctly, the remote helper from the first half would\n> do essentially the same thing as Dmitry's remote-svn-alpha script.\n> Since in shell script form it is very simple, I don't think it should\n> take more than a couple of days to write such a thing in C.\n\nIf the remote-svn-alpha script is really all that needs to be done, you're \nright. It just pipes through svn-fe. I thought svn-fe could only import an svn \nrepo initially, and there would be some difference between importing the whole \nhistory and fetching new revisions later, (?).\n\n> Via\n> > Timeline\n> > \n> > GSoC timeline and summer holidays\n> > Summer holidays in Austria at 9th of July. So until the mid-term\n> > evaluations my git project will have co-exist with my regular\n> > university work and projects. But holidays extend until the beginning\n> > of October, so there’s some time left to catch up after the official\n> > end of GSoC.\n> \n> Another possibility that some people in similar situations have\n> followed is to start early. That works a little better since it means\n> that by the time midterm evaluations come around we can have a\n> reasonable idea of whether a change in strategy is needed for the\n> project to finished on time.\n> \n> > I plan to split the project in two parts:\n> > \n> > Writing the remote helper using existing functions in vcs-svn to\n> > import svn history without detecting branches, like svn-fe does.\n> > Milestone: 9th of July, GSoC mid-term\n> > \n> > Writing a branch mapper for the remote helper that reads the config\n> > language (SBL) and imports branches trying to deal as good as possible\n> > with all the little pitfalls that will occur. Milestone: 20th of\n> > August, GSoC end\n> \n> Could you flesh out this timeline more? Ideally it would be nice to\n> have a definite plan here, even to the point of listing what patches\n> would need to be written, so during the summer all that would need to\n> happen is to execute and deal with bugs as they come.\n\nListing patches and planing all details in the submitted proposal would \nrequire me to know what I do and how I will do it all before last Friday! As \nI'm not yet an expert on this topic, I don't know how I could have known all \ndetails a-priori.\nOf course the project's documentation will evolve outside the GSoC project \nproposal, which cannot be changed anymore.\n\n> \n> Given the goal described here of an import with support for\n> automatically detecting branches, here are some rough steps I imagine\n> would be involved:\n> \n> . baseline: remote helper in C\n> \n> . option to import starting with a particular numbered revision.\n> This would be good practice for seeing how options passed to\n> \"git clone -c\" can be read from the config file.\n> \n> . option or URL schema to import a single project from a large\n> Subversion repository that houses several projects. This would\n> already be useful in practice since importing the entire Apache\n> Software Foundation repository takes a while which is a waste\n> when one only wants the history of the Subversion project.\n> \n> How should the importer handle Subversion copy commands that\n> refer to other projects in this case?\n> \n> . automatically detecting trunk when importing a project with the\n> standard layout. The trunk usually is not branched from elsewhere\n> so this does not require copyfrom info. Some design questions\n> come up here: should the remote helper import the entire project\n> tree, too? (I think \"yes\", since copy commands that copy from\n> other branches are very common and that would ensure the relevant\n> info is available to git.) What should the mapping of git commit\n> names to Subversion revision numbers that is stored in notes say\n> in this case?\n> \n> . detecting trunk and branches and exposing them as different remote\n> branches. This is a small step that just involves understanding\n> how remote helpers expose branches.\n> \n> . storing path properties and copyfrom information in the commits\n> produced by the vcs-svn/ library. How should these be stored?\n> For example, there could be a parallel directory structure\n> in the tree:\n> \n> \tfoo/\n> \t\tbar.c\n> \tbaz/\n> \t\tqux.c\n> \t.properties/\n> \t\tfoo.properties\n> \t\tfoo/\n> \t\t\tbar.c.properties\n> \t\tbaz/\n> \t\t\tqux.c.properties\n> \n> with properites for stored at .properties/.properties.\n> This strawman scheme doesn't work if the repository being imported\n> has any paths ending with \".properties\", though. Ideas?\n> \n> . tracing history past branch creation events, using the now-saved\n> copyfrom information.\n> \n> . tracing second-parent history using svn:mergeinfo properties.\n> \n> In other words, in the above list the strategy is:\n> \n> 1. First convert the remote helper to C so it doesn't have to be\n> translated again later.\n> \n> 2. Teach the remote helper to import a single project from a\n> repository that houses multiple projects (i.e., path limiting).\n> \n> 3. Teach the remote helper to split an imported project that uses\n> the standard layout into branches (an application of the code\n> from (2)). This complicates the scheme for mapping between\n> Subversion revision numbers and git commit ids.\n> \n> 4. Teach the SVN dumpfile to fast-import stream converter not to\n> lose the information that is needed in order to get parenthood\n> information.\n> \n> 5. Use the information from step (4) to get parenthood right for a\n> project split into branches.\n> \n> 6. Getting the second parent right (i.e., merges). I mentioned\n> this for fun but I don't expect there to be time for it.\n> \n> Does that seem right, or does it need tweaks? How long would each\n> step take? Can the steps be subdivided into smaller steps?\n> \n> Another question is: what is the design for this? With the existing\n> remote-svn-alpha script, there are a few different components with\n> well defined interfaces:\n> \n> \tcommands like \"git fetch\"\n> \n> \t | (1)\n> \n> \ttransport-helper --- (2) --- git fast-import\n> \n> \t | (2, 3) |\n> \n> \tremote-svn-alpha | (3)\n> \n> \t | ''.. |\n> \t | \n> \t | (2) ''(2).. |\n> \t | \n> \t | ''.. |\n> \n> \tsvnrdump --------- (3) -------- svn-fe\n> \n> (1) communicates using function calls and shared data\n> (2) launches\n> (3) communicates over pipe\n> \n> Once remote-svn-alpha is rewritten in C, the same structure is still\n> present, though it might be less obvious because some of the (2)\n> and (3) can change into (1).\n> \n> Where does the functionality you are adding fit into this picture?\n> Are there any new components being added, and if so what do they take\n> as input and output?\n\nI planned to implement a remote-helper using the existing interface \nspecification to communicate over pipes with git's transport-helper. \nInstead of invoking svn-fe as a subprocess, I want to call vcs-svn/ functions \ndirectly from the remote-helper and place new functions in this directory (?).\nTo communicate with svn, the remote-helper launches svnrdump as a subprocess.\nAdditionally the remote-helper will read a configuration file containing \nadditional information about branch-mapping, this should be closely related to \nAndrew's SBL.\n\n> \n> Hope that helps,\n> Jonathan\n> \n> > [1] https://github.com/flyingflo/git/wiki/\n\nFlorian\n"},{"id":"189023","messageId":"CA+gfSn8CW_eqnd-wpyLktbJrVeZ993L9MigE7ehiUStGM15gFw@mail.gmail.com","threadId":"29992","inReplyTo":"2866164.rI5svgrW1x@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2012-04-11T19:44:28Z","receivedAt":"2012-04-11T19:44:28Z","isPatch":false,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"On Thu, Apr 12, 2012 at 1:20 AM, Florian Achleitner\n wrote:\n> On Tuesday 10 April 2012 12:17:07 Jonathan Nieder wrote:\n>> Hi,\n>>\n>> Florian Achleitner wrote:\n>> > Thanks for your inputs. I've now submitted a slightly updated version of\n>> > my proposal to google. Additionally it's on github [1].\n>> >\n>> > Summary of diffs:\n>> > I'll concentrate on the fetching from svn, writing a remote helper\n>> > without branch detection (like svn-fe) first, and then creating the\n>> > branch mapper.\n>> Thanks for the update.\n>>\n>> If I understand correctly, the remote helper from the first half would\n>> do essentially the same thing as Dmitry's remote-svn-alpha script.\n>> Since in shell script form it is very simple, I don't think it should\n>> take more than a couple of days to write such a thing in C.\n>\n> If the remote-svn-alpha script is really all that needs to be done, you're\n> right. It just pipes through svn-fe. I thought svn-fe could only import an svn\n> repo initially, and there would be some difference between importing the whole\n> history and fetching new revisions later, (?).\nI've already forgotten the exact details, but svnrdump --incremental\nfrom r0 to rX and then from rX+1 to Y is the same (modulo small dump\nheader) as from r0 to rY. And svn-fe is able to continue like this too\n(maybe some bits of this are not merged, sadly I've forgotten this\ntoo).\n\nA side note is that svnrdump can't do the same trick for rZ, Z>1\n(that's shallow clone) starting point as --incremental may produce\ndelta references to rX, X\n>> Via\n>> > Timeline\n>> >\n>> > GSoC timeline and summer holidays\n>> > Summer holidays in Austria at 9th of July. So until the mid-term\n>> > evaluations my git project will have co-exist with my regular\n>> > university work and projects. But holidays extend until the beginning\n>> > of October, so there’s some time left to catch up after the official\n>> > end of GSoC.\n>>\n>> Another possibility that some people in similar situations have\n>> followed is to start early.  That works a little better since it means\n>> that by the time midterm evaluations come around we can have a\n>> reasonable idea of whether a change in strategy is needed for the\n>> project to finished on time.\n>>\n>> > I plan to split the project in two parts:\n>> >\n>> > Writing the remote helper using existing functions in vcs-svn to\n>> > import svn history without detecting branches, like svn-fe does.\n>> > Milestone: 9th of July, GSoC mid-term\n>> >\n>> > Writing a branch mapper for the remote helper that reads the config\n>> > language (SBL) and imports branches trying to deal as good as possible\n>> > with all the little pitfalls that will occur. Milestone: 20th of\n>> > August, GSoC end\n>>\n>> Could you flesh out this timeline more?  Ideally it would be nice to\n>> have a definite plan here, even to the point of listing what patches\n>> would need to be written, so during the summer all that would need to\n>> happen is to execute and deal with bugs as they come.\n>\n> Listing patches and planing all details in the submitted proposal would\n> require me to know what I do and how I will do it all before last Friday! As\n> I'm not yet an expert on this topic, I don't know how I could have known all\n> details a-priori.\n> Of course the project's documentation will evolve outside the GSoC project\n> proposal, which cannot be changed anymore.\n>\n>>\n>> Given the goal described here of an import with support for\n>> automatically detecting branches, here are some rough steps I imagine\n>> would be involved:\n>>\n>>  . baseline: remote helper in C\n>>\n>>  . option to import starting with a particular numbered revision.\n>>    This would be good practice for seeing how options passed to\n>>    \"git clone -c\" can be read from the config file.\n>>\n>>  . option or URL schema to import a single project from a large\n>>    Subversion repository that houses several projects.  This would\n>>    already be useful in practice since importing the entire Apache\n>>    Software Foundation repository takes a while which is a waste\n>>    when one only wants the history of the Subversion project.\n>>\n>>    How should the importer handle Subversion copy commands that\n>>    refer to other projects in this case?\n>>\n>>  . automatically detecting trunk when importing a project with the\n>>    standard layout.  The trunk usually is not branched from elsewhere\n>>    so this does not require copyfrom info.  Some design questions\n>>    come up here: should the remote helper import the entire project\n>>    tree, too?  (I think \"yes\", since copy commands that copy from\n>>    other branches are very common and that would ensure the relevant\n>>    info is available to git.)  What should the mapping of git commit\n>>    names to Subversion revision numbers that is stored in notes say\n>>    in this case?\n>>\n>>  . detecting trunk and branches and exposing them as different remote\n>>    branches.  This is a small step that just involves understanding\n>>    how remote helpers expose branches.\n>>\n>>  . storing path properties and copyfrom information in the commits\n>>    produced by the vcs-svn/ library.  How should these be stored?\n>>    For example, there could be a parallel directory structure\n>>    in the tree:\n>>\n>>       foo/\n>>               bar.c\n>>       baz/\n>>               qux.c\n>>       .properties/\n>>               foo.properties\n>>               foo/\n>>                       bar.c.properties\n>>               baz/\n>>                       qux.c.properties\n>>\n>>    with properites for stored at .properties/.properties.\n>>    This strawman scheme doesn't work if the repository being imported\n>>    has any paths ending with \".properties\", though.  Ideas?\n>>\n>>  . tracing history past branch creation events, using the now-saved\n>>    copyfrom information.\n>>\n>>  . tracing second-parent history using svn:mergeinfo properties.\n>>\n>> In other words, in the above list the strategy is:\n>>\n>>  1. First convert the remote helper to C so it doesn't have to be\n>>     translated again later.\n>>\n>>  2. Teach the remote helper to import a single project from a\n>>     repository that houses multiple projects (i.e., path limiting).\n>>\n>>  3. Teach the remote helper to split an imported project that uses\n>>     the standard layout into branches (an application of the code\n>>     from (2)).  This complicates the scheme for mapping between\n>>     Subversion revision numbers and git commit ids.\n>>\n>>  4. Teach the SVN dumpfile to fast-import stream converter not to\n>>     lose the information that is needed in order to get parenthood\n>>     information.\n>>\n>>  5. Use the information from step (4) to get parenthood right for a\n>>     project split into branches.\n>>\n>>  6. Getting the second parent right (i.e., merges).  I mentioned\n>>     this for fun but I don't expect there to be time for it.\n>>\n>> Does that seem right, or does it need tweaks?  How long would each\n>> step take?  Can the steps be subdivided into smaller steps?\n>>\n>> Another question is: what is the design for this?  With the existing\n>> remote-svn-alpha script, there are a few different components with\n>> well defined interfaces:\n>>\n>>       commands like \"git fetch\"\n>>\n>>         | (1)\n>>\n>>       transport-helper --- (2) --- git fast-import\n>>\n>>         | (2, 3)                        |\n>>\n>>       remote-svn-alpha                  | (3)\n>>\n>>         |             ''..              |\n>>         |\n>>         | (2)             ''(2)..       |\n>>         |\n>>         |                        ''..   |\n>>\n>>       svnrdump --------- (3) -------- svn-fe\n>>\n>>  (1) communicates using function calls and shared data\n>>  (2) launches\n>>  (3) communicates over pipe\n>>\n>> Once remote-svn-alpha is rewritten in C, the same structure is still\n>> present, though it might be less obvious because some of the (2)\n>> and (3) can change into (1).\n>>\n>> Where does the functionality you are adding fit into this picture?\n>> Are there any new components being added, and if so what do they take\n>> as input and output?\n>\n> I planned to implement a remote-helper using the existing interface\n> specification to communicate over pipes with git's transport-helper.\n> Instead of invoking svn-fe as a subprocess, I want to call vcs-svn/ functions\n> directly from the remote-helper and place new functions in this directory (?).\n> To communicate with svn, the remote-helper launches svnrdump as a subprocess.\n> Additionally the remote-helper will read a configuration file containing\n> additional information about branch-mapping, this should be closely related to\n> Andrew's SBL.\n>\n>>\n>> Hope that helps,\n>> Jonathan\n>>\n>> > [1] https://github.com/flyingflo/git/wiki/\n>\n> Florian\n"},{"id":"189024","messageId":"20120411195351.GG4248@burratino","threadId":"29992","inReplyTo":"2866164.rI5svgrW1x@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-11T19:53:51Z","receivedAt":"2012-04-11T19:53:51Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Florian Achleitner wrote:\n\n> If the remote-svn-alpha script is really all that needs to be done, you're \n> right. It just pipes through svn-fe. I thought svn-fe could only import an svn \n> repo initially, and there would be some difference between importing the whole \n> history and fetching new revisions later, (?).\n\nYes, Dmitry's script (not the first version, but a later one) supports\nincremental imports without trouble if I remember correctly.\n\n[...]\n> Listing patches and planing all details in the submitted proposal would \n> require me to know what I do and how I will do it all before last Friday! As \n> I'm not yet an expert on this topic, I don't know how I could have known all \n> details a-priori.\n\nOh, I didn't mean you would need to do that alone. :) Dmitry, David,\nRam, Sverre, and I should be able to answer any questions you have\nabout how git, vcs-svn, svnrdump, and the transport-helper currently\nwork in the importer.\n\nI've marked the proposal editable to allow details to be filled in.\n\n[...]\n> I planned to implement a remote-helper using the existing interface \n> specification to communicate over pipes with git's transport-helper. \n> Instead of invoking svn-fe as a subprocess, I want to call vcs-svn/ functions \n> directly from the remote-helper and place new functions in this directory (?).\n\nAh, this is a good place to start. In my diagram I lumped everything\nunder vcs-svn/ together as svn-fe for convenience, but in fact the\nvcs-svn lib is made up of multiple components:\n\n\tcaller\n\t .\n\t :\n\t |\n\tpublic interface (svndump_init, svndump_read, etc)\n\t |\n\t |\n\t |\n\tdump file parser (svndump_read body)\n\t |\n\t |\n\t |\n\tfast-export interface (fast_export_*, repo_*) --------- svndiff0 parser\n\t |\n\t :\n\t .\n\tgit fast-import\n\nEach component has a narrow interface. For each action in the dump,\nsvndump_read() calls some appropriate function from the fast-export\ninterface to bring about the corresponding change on the git side.\nDetails of svndump syntax and the state needed to parse it are\nisolated in svndump.c and details of fast-import syntax are in\nfast-export.c and repo_tree.c.\n\n(The structure used to be more complicated when the repo_* functions\nhad to keep track of the repository state instead of relying on\nfast-import for that.)\n\nWhere would the branch mapping go? What kind of state needs to be\nmaintained as it occurs? What steps would I follow to imitate the\ncode and work out a branch mapping on paper? How do I invoke the code\nif I want to try it out (i.e., what functions form the public\ninterface needed to support branch mapping)?\n\nI don't expect you to have answers to all these questions already; I\nunderstand that getting used to what's already there and trying out\nideas will take time. However, I do think we have a much better\nchance of this going well if there are answers to these questions by\nthe time the coding period starts.\n\n[...]\n> Additionally the remote-helper will read a configuration file containing \n> additional information about branch-mapping, this should be closely related to \n> Andrew's SBL.\n\nThat sounds reasonable to me. I am somewhat unconvinced (but\nconvinceable) about the need to use a configuration scheme that\nhandles all the edge cases right away. Shouldn't it be enough to tell\nthe importer the following?\n\n - the path to the repository (from which it can deduce $SVNROOT\n and the path within there to the subproject of interest)\n\n - a single bit of information on top of that: \"this repository uses\n the standard layout\"\n\nOnce that works, the tools could easily be tweaked to respect a\nconfiguration file that describes more complex situations, and as a\nbonus the SBL tools for making sense of those situations would have\ntime to become more mature in the meantime.\n\nThanks for some useful clarifications.\nJonathan\n"},{"id":"189056","messageId":"4F86091F.7010800@pileofstuff.org","threadId":"29992","inReplyTo":"20120411195351.GG4248@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-04-11T22:43:43Z","receivedAt":"2012-04-11T22:43:43Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"On 11/04/12 20:53, Jonathan Nieder wrote:\n> [...]\n>> Additionally the remote-helper will read a configuration file containing \n>> additional information about branch-mapping, this should be closely related to \n>> Andrew's SBL.\n> \n> That sounds reasonable to me. I am somewhat unconvinced (but\n> convinceable) about the need to use a configuration scheme that\n> handles all the edge cases right away. Shouldn't it be enough to tell\n> the importer the following?\n> \n> - the path to the repository (from which it can deduce $SVNROOT\n> and the path within there to the subproject of interest)\n> \n> - a single bit of information on top of that: \"this repository uses\n> the standard layout\"\n> \n> Once that works, the tools could easily be tweaked to respect a\n> configuration file that describes more complex situations, and as a\n> bonus the SBL tools for making sense of those situations would have\n> time to become more mature in the meantime.\n\nSBL itself is just a plain text description of which directories are\nbranches etc. - there are a handful of tricky bits on Florian's side of\nthe fence, but it shouldn't be that hard to add everything necessary to\nparse any arbitrary SBL file. For example, if he gets an SBL action\nthat looks like this:\n\nIn r105, create branch \"/foo\" as \"foo-bar\" from \"/bar/baz\" r25\n\n... then the logic that produced that line doesn't really matter, so\nlong as he can convert it to a series of fast-import commands.\n\nI started work on exporting branches from SVN a few months ago, and\nhappened to be polishing off SBL when GSoC got going, so my work ties\nnicely into Florian's. I've been keen to talk about edge cases lately\nbecause that's the point I'm at in my work - to make a long story short,\nI know how to do the easy cases now, and need to veer off into some\nweird edge cases for a month or two, before swinging back by the\nstandard layout and optimising for that. If Florian needs something\nthat generates SBL before I'm ready, I'd be happy to cobble a basic\n\"standard layout only\" script from the modules I've got.\n\n\t- Andrew\n"},{"id":"189094","messageId":"87ehrtfhmc.fsf@thomas.inf.ethz.ch","threadId":"29992","inReplyTo":"20120411195351.GG4248@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-04-12T09:02:03Z","receivedAt":"2012-04-12T09:02:03Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"[clean up Cc; +Peff]\n\nJonathan Nieder writes:\n\n> I've marked the proposal editable to allow details to be filled in.\n\nThat went wrong, or somebody toggled it back. Since nobody objected\nhere, I'm assuming it was fine, and set it to editable again.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"189108","messageId":"2104868.dCxFQtJHdU@flomedio","threadId":"29992","inReplyTo":"20120410171707.GA3869@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-12T15:28:03Z","receivedAt":"2012-04-12T15:28:03Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi!\n\nLet's discuss the details as suggested by Jonathan! I will collect them in the \nwiki, leading to a more elaborated project plan at the end.\nIt's rather hard to keep an overview over all the issues and pitfalls that may \nexist, and over all the existing discussions, and whether there was an \nsolution or the issue is still unsolved.\nSo I want to create some collection of information with your support.\n\nOn Tuesday 10 April 2012 12:17:07 Jonathan Nieder wrote:\n> Given the goal described here of an import with support for\n> automatically detecting branches, here are some rough steps I imagine\n> would be involved:\n> \n> . baseline: remote helper in C\n> \n> . option to import starting with a particular numbered revision.\n> This would be good practice for seeing how options passed to\n> \"git clone -c\" can be read from the config file.\n\nReally -c? My installed git doesn't have that switch. Should it pass arguments \nto the remote-helper?\n\n> \n> . option or URL schema to import a single project from a large\n> Subversion repository that houses several projects. This would\n> already be useful in practice since importing the entire Apache\n> Software Foundation repository takes a while which is a waste\n> when one only wants the history of the Subversion project.\n> \n> How should the importer handle Subversion copy commands that\n> refer to other projects in this case?\n\nJonathan tried that, it's handled by svnrdump nicely.\n\n> \n> . automatically detecting trunk when importing a project with the\n> standard layout. The trunk usually is not branched from elsewhere\n> so this does not require copyfrom info. Some design questions\n> come up here: should the remote helper import the entire project\n> tree, too? (I think \"yes\", since copy commands that copy from\n> other branches are very common and that would ensure the relevant\n> info is available to git.) What should the mapping of git commit\n> names to Subversion revision numbers that is stored in notes say\n> in this case?\n\nWhat does it mean, \"import the entire project tree\"? Importing other \ndirectories than \"trunk\"?\nAbout the mapping of git commits to svn refs .. I've seen the thread about the \nmarks-to-notes converter.\nBut can somebody please explain what it's for? There is this mark file \nmentioned in the git-fast-import help page ..\n\nDo we create two commits from one revision if it's some special case, like \nmodifying two branches at once?\n\n> \n> . detecting trunk and branches and exposing them as different remote\n> branches. This is a small step that just involves understanding\n> how remote helpers expose branches.\n> \n> . storing path properties and copyfrom information in the commits\n> produced by the vcs-svn/ library. How should these be stored?\n> For example, there could be a parallel directory structure\n> in the tree:\n> \n> foo/\n> bar.c\n> baz/\n> qux.c\n> .properties/\n> foo.properties\n> foo/\n> bar.c.properties\n> baz/\n> qux.c.properties\n> \n> with properites for stored at .properties/.properties.\n> This strawman scheme doesn't work if the repository being imported\n> has any paths ending with \".properties\", though. Ideas?\n\nThis includes collecting which metadata we actually need to store? We could \nprobably collect a list of important svn properties.\n\nIs there a general policy how to store additional metadata for git's helpers? \nI guess it would live somewhere in the .git dir. (.git/info/ ?)\nDmitry mentioned the case where a git repository that fetched from svn is \ncloned, and the cloned repo should be able to fetch from svn too. Is there an \nexisiting concept about metadata in this case?\n\nI'm not sure if storing this in a seperate directory tree makes sense, mostly \nlooking at performance. All these files will only contain some bytes, I guess.\nAndrew, why did you choose JSON?\n\n> \n> . tracing history past branch creation events, using the now-saved\n> copyfrom information.\n> \n> . tracing second-parent history using svn:mergeinfo properties.\n\nThis is about detection when to create a git merge-commit, right?\n\n> \n> In other words, in the above list the strategy is:\n\n.. still to come..\n\nFlorian\n"},{"id":"189153","messageId":"4F875785.6040103@pileofstuff.org","threadId":"29992","inReplyTo":"2104868.dCxFQtJHdU@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-04-12T22:30:29Z","receivedAt":"2012-04-12T22:30:29Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"On 12/04/12 16:28, Florian Achleitner wrote:\n> \n> I'm not sure if storing this in a seperate directory tree makes sense, mostly \n> looking at performance. All these files will only contain some bytes, I guess.\n> Andrew, why did you choose JSON?\n> \n\nJSON has become my default storage format in recent years, so it seemed\nlike the natural thing to use for a format I wanted to chuck in and get\non with my work :)\n\nJSON is my default format because it's reasonably space-efficient,\nhuman-readable, widely supported and can represent everything I care\nabout except recursive data structures (which I didn't need for this\njob). You can do cleverer things if you don't mind being\nlanguage-specific (e.g. Perl's \"Storable\" module supports recursive data\nstructures but can't be used with other languages) or if you don't mind\nneeding special tools (e.g. git's index is highly efficient but can't be\ndebugged with `less`). I've found you won't go far wrong if you start\nwith JSON and pick something else when the requirements become more obvious.\n\nI gzipped the file because JSON isn't *that* space-efficient, and\nbecause very large repositories are likely to produce enough JSON that\npeople will notice. I found that gzipping the file significantly\nreduced its size without having too much effect on run time.\n\nI've attached a sample file representing the first few commits from the\nGNU R repository. The problem I referred to obliquely before isn't with\nJSON, but with gzip - how would you add more revisions to the end of the\nfile without gunzipping it, adding one line, then gzipping it again?\nOne very nice feature of a directory structure is that you could store\nit in git and get all that stuff for free.\n\nTo be clear, I'm not pushing any particular solution to this problem,\njust offering some anecdotal evidence. I'm pretty sure that SVN branch\nexport is an I/O bound problem - David Barr has said much the same about\nsvn-fe, but I was surprised to see it was still the bottleneck with a\nproblem that stripped out almost all the data from the dump and pushed\nit through not-particularly-optimised Perl. Having said that, the\ninitial import problem (potentially hundreds of thousands of revisions\nneeding manual attention) doesn't necessarily want the same solution as\nupdate (tens of revisions that can almost always be read automatically).\n\n>> . tracing history past branch creation events, using the now-saved\n>> copyfrom information.\n>>\n>> . tracing second-parent history using svn:mergeinfo properties.\n> \n> This is about detection when to create a git merge-commit, right?\n\nYes - SVN has always stored metadata about where a directory was copied\nfrom (unlike git, which prefers to detect it automatically), and since\nversion 1.0.5, SVN has added \"svn:mergeinfo\" metadata to files and\ndirectories specifying which revisions of which other files or\ndirectories have been cherry-picked in to them.\n\nIf you know a directory is a branch, \"copyfrom\" metadata is a very\nuseful signal for detecting branches created from it. Unfortunately,\n\"svn:mergeinfo\" is not as useful - aside from anything else, older\nrepositories often exhibit a period where there's no metadata at all,\nthen a gradual migration through SVN's early experiments with merge\ntracking (like svnmerge.py), before everyone gradually standardises on\nsvn:mergeinfo and leaves the other tools behind. Oh, and the interface\ndoesn't tell you about unmerged revisions, so if anybody ever forgets to\nmerge a revision then you'll probably never notice.\n\nI'm planning to tackle this stuff in the work I'm doing, but I expect\npeople will be reporting edge cases until the day the last SVN\nrepository shuts down. You shouldn't need to worry about it much on the\ngit side of SBL, which is probably best for your sanity ;)\n\n\t- Andrew\n"},{"id":"189208","messageId":"20120413191908.GC2387@burratino","threadId":"29992","inReplyTo":"2104868.dCxFQtJHdU@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-04-13T19:19:08Z","receivedAt":"2012-04-13T19:19:08Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi again,\n\nFlorian Achleitner wrote:\n\n> So I want to create some collection of information with your support.\n\nSounds like a plan. Thanks, Florian.\n\n[...]\n> Really -c? My installed git doesn't have that switch. Should it pass arguments \n> to the remote-helper?\n\nWhat git version do you use? \"man git clone\" tells me that -c is an\nabbreviation for --config and \"grep -e --config Documentation/RelNotes/*\"\ntells me it was introduced in v1.7.7.\n\n[...]\n> On Tuesday 10 April 2012 12:17:07 Jonathan Nieder wrote:\n\n>> How should the importer handle Subversion copy commands that\n>> refer to other projects in this case?\n>\n> Jonathan tried that, it's handled by svnrdump nicely.\n\nYes, except that it does not follow the history of the copy source.\nSo if your project was renamed, then \"svnrdump \" will dump a fictional history in which the first rev under\nthe new name created the project out of thin air.\n\nThat is not ideal, but it seems tolerable in the short term.\n\n>> Some design questions\n>> come up here: should the remote helper import the entire project\n>> tree, too? (I think \"yes\", since copy commands that copy from\n>> other branches are very common and that would ensure the relevant\n>> info is available to git.) What should the mapping of git commit\n>> names to Subversion revision numbers that is stored in notes say\n>> in this case?\n>\n> What does it mean, \"import the entire project tree\"? Importing other \n> directories than \"trunk\"?\n\nYes. For an import that is going to be dumping the subdirectories of\ntags/ and branches/ anyway, it seems sensible to ask svnrdump to dump\nthe entire {trunk,tags,branches} hierarchy and sort it out on the git\nside. The question is then: for each rev, in addition to making\ncommits for each branch that changed, should we keep a commit\nrepresenting the state of the combined whole-project tree for internal\nuse? A person trying to check out this commit would get to see the\nenormous\n\n\ttrunk/\n\ttags/\n\tbranches/\n\ndirectories. My rough answer was \"yes, it's convenient to keep that\ninformation around, especially given that with git's repository model\nit doesn't waste a lot of space and makes debugging easier\".\n\n> About the mapping of git commits to svn refs .. I've seen the thread about the \n> marks-to-notes converter.\n> But can somebody please explain what it's for? There is this mark file \n> mentioned in the git-fast-import help page ..\n\nThere are two operations that need to be very fast:\n\n 1. Given a Subversion revision number, what is the corresponding git\n commit? svn-fe uses this to get the preimage data when executing\n an \"svn copy\" operation that refers to an old rev. For example:\n\n\tsvn copy some/path@a-long-time-ago another/path\n\n Code tracking branches would use this same map to find the\n appropriate parent commit for a new branch. For example:\n\n\tsvn copy trunk@a-long-time-ago branches/new-branch\n\n becomes:\n\n\tparent f572d396fae9206628714fb2ce00f72e94f2258f\n\n 2. Given a git commit, what is the corresponding Subversion revision\n number? For example, \"git fetch\" needs this information in order\n to get a first unfetched revision number when updating an existing\n clone of a Subversion repository.\n\n\"git notes\" is a mechanism for efficiently storing a mapping from git\ncommit names to arbitrary data. For example, it can be used to cache\nthe compiled form of some slow-to-compile source code, or it can be\nused to store reminders to a human that has reviewed these commits and\nwanted to scribble a little in the margin. A patch (in Dmitry's tree,\nnot in git.git yet) teaches svn-fe to use the notes facility to store\nthe mapping from git commit names to Subversion revision numbers,\naddressing problem (2) above. Tomas's human-friendly importer used\nthe same trick.\n\nAs you noticed, \"git fast-import\" has a facility that fits well for\nmapping in the other direction: a marks file can store an arbitrary\nmapping from numbers to objects (usually objects that were part of the\nimport). svn-fe writes a mark for each Subversion revision it imports\nto address problem (1) above.\n\nBecause \"git notes\" are stored in the git object db as native objects,\nthey can be shared using the usual \"git fetch\" / \"git push\" commands\nas long as you specify the appropriate source and destination refs on\nthe command line or in git's configuration file. Commands like \"git\nrebase\" that modify history also have some support for carrying notes\nalong. By contrast, a marks file is just a flat text file and there\nis no standard facility for updating it when commit names change or\nsharing it using ordinary git transport.\n\nThe marks-to-notes converter I wrote was a toy to show how the notes\nand marks can easily be kept in sync. If I remember correctly the\nlast time this was discussed there was some feeling that when the two\ntables fall out of synch the notes should be considered authoritative\nand marks can be recomputed from them.\n\n> Do we create two commits from one revision if it's some special case, like \n> modifying two branches at once?\n\nremote-svn-alpha and svn-fe do not currently split by branch at all so\nthe problem doesn't come up.\n\nYes, I think the only sane way to represent a Subversion revision that\nmodifies multiple branches is with a git commit on each branch.\n\n[...]\n>> For example, there could be a parallel directory structure\n>> in the tree:\n>>\n>> foo/\n>> bar.c\n>> baz/\n>> qux.c\n>> .properties/\n>> foo.properties\n>> foo/\n>> bar.c.properties\n>> baz/\n>> qux.c.properties\n>>\n>> with properites for stored at .properties/.properties.\n>> This strawman scheme doesn't work if the repository being imported\n>> has any paths ending with \".properties\", though. Ideas?\n>\n> This includes collecting which metadata we actually need to store? We could \n> probably collect a list of important svn properties.\n\nI imagined the importer just collecting all path properties, like \"git\nsvn\" does in its .git/svn/refs/remotes/git-svn/unhandled.log. They're\neasy to iterate through on the svn side.\n\n> Is there a general policy how to store additional metadata for git's helpers? \n> I guess it would live somewhere in the .git dir. (.git/info/ ?)\n\nOne simple design would be to keep properties in the \"entire project\"\ncommit objects for internal use, since that's easy to share.\n\nI think David had a few other ideas. ;-)\n\n[...]\n>> . tracing second-parent history using svn:mergeinfo properties.\n>\n> This is about detection when to create a git merge-commit, right?\n\nYep. A goal would be to allow a person would be able to push a git\nmerge to an svn repository, fetch from another machine, and get the\nsame commit back.\n\n>> In other words, in the above list the strategy is:\n>\n> .. still to come..\n\nThanks for your thoughtfulness.\n\nJonathan\n"},{"id":"189285","messageId":"1472353.TRfidGPc01@flomedio","threadId":"29992","inReplyTo":"4F875785.6040103@pileofstuff.org","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-14T20:09:39Z","receivedAt":"2012-04-14T20:09:39Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi!\n\nThanks for your explainations.\n\nOn Thursday 12 April 2012 23:30:29 Andrew Sayers wrote:\n> On 12/04/12 16:28, Florian Achleitner wrote:\n> > I'm not sure if storing this in a seperate directory tree makes sense,\n> > mostly looking at performance. All these files will only contain some\n> > bytes, I guess. Andrew, why did you choose JSON?\n> \n> JSON has become my default storage format in recent years, so it seemed\n> like the natural thing to use for a format I wanted to chuck in and get\n> on with my work :)\n> \n> JSON is my default format because it's reasonably space-efficient,\n> human-readable, widely supported and can represent everything I care\n> about except recursive data structures (which I didn't need for this\n> job). You can do cleverer things if you don't mind being\n> language-specific (e.g. Perl's \"Storable\" module supports recursive data\n> structures but can't be used with other languages) or if you don't mind\n> needing special tools (e.g. git's index is highly efficient but can't be\n> debugged with `less`). I've found you won't go far wrong if you start\n> with JSON and pick something else when the requirements become more obvious.\n> \n> I gzipped the file because JSON isn't *that* space-efficient, and\n> because very large repositories are likely to produce enough JSON that\n> people will notice. I found that gzipping the file significantly\n> reduced its size without having too much effect on run time.\n> \n> I've attached a sample file representing the first few commits from the\n> GNU R repository. The problem I referred to obliquely before isn't with\n> JSON, but with gzip - how would you add more revisions to the end of the\n> file without gunzipping it, adding one line, then gzipping it again?\n> One very nice feature of a directory structure is that you could store\n> it in git and get all that stuff for free.\n> \n> To be clear, I'm not pushing any particular solution to this problem,\n> just offering some anecdotal evidence. I'm pretty sure that SVN branch\n> export is an I/O bound problem - David Barr has said much the same about\n> svn-fe, but I was surprised to see it was still the bottleneck with a\n> problem that stripped out almost all the data from the dump and pushed\n> it through not-particularly-optimised Perl. Having said that, the\n> initial import problem (potentially hundreds of thousands of revisions\n> needing manual attention) doesn't necessarily want the same solution as\n> update (tens of revisions that can almost always be read automatically).\n\nJSON seems to be a good initial choice..\n\n> \n> >> . tracing history past branch creation events, using the now-saved\n> >> \n> >> copyfrom information.\n> >> \n> >> . tracing second-parent history using svn:mergeinfo properties.\n> > \n> > This is about detection when to create a git merge-commit, right?\n> \n> Yes - SVN has always stored metadata about where a directory was copied\n> from (unlike git, which prefers to detect it automatically), and since\n> version 1.0.5, SVN has added \"svn:mergeinfo\" metadata to files and\n> directories specifying which revisions of which other files or\n> directories have been cherry-picked in to them.\n> \n> If you know a directory is a branch, \"copyfrom\" metadata is a very\n> useful signal for detecting branches created from it. Unfortunately,\n> \"svn:mergeinfo\" is not as useful - aside from anything else, older\n> repositories often exhibit a period where there's no metadata at all,\n> then a gradual migration through SVN's early experiments with merge\n> tracking (like svnmerge.py), before everyone gradually standardises on\n> svn:mergeinfo and leaves the other tools behind. Oh, and the interface\n> doesn't tell you about unmerged revisions, so if anybody ever forgets to\n> merge a revision then you'll probably never notice.\n\nThis doesn't look very straight forward. In the svn docs they say there is a \ncommand that outputs which changesets are eligible to merge.\nhttp://svnbook.red-\nbean.com/en/1.7/svn.branchmerge.basicmerging.html#svn.branchmerge.basicmerging.mergeinfo\n\nBut I don't know if that helps.\n>\n> I'm planning to tackle this stuff in the work I'm doing, but I expect\n> people will be reporting edge cases until the day the last SVN\n> repository shuts down. You shouldn't need to worry about it much on the\n> git side of SBL, which is probably best for your sanity ;)\n\n:)\n\n> \n> \t- Andrew\n"},{"id":"189286","messageId":"2407098.F5IbgLFxk2@flomedio","threadId":"29992","inReplyTo":"20120413191908.GC2387@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-14T20:15:53Z","receivedAt":"2012-04-14T20:15:53Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi!\n\nThanks for your help.\nI updated the wiki page.\n\nOn Friday 13 April 2012 14:19:08 Jonathan Nieder wrote:\n\n> > Really -c? My installed git doesn't have that switch. Should it pass\n> > arguments to the remote-helper?\n> \n> What git version do you use? \"man git clone\" tells me that -c is an\n> abbreviation for --config and \"grep -e --config Documentation/RelNotes/*\"\n> tells me it was introduced in v1.7.7.\n\nSorry, that was clumsy, I should use build and search the docs of the current \nversion, not the one my distro ships!\n\n> > \n> > What does it mean, \"import the entire project tree\"? Importing other\n> > directories than \"trunk\"?\n> \n> Yes. For an import that is going to be dumping the subdirectories of\n> tags/ and branches/ anyway, it seems sensible to ask svnrdump to dump\n> the entire {trunk,tags,branches} hierarchy and sort it out on the git\n> side. The question is then: for each rev, in addition to making\n> commits for each branch that changed, should we keep a commit\n> representing the state of the combined whole-project tree for internal\n> use? A person trying to check out this commit would get to see the\n> enormous\n> \n> \ttrunk/\n> \ttags/\n> \tbranches/\n> \n> directories. My rough answer was \"yes, it's convenient to keep that\n> information around, especially given that with git's repository model\n> it doesn't waste a lot of space and makes debugging easier\".\n\nSounds reasonable.\n\n> \n> > About the mapping of git commits to svn refs .. I've seen the thread\n> > about the marks-to-notes converter.\n> > But can somebody please explain what it's for? There is this mark file\n> > mentioned in the git-fast-import help page ..\n> \n> There are two operations that need to be very fast:\n> \n> 1. Given a Subversion revision number, what is the corresponding git\n> commit? svn-fe uses this to get the preimage data when executing\n> an \"svn copy\" operation that refers to an old rev. For example:\n> \n> \tsvn copy some/path@a-long-time-ago another/path\n> \n> Code tracking branches would use this same map to find the\n> appropriate parent commit for a new branch. For example:\n> \n> \tsvn copy trunk@a-long-time-ago branches/new-branch\n> \n> becomes:\n> \n> \tparent f572d396fae9206628714fb2ce00f72e94f2258f\n> \n> 2. Given a git commit, what is the corresponding Subversion revision\n> number? For example, \"git fetch\" needs this information in order\n> to get a first unfetched revision number when updating an existing\n> clone of a Subversion repository.\n> \n> \"git notes\" is a mechanism for efficiently storing a mapping from git\n> commit names to arbitrary data. For example, it can be used to cache\n> the compiled form of some slow-to-compile source code, or it can be\n> used to store reminders to a human that has reviewed these commits and\n> wanted to scribble a little in the margin. A patch (in Dmitry's tree,\n> not in git.git yet) teaches svn-fe to use the notes facility to store\n> the mapping from git commit names to Subversion revision numbers,\n> addressing problem (2) above. Tomas's human-friendly importer used\n> the same trick.\n> \n> As you noticed, \"git fast-import\" has a facility that fits well for\n> mapping in the other direction: a marks file can store an arbitrary\n> mapping from numbers to objects (usually objects that were part of the\n> import). svn-fe writes a mark for each Subversion revision it imports\n> to address problem (1) above.\n> \n> Because \"git notes\" are stored in the git object db as native objects,\n> they can be shared using the usual \"git fetch\" / \"git push\" commands\n> as long as you specify the appropriate source and destination refs on\n> the command line or in git's configuration file. Commands like \"git\n> rebase\" that modify history also have some support for carrying notes\n> along. By contrast, a marks file is just a flat text file and there\n> is no standard facility for updating it when commit names change or\n> sharing it using ordinary git transport.\n> \n> The marks-to-notes converter I wrote was a toy to show how the notes\n> and marks can easily be kept in sync. If I remember correctly the\n> last time this was discussed there was some feeling that when the two\n> tables fall out of synch the notes should be considered authoritative\n> and marks can be recomputed from them.\n\nOh, thats intersting, I haven't heard of git notes yet. (I should have greped \nthe Documentation ..). \nBecause of the possibility that one revision is transformed into two commits, \nthe bi-directional mapping has to support 1-to-n or probably n-to-n mappings, \nI think. But this should be possible with these mechanisms.\n\n> \n> > Do we create two commits from one revision if it's some special case,\n> > like modifying two branches at once?\n> \n> remote-svn-alpha and svn-fe do not currently split by branch at all so\n> the problem doesn't come up.\n> \n> Yes, I think the only sane way to represent a Subversion revision that\n> modifies multiple branches is with a git commit on each branch.\n> \n> [...]\n> \n> >> For example, there could be a parallel directory structure\n> >> \n> >> in the tree:\n> >> foo/\n> >> \n> >> bar.c\n> >> \n> >> baz/\n> >> \n> >> qux.c\n> >> \n> >> .properties/\n> >> \n> >> foo.properties\n> >> foo/\n> >> \n> >> bar.c.properties\n> >> \n> >> baz/\n> >> \n> >> qux.c.properties\n> >> \n> >> with properites for stored at\n> >> .properties/.properties.\n> >> This strawman scheme doesn't work if the repository being\n> >> imported\n> >> has any paths ending with \".properties\", though. Ideas?\n> > \n> > This includes collecting which metadata we actually need to store? We\n> > could probably collect a list of important svn properties.\n> \n> I imagined the importer just collecting all path properties, like \"git\n> svn\" does in its .git/svn/refs/remotes/git-svn/unhandled.log. They're\n> easy to iterate through on the svn side.\n\nOk, and it will be useful for pushing to svn in the future.\n\n> \n> > Is there a general policy how to store additional metadata for git's\n> > helpers? I guess it would live somewhere in the .git dir. (.git/info/\n> > ?)\n> \n> One simple design would be to keep properties in the \"entire project\"\n> commit objects for internal use, since that's easy to share.\n> \n> I think David had a few other ideas. ;-)\n\nCommit objects that are actually not commits but store metadata?\n\n\n> \n> [...]\n> \n> >> . tracing second-parent history using svn:mergeinfo properties.\n> > \n> > This is about detection when to create a git merge-commit, right?\n> \n> Yep. A goal would be to allow a person would be able to push a git\n> merge to an svn repository, fetch from another machine, and get the\n> same commit back.\n> \n> >> In other words, in the above list the strategy is:\n> > .. still to come..\n> \n> Thanks for your thoughtfulness.\n> \n> Jonathan\n\nFlorian\n"},{"id":"189287","messageId":"4F89EDBF.9050906@pileofstuff.org","threadId":"29992","inReplyTo":"1472353.TRfidGPc01@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-04-14T21:35:59Z","receivedAt":"2012-04-14T21:35:59Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"On a slightly different topic, here's the only branching edge case I\nknow of that will affect you. I agree with Jonathan that you should\nfocus on the standard layout for now, but I think it's worth having the\ntrickier cases in your head when you're planning things out.\n\nImagine a team does this:\n\n\n# Slight misunderstanding of the standard layout at first:\n\nmkdir trunk/project1 trunk/project2\nsvn add trunk\nsvn ci -m \"Initial revsion\" # r1\n\n# Time passes, commits are made, people get smarter.\n# In revision 1000, the team decides to put the structure right:\n\nsvn rm trunk\nsvn ci -m \"Removed incorrect directory name\" # r1000\n\nmkdir trunk\ntouch trunk/MOVED_TO_PROJECT_TRUNK\nsvn ci -m \"Added signpost file for future reference\" # r1001\n\nmkdir project1 project2\nsvn cp -r 999 trunk/project1 project1/trunk\nsvn cp -r 999 trunk/project2 project2/trunk\nsvn ci -m \"Recreated projects with correct directory names\" # r1002\n\n\nThis would be represented in SBL something like:\n\nIn r1, create branch \"trunk/project1\"\nIn r1, create branch \"trunk/project2\"\n\n# We would prefer just to deactivate these...\nIn r1000, deactivate \"trunk/project1\"\nIn r1000, deactivate \"trunk/project2\"\n\n# ... but we have to delete them,\n# because git doesn't support recursive branch names:\nIn r1001, delete branch \"trunk/project1\"\nIn r1001, delete branch \"trunk/project2\"\nIn r1001, create branch \"trunk\"\n\n# We deleted the branches, so how do we get the commit to fork from?\nIn r1002, create branch \"project1/trunk\" from \"trunk/project1\" r999\nIn r1002, create branch \"project2/trunk\" from \"trunk/project2\" r999\n\n\nIf you look in your \".git/refs/heads/\" directory, you'll see git\nbranches are stored as files on disk. So if you have a branch\n\"trunk/project1\", you can't create a branch called \"trunk\" unless you\ndelete the directory called \"trunk\" first. This unfortunate limitation\nof an otherwise neat solution means you can't reliably use git branches\nwhen retrieving older revisions.\n\nOther people will be able to tell you if there's any interest in\nremoving this limitation, but even if there is, users will occasionally\nchange their mind after asking for a branch to be deleted, and be\nsurprised if SVN lets them but git doesn't.\n\nOne solution you could look at would be storing dead branches in a JSON\nfile somewhere. If you go down that route, remember that `git gc` will\ntry to garbage collect the commits once the branches have been dead for\nlong enough.\n\n\t- Andrew\n"},{"id":"189299","messageId":"4F8A00BE.6020409@pileofstuff.org","threadId":"29992","inReplyTo":"1533147.bdVc1SQHSj@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Andrew Sayers","fromEmail":"andrew-git@pileofstuff.org","sentAt":"2012-04-14T22:57:02Z","receivedAt":"2012-04-14T22:57:02Z","isPatch":false,"sender":{"key":"andrew-git@pileofstuff.org","avatar":null},"body":"On 11/04/12 20:09, Florian Achleitner wrote:\n> Furthermore the remote-helper has no way of asking the user something, right?\n> So it can only fail if something is ambigous in the svn repository layout. So \n> I thought the SBL is exactly to describe these cases, and that's what I need.\n\nSorry, I missed this when it was first posted.\n\nI'm not sure whether the remote helper is allowed to ask the user\nthings, but there can be times when that would be helpful. The one that\njumps to mind is tag handling.\n\nSVN considers tags and branches to be functionally identical, whereas\ngit likes to create \"annotated tags\" (commits with a special tag message\non top of the normal commit message) that can't be changed once they've\nbeen created. So if e.g. a tag is created then later committed to\nagain, what do you do? Do you refuse to make annotated tags in case you\nneed to change them later? Do you ignore later commits so that\nannotated tags work nicely?\n\nSBL can't provide much help here, as a tag could be created in one\nupdate, then committed to again in another update. Last time this was\ndiscussed[1], the consensus seemed to be that there any clever solution\nwould drive straight past \"it just works\" into \"why did it do that?\"\nterritory, so the only sensible solution would be to ask what to do.\n\nAs I say, I don't really know anything about remote helpers, but I'd be\nvery surprised if you weren't allowed to at least fail with a message\nlike \"Please set svn.tagStrategy, see `man git-config` for details\".\n\n\t- Andrew\n\n[1]http://thread.gmane.org/gmane.comp.version-control.git/192106/focus=192286\n"},{"id":"189314","messageId":"b2ae4143-e771-4219-8727-d1c4048b61cc@mail","threadId":"29992","inReplyTo":"4F89EDBF.9050906@pileofstuff.org","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2012-04-15T03:13:34Z","receivedAt":"2012-04-15T03:13:34Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"----- Original Message -----\n> From: \"Andrew Sayers\" \n> Sent: Saturday, April 14, 2012 5:35:59 PM\n> Subject: Re: GSOC Proposal draft: git-remote-svn\n>\n> ... snip ...\n> \n> One solution you could look at would be storing dead branches in a\n> JSON file somewhere. If you go down that route, remember that `git\n> gc` will try to garbage collect the commits once the branches have\n> been dead for long enough.\n\nI don't remember if this has already been discussed, but as I see it there are basically three approaches to closed/deleted SVN branches in the Git world:\n\n 1) Just delete the branch, allow git gc to later cleanup the objects\n 2) Just leave them be for the user to deal with at a later date\n 3) Move them to another namespace\n\nI think (3) is the only semi-tricky one. If you read the git-gc manpage, it turns out gc will consider any object reachable from any ref under refs/ as safe. When cloning/pushing/pulling/etc. git only looks at refs/heads and refs/tags (unless told otherwise). So for our conversion I created refs/hidden/heads and refs/hidden/tags (other choices could be refs/svn or refs/junk, but you get the idea). Just as a fun stat, the hidden namespace in our central repo has 280 refs in it vs 502 in the visible/normal namespace (surprisingly the hidden ones are almost perfectly split with 138 dead Subversion branches and 142 SVN tags that were later retagged/committed to).\n\nThanks,\nStephen\n"},{"id":"189657","messageId":"16489638.QvpMpkdxMd@flomedio","threadId":"29992","inReplyTo":"20120410171707.GA3869@burratino","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-18T20:16:06Z","receivedAt":"2012-04-18T20:16:06Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Tuesday 10 April 2012 12:17:07 Jonathan Nieder wrote:\n> In other words, in the above list the strategy is:\n> \n> 1. First convert the remote helper to C so it doesn't have to be\n> translated again later.\nStore rev <--> commit mappings using marks and notes.\nStore svn metadata.\n> \n> 2. Teach the remote helper to import a single project from a\n> repository that houses multiple projects (i.e., path limiting).\n\nI would plan to have this until the mid-term. From that point my summer \nholidays start ..\n\n> \n> 3. Teach the remote helper to split an imported project that uses\n> the standard layout into branches (an application of the code\n> from (2)). This complicates the scheme for mapping between\n> Subversion revision numbers and git commit ids.\n\nRead ambigouos branches/tags from SBL.\n> \n> 4. Teach the SVN dumpfile to fast-import stream converter not to\n> lose the information that is needed in order to get parenthood\n> information.\n\nThis means actually saving svn:copyfrom properties. (right?)\n\n> \n> 5. Use the information from step (4) to get parenthood right for a\n> project split into branches.\n\n.. and using svn:copyfrom properties. (right?)\n\n> \n> 6. Getting the second parent right (i.e., merges). I mentioned\n> this for fun but I don't expect there to be time for it.\n\nI think this needs a little morge discussion, let's do this if it's the time.\nmergeinfo stores a list of revs merged for a file. This looks like a list of \ngit cherry-picks to me ..\n> \n> Does that seem right, or does it need tweaks? How long would each\n> step take? Can the steps be subdivided into smaller steps?\n\nWhat do you think?\nI will finally add this strategy to the proposal.\n\n-- Florian\n"},{"id":"189696","messageId":"1377122.6Q972gDNtx@flomedio","threadId":"29992","inReplyTo":"16489638.QvpMpkdxMd@flomedio","subject":"Re: GSOC Proposal draft: git-remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner2.6.31@gmail.com","sentAt":"2012-04-19T12:26:03Z","receivedAt":"2012-04-19T12:26:03Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"I have now updated the proposal in the github wiki [1] and on melange.\nMost important change: Added a more detailed timeline.\n\n[1] https://github.com/flyingflo/git/wiki\n\n-- Florian \n"}]}