Re: Update on SoC proposal: git-remote-svn

>
> Hi,
>
> Sam Vilain commented on my SoC proposal using Google's SoC interface,
> and requested me to CC my response to the mailing list. His original
> comment is also quoted below.
>
> ----------------------->8----------------------<8-------------------
> Hi Sam,
>
>> Hi Ramkumar, I've looked at this proposal and seen that it differs a
>> bit from the version on the list, and I can't see the relevant
>> discussion, so I'll just throw my bit in here - though note this is a
>> technical comment and not a critique of your proposal.
>
> There's been a lot of discussion, some on the list, and some more off
> the list.
>
>> First, consider not using the SVN API at all while prototyping the
>> import part of the chain.  Instead, parse the 'svnadmin dump' stream
>> from a local mirror.  This will allow you to tackle the actual
>> problems involved and importing the data effectively, without
>> suffering from the brain-damage that is the SVN API.  After all, the
>> SVN API should be returning you all of the same information that the
>> dump stream does, so you can treat making it work using the remote
>> access API (eg svn_ra_replay, which is faster for mirroring AIUI)  
>> as a
>> separate task.  You will also more easily spot information which you
>> should be extracting from the API, but aren't - it's definitely all  
>> in
>> the dump format; it has to be.  I received similar advice to this
>> before building a perforce importer and let's just say it was
>> invaluable.
>
> Yes. I've studied the SVN API, and I agree with you- it's quite
> horrible. Instead of providing a API that's transparently
> backward-compatible, they've provided different methods for different
> versions. There are also several variations of certain methods, and
> this is quite confusing.
>
> `svnadmin dump` is exactly how I plan to start out- I've already
> discussed this with my to-be mentor, Sverre Rabbelier and David
> Michael-Barr, who's building a new SVN exporter in his own time.
>
>> Second, consider making the mirror phase emit directly to a tracking
>> branch via git-fast-export, that is not intended to be checked out.
>> Instead, it contains trees which correspond to revisions in the
>> mirrored SVN repository.  Directory and file properties can be saved
>> in the tree using specially named dotfiles, and revision properties
>> can be saved in the commit log.  Perhaps I misread your intentions
>> with the "stripped down svnsync" part, but syncing to a local SVN
>> repository seems to me like a waste of time; people can just do that
>> themselves if they choose anyway.  An SVN repository can easily be 10
>> times the size of the corresponding git store, and it just seems like
>> double-handling of the data and will make the whole process slower  
>> and
>> more cumbersome than it needs to be.
>
>> With all the blobs already in the git store, and the information
>> needed to perform the data mining operation which is the extraction  
>> of
>> git-style branch histories from the svn data, you will be working  
>> with
>> data which is all in git-land, and exporting referencing blobs which
>> are already in the store.  This will save you a LOT of time, as it
>> means in this stage you are not handling the actual file images; just
>> constructing branch histories in the git-fast-import stream.  Your
>> branch miner will potentially be able to process thousands of
>> revisions per second this way, even from python.
>
> Agreed. Sverre, David and I discussed exactly this- The final version
> of mirror-client will dump all the SVN information to a Git store
> first, so we can do the mapping painlessly in Git. There are some
> concerns about information loss though, which we'll have to deal with
> as we go on.
>
>> Also bear in mind
>> that people might use SVN in a way that violates the expectations of
>> this branch miner.  An example is putting a README file in the
>> top-level projects directory, a heuristic approach might consider  
>> that
>> the start of a new project and then mess up later stages.  Another
>> example is people accidentally deleting trunk and re-adding it; the
>> nice thing about this two-stage approach is that it allows advanced
>> users to muck with the "raw" data (ie, this whole repository tracking
>> branch) using git to do things like graft away the bad revisions, and
>> then the second stage will use the corrected data.  Of course
>> eventually, this detail will be hidden by the remote helper.
>
> Excellent suggestion! I'll attempt to build the plumbing for the
> mapping in a manner that exposes a sane interface.
>
>> As a general comment - you must be careful in trying to assume that
>> what you are attempting is even possible.  Sure, you want 'git clone
>> svn://example.com/myrepo' to work, but what does that mean?  A
>> repository in SVN is a filesystem, which can contain multiple
>> projects.  In git, a repo is a single project.  People might expect  
>> to
>> be able to clone the trunk URL for instance.  My advice there is to
>> not support that use case at all, it's a complete can of worms which
>> you will discover as you tackle the conversion algorithms.  Just  
>> focus
>> on making the case where the complete repository is mirrored work for
>> this project.  Mining a single branch out of SVN without all data
>> available is the domain of git-svn and really you don't want to go
>> there.
>
> Hm, this is something that I hadn't thought about earlier. Thanks for
> the suggestion- I will not attempt to go into complicated cases,
> atleast in my summer term.
>
>> Anyway like I say, please follow-up on the mailing list, and this
>> advice can receive wider scrutiny.
>
> Thank you for your valuable comment!
>
> -- Ram
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html