Re: [spf:guess,iffy] Re: [spf:guess] Re: Approaches to SVN to Git conversion (was: Re: [RFC] "Remote helper for Subversion" project)

From: Sam Vilain <sam@vilain.net>
Date: Mar 7, 2012, 23:15 UTC
Message-ID: <4F57EC04.8060705@vilain.net>
In-Reply-To: <4F57DBF0.4060101@pileofstuff.org>

On 3/7/12 2:06 PM, Andrew Sayers wrote:

Show 5 quoted lines

> It sounds like we've approached two similar problems in similar ways, so
> I'm curious about the differences where they exist.  I've been reading
> this message of yours from 18 months ago alongside this thread:
> http://article.gmane.org/gmane.comp.version-control.git/150007
> Unfortunately these comprise everything I know about Perforce.

Right, I went into more detail back then than I did with my more recent message.

Show 6 quoted lines

> I notice that git-p4raw stores all of its data in Postgres and provides
> a programmatic interface for querying it, whereas I've focussed on
> providing ASCII interfaces at relevant points.  I can see how a DB store
> would help manage the amount of data you'd need to process in a big
> repository, but were there any other issues that drove you down this
> route?  Did you consider a text-based interface?

I wrote it like this mostly because the source metadata was already in a tabular form. It allowed me to load the data, and then convert deductions I could make of the data into unique and foreign key constraints. It provided me with ACID semantics to make it so that if my program ran and failed the changes would not be applied. Despite the popular opinion of "web–scale" technologists, databases do have large advantages over unstructured hierarchical data :-).

I didn't really intend to provide a programmatic interface, that was a set of user tools. The SQL store is the programmatic interface :)

Show 11 quoted lines

>> What I did for the Perl Perforce conversion is make this a multi–step
>> process; first, the heuristic goes through and detects branches and
>> merge parents.  Then you do the actual export.  If, however, the
>> heuristic gets it wrong, then you can manually override the branch
>> detection for a particular revision, which invalidates all of the
>> _automatic_ decisions made for later revisions the next time you run it.
>
> Could you give an example of overriding branch/merge detection?  It
> sounds like you're saying that if there's some problem detecting merge
> parents in an early revision, then all future merges are ignored by the
> script.

The wrong decision can make things much worse down the line. With the Perl history, the repository was about 350MB of pack, until I got the merge history correct. Afterwards, it packed down to about 70MB. This is because there was a lot of criss–cross merging, and by marking them correctly git's repack algorithm was more able to locate similar blobs and compress correctly. The pack size was not the goal, but a good verification that I had brought the correct commits together in history.

The bigger problems with it range from thinking changes are merged in your branch which weren't really, or depending on how branch detection etc works, getting thrown off completely and emitting garbage branch histories. So, it does help to be able to "rewind" the heuristics, poke information in and then resume again and see if things are improved. The information could be inserted into a single file which has configured the entire import, and also serves as a set of notes as to the amendments carried out. I was happy with a database dump :-).

Show 9 quoted lines

> <snip>
>> The manual input is extremely useful for bespoke conversions; there will
>> always be warts in the history and no heuristic is perfect (even if you
>> can supply your own set of expressions, a way to override it for just
>> one revision is handy).
>
> Again, would you mind providing a few examples?  It sounds like you have
> some edge cases that could be handled by extending the branch history
> format, but I'd like to pin it down a bit more before discussing solutions.

There's a few,

* a branch contains a subproject and is merged into a subtree
* someone puts a "README" or similar file in a funny place, which isn't 
inside a project root
* someone starts a project with no files in its root directory
* someone records a merge incorrectly (or using a young or middle–aged 
SVN which didn't record merges).  You don't want your annotate to hit a 
merge commit which isn't recorded as a merge, and then have to go 
hunting around in history for the real origin of a line of code
* the piecemeal merge case you have seen yourself.

It's just very useful to be able to reparent during the data mining stage.

Show 48 quoted lines

> <snip>
>>    3. skip bad sections of history, for instance squash merging merges
>> which happened over several commits (SVN and Perforce, of course,
>> support insane piecemeal merging prohibited by git)
>
> This is an excellent point I've stumbled past in my experiments without
> realising what I was seeing.  A simple SVN example might look like this:
>
> 	svn add trunk branches
> 	svn add trunk/foo trunk/bar
> 	svn ci -m "Initial revision" # r1
>
> 	svn cp trunk branches/my_branch
> 	svn ci -m "Created my_branch" # r2
>
> 	# edit files in my_branch
>
> 	svn merge branches/my_branch/foo trunk/foo
> 	svn ci -m "Merge my_branch ->  trunk (1/3)" # r11
>
> 	svn merge branches/my_branch/bar trunk/bar
> 	svn ci -m "Merge my_branch ->  trunk (2/3)" # r12
>
> 	svn cp branches/my_branch/new_file trunk/new_file
> 	svn ci -m "Merge my_branch ->  trunk (3/3)" # r13
>
> This strikes me as a sensibly cautious workflow in SVN, where merge
> conflicts are common and changes are hard to revert.  The best
> representation for this in the current branch history format would be
> something like this:
>
> 	In r1, create branch "trunk"
> 	In r2, create branch "branches/my_branch" from "trunk"
> 	In r13, merge "branches/my_branch" r13 into "trunk"
>
> In other words, pretend r11 and r12 are just normal commits, and that
> r13 is a full merge.  A more useful (and arguably more accurate)
> representation would be possible if we extended the format a bit:
>
> 	In r1, create branch "trunk"
> 	In r2, create branch "branches/my_branch" from "trunk"
> 	In r12, squash changes in "branches/my_branch"
> 	In r13, squash changes in "branches/my_branch"
> 	In r13, merge "branches/my_branch" r13 into "trunk"
>
> Adding "squash" and "fixup" commands would let us represent the whole
> messy business as a single commit, which is closer to what the user was
> trying to say even if it's further from what they actually had to say.

Right, you see the problem.

I think your text syntax is fine so long as it is precise enough, and similar to what I mention earlier in this e–mail with having a single file to drive a conversion run. That really is the kind of input data that I had, it's just that I set it up as a useful set of commands.

Sam.