GSOC remote-svn: branch detection

Hi!

I'm playing around in vcs-svn/ to start a framework for detecting and processing branches in svndumps. So I wanted to let you know about my ideas.

Two approaches:
1. Import linearly and split later:
One idea is to import from svn linearly, i.e. one revision on top of it's 
predecessor, like now, and detect and split branches afterwards. The svn 
metadata is stored in git notes, so the required information would be 
available.
+ allows recovery, because the linear history is always here.
+ it's easier to peek around in the git history than in the svn dump during 
import to do the branch detection.
- requires creation of new commits in the branch detection stage.
- this results in double commits and awkward history, linear vs. branched.

2. Split during import:
Detect branches as they are created while reading the svn dump and identify to 
which branch a following node belongs.
First step is to restructure svndump.c to be able to buffer one complete 
revision for inspection before starting to write a commit to fast import.
Probably it's possible to feed the blobs to fast import directly and only 
buffer node data and defer commit creation, but not the data.
Currently, at the beginning of a new revision on the svn side, a new commit is 
created on top of a constant ref. When we support branches, we don't know the 
ref, i.e. the branch(es), the revision changes, before reading all the 'Node-
*' lines.
+ feels more 'right'
- requires revision buffering

Generally: Detect branches as they are created by 'Node-copyfrom*' to some commonly used branch directories, like branches/. More complex branch detection can be implemented later, of course. Store detected branches permanently (necessary for incremental fetches), and assign every file modification to one of those branches, if possible. Else assign them to, hm .. If a revision modifies more than one branch, create multiple commits.

Thanks for your comments and ideas! 

-- Florian

Hi,

Florian Achleitner wrote:

I don't think you've captured the real pros and cons here.

+ Divides responsibility between a component that fetches and a component
that splits branches, making for easier debugging, independent refactoring
of components, reuse in other contexts (e.g., splitting out branches in
other similar VCSen, etc)

- Divides responsibility between a component that fetches and a component
that splits branches, which is tricky because it involves designing an
interface between them and documenting it.  And maybe a different
interface would be better.

There are also performance and history-clarity ramifications as you've mentioned, but they do not seem as important.

Hope that helps, Jonathan

Hi,

On Sat, Aug 4, 2012 at 12:17 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:

This is a good one, but I'd put questions another way:
- do we want to query svn server only for newer revisions even if our
settings changed (branch layout ones for example), maybe we don't mind
some queries in settings change case (like git-svn.perl)?
- do we want to be able to filter svn history early (like take
trunk,branches,tags, skip tests_data as it's huge but sometimes there
are svn cp to/from it, or maybe the repo has weird permissions or even
is corrupted)?
- do we just want a completely separate (fast) (local) storage like
svn dump file to use it for imports and settings changes?

I personally still haven't decided on those. My set of pros/cons:
+ should be the simplest thing for simple small repos
+ keeps all the original data details and looks quite robust
- becomes complicated if we don't want or can't import some parts of
the history. While git-svn.perl somehow handles is.
- looks like a thing to store and access svn dump information, do we
really want it to be in a form of git objects (almost sure), how
stable, flexible, independent from svn helper should it be (that's
what Jonathan talks about).

Weird idea: what if we keep everything in one huge git tree like rXX/{data,props,copy-from,..}/path/path/path/file. It should represent all the known svn info so far. Ok, I know it's a late stage now and this thing is completely raw, just posting to have it written out somewhere :)

Hi,

Florian Achleitner wrote:

I think this approach will be a lot less messy if you can cleanly separate the fetching component from the mapper. Currently, svndump re-creates the layout of the SVN repository. And the series you posted last week contains a patch that attaches a note with SVN metadata to each commit. Do you have thoughts on how the mapping will take place?

Ram

On Saturday 04 August 2012 23:53:58 Ramkumar Ramachandra wrote:

The mapping itself is currently a black box for me, it's internals could be rather complex. It could get a function like is_branch_start, that is called with a node ctx and tells if this is likely to be the start of branch. The detected branches are stored and upcoming changes in the associated directories are mapped to a commit on a branch. The detection of branch starts and the list of existing branches can be taken from whatever logic we want. So that's approx. the idea.

Currently I'm working on more basic preparations. I want to split the creation of commits and the creation of blobs in svndump.c. This is necessary because fast import requires a branch name as an argument to the 'commit' command, and currently a 'commit' command is started when a new revision is encountered in the svndump. But to decide on which branch the commit should go, or even if it will be more than one commit, it is necessary to read all the nodes first. To prevent buffering the node content, I want to replace the inline data format (currently used) by 'blob' commands. While parsing the dump, every node change creates a blob command to feed the data immediately into fast-import while the node metadata (struct node_ctx) is stored at least until the revision ends. Then the blobs can be put on a linear master tree and other branch trees. The node metadata could also be read from notes, if remapping branches. That's not so easy to do, because the current implementation mixes tree- operations and blob-operations heavily, and relies on only one global node_ctx.

Flo