Tracking and committing back to Subversion?

Hi all,

Has anyone done any work on bidirectional access to SVN repositories? ie, tracking and committing.

That would be porcelain that behaves like SVK (http://svk.elixus.org)

Ideally it would probably need to link against the Subversion RA (remote access) library, neon.

I can see forsee two potential issues;

  1. file properties - such as mime type, ignores and custom properties.
     Linus, when I asked you about this in Dunedin, you mentioned that
     there is a place at the end of the directory entry where this could
     fit without breaking backwards compatibility.  Perhaps this could
     be an optional pointer to another directory node;

     /
      trunk/       # after all, we're tracking an SVN repo ;)
        MyModule/
          Makefile.PL
          MyModule.pm
          <prop-dir>/
            svn:ignore

     <prop-dir> is a directory object, with one file in this example;
     called svn:ignore.  The contents of this file would be something
     like the .gitignore;

          Makefile
          Makefile.old
          blib
          ...etc...

     This configures the directory "/trunk/MyModule" to have a property
     "svn:ignore", which makes SVN ignore those files.

  2. "forensic" file movement history - as opposed to the uncached,
     (and unversioned), automatic "analytical" file movement history.

     It would be easy for a tool to provide 100% interface compatibility
     with SVN client/SVK using properties, but properties that hang off
     the head rather than the file itself (so that they don't stuff up
     the ability to merge identical trees reached via independent
     paths).  SVN calls these "revision properties".  If a good
     convention is adopted for this, it could be used as a nice way to
     supplement git's automatic analysis of the revision history.

     So, then, git could support these three methods of finding
     revision history;

        1. automatic/analytical, as at present.  Cachable information,
           and such a cache would not form part of the revision history.

        2. forensic, if using porcelain that supports it; this would
           form part of the revision history, but be out of band to the
           content.

        3. (potentially) retrospective, for instance, somebody studying
           an old code base and tracking history for some reason - or
           even 'correcting' forensic revision history that is causing
           their advanced patch calculus algorithm to trip up (should
           someone be using git as a storage back-end for darcs, for
           instance)

     Ideally, it would be possible to distinguish between 2 and 3, of
     course this would require two head identifiers; the content head,
     and the revision history head.

This might seem like functionality not needed for merging kernel trees ;) but I think it would add value to be able to make git truly join the trump suit in the VCS deck (along with SVK, which can, for instance, track Perforce and even git repositories).

Sam.

Sam Vilain <sam@vilain.net> wrote:

AFAIK, Not yet. But it's something that's been on the back of my mind for a while. I've attempted similar things with Arch <-> SVN in the past but didn't get anything extremely robust going from Arch -> SVN although I'm pretty satisfied with my SVN -> Arch product (svn-arch-mirror).

I prefer to use git-svnimport for pulling from svn since it's pretty good at what it does. That already depends on SVN::Core and SVN::Ra. SVK is really nice tool, just a bit slow after I've been using git for a while.

Mostly ignore-able, imho. svn:executable is the one that makes the most sense (and is easiest) to support.

svn:ignore <-> .gitignore mapping are pretty close (identical, save for recursive properties that svn gets very wrong. I don't think svn:ignore is affected by line order, whereas .gitignore is a plain file and line order does matter.

svn:external is almost definitely better off ignored entirely when interfacing with other RCSes

svn:keywords: I don't think there's a way to disable this like there is with CVS, is there? keywords are evil imho.

I don't use or know much about the other properties...

Just parsing the output of diff-tree -C and marking them in SVN as copies/renames should be sufficient for letting SVN do its thing.

Doing this kind of file movement history on the git side sounds like overkill to me. I was a _huge_ fan of logical file-identities in GNU Arch in the past, but the complexity destroyed it from both a UI and performance perspective.

On  3 Feb 2006, normalperson@yhbt.net wrote:

I think tailor claims to be able to handle this: http://www.darcs.net/DarcsWiki/Tailor

The README in the devel version seems to indicate that two-way sync for svn <--> git is implemented (despite the table on the main page).

--
 + seth

Eric Wong wrote:

  [...]

Right, but it is also possible to hang directory or file properties, with any name, off any file or directory. It is often important to at least preserve or be able to update these when dealing with some SVN repositories, where people are using them for their own purpose.

Perhaps a "standard" mapping to dotfiles would be the best solution then. A .svnprops file or something like that.

Right now, tagging gets an exemption from this requirement, as does the extra state required by tools like StGIT. I think this could be generalised to support this kind of application; however we will see whether this has any real effect or just operational side effect (eg, to make sure renames are tracked correctly then commit before changing the copied version, etc).

Sam.

Well..

On Wednesday 01 February 2006 14:51, Sam Vilain wrote:

I'm working on something that does just that, I haven't gotten to the bidirectional part just yet. (The only reason I didn't use git-svnimport is that it has a /terrible/ time importing from the KDE svn repo. )

My take on the properties, though I like your thoughts on how to handle at least svn:ignore, is to basically just let svn deal with them. I will make a couple tools that will modify them, for interface reasons, but they'll just be wrappers around the svn commands themselves.

At the moment, the tools are written as bash shell scripts with an interface I'm trying to keep as intuitive as cogito (even though I use the core-git tools now.)

Brian Smith wrote:

Indeed, mirroring SVN repositories via SVN::Mirror (which is used by SVK) can take hours for large projects with thousands of commits.

 > At the moment, the tools are written as bash shell scripts with an
 > interface I'm trying to keep as intuitive as cogito (even though I use
 > the core-git tools now.)

Right; I was looking for an approach that did not require working copies of the remote subversion repository to be kept locally. Still, perhaps that approach has merit, though I would probably start in Perl and use SVK::Simple (see CPAN) to give a richer SVN mirroring API.

Are you planning on publishing these scripts?

Sam.

On Monday 06 February 2006 15:23, Sam Vilain wrote:

Actually, I wasn't so much talking about the time required as the fact that git-svnimport expects a very specific directory format, and KDE, since it was migrated from CVS doesn't follow it, not to mention, I'm writing my tools for the purpose of tracking single projects (i.e. Kopete which is part of KDE project).

While that is an admirable goal, unless you can point me to something that will allow you to actually commit back to SVN without a working copy, it defeats the purpose of my tools which is basically to use to git for the purpose of holding intermediate development before sending it into SVN as a final commit. That, and being able to use git tools which speak to me on a level far greater than SVN tools. ;)

Sure, you can actually pick up the current development straight from git://linuxfood.net/pub/git/kosek.git

Brian

Brian Smith <linuxfood@linuxfood.net> writes:

Wouldn't svk (or svl or whatever it is called these days) be a better match for that kind of "keep my work while disconnected from master svn repository" purpose?

On Thursday 09 February 2006 16:54, Junio C Hamano wrote:

It would, if it wasn't god awful slow. And besides, AFAIK, it doesn't allow you to do anything other than keep offline work in a separate SVN repo hosted on your machine, which means I can't use the git tools.

Brian Smith wrote:

Such as SVK

   http://svk.elixus.org/
   (svn url: http://svn.openfoundry.org/svk/)

Your solution may ultimately be the most pragmatic approach. Issues surrounding the C (SWIG) bindings between Perl and the Subversion RA (remote access) API were "the hard part" in making SVK work, according to the lead author.

That being said, I am unaware of any drastic outstanding issues, so perhaps there are useful components there.

I will take a look.

Sam.

On Thursday 09 February 2006 20:27, Sam Vilain wrote:

Well, after reading through a number of pages I discovered something:

"The repository created is actually a Subversion repository. In other words, you could use any subversion commands with that repository." [*1*]

So, while it is a technicality, there is an SVN Repo, which is what I'm trying to avoid (the point being to have a git repo after all, for the purpose of using git commands), perhaps it would be worth looking into to see if I could somehow utilize the svk svn repo (since there are space savings) for the SVN working copy. [*2*]

There are useful things in there. I have considered rewriting in perl so that I don't have to use my kludge of a rewrite of Getopt::Long (yes, I did. No, not everything is supported.) but have decided against it mostly because it would take weeks, and Kosek (my program) isn't even complete yet.

(Incidentally, the allure of the SVN perl bindings had me for awhile, but after trying to decipher the horrid PODs for it decided that Bash was just easier. )

Awesome let me know what you think. Please try to ignore the massive lack of documentation, up until someone in the kopete irc channel showed interest I had been writing it for my own purposes. Though, since I posted the link here I did go ahead and fixup the readme with a little howto. If you like it, feel free to email me with questions and suggestions.

[*1*] http://svk.elixus.org/?SingleUserSVK (under Mapping Repositories)

[*2*] While writing this, I had an absolutely insane idea, which I may or may not persue. The idea being that it may be possible to remove 99% of the actual SVN working copy (except for perhaps the base directory at the root of the project), and just use symlinks for the rest of the hierarchy. This would require careful management though, since every .svn directory contains specific and distinct information.

On Thu, 2006-02-09 at 23:01 -0800, Brian Smith wrote:

Yes, currently there is - but let me show you what's under the hood briefly... as you notice later;

Yes, there are SWIG bindings for various languages. In debian/ubuntu the Perl bindings will be built to the package libsvn-core-perl.

Take a look at this module:

  http://search.cpan.org/dist/SVN-Simple/lib/SVN/Simple/Edit.pm

That uses the above module and allows you to construct a Subversion commit and try to apply it.

The documentation's a bit weak, but AFAIK the basic functionality is there and a lot better an abstraction than the raw bindings themselves.

SVN::Mirror:

  http://search.cpan.org/dist/SVN-Mirror/lib/SVN/Mirror.pm

can go the other way.

That's right - in principle, SVK could be plumbed to have a git repository in its ~/.svk/local - then you could have one git repo, and a head for each mirrored project.

But don't concern yourself with my plans - it seems our tastes in porcelain differ ;)

Sam.