SVN -> Git *but* with special changes

I have a couple big projects in SVN that I'd like to convert to Git. Being in SVN, they've operated under a couple assumptions that are not true under Git. These assumptions are:

1. Directories can exist even if there's nothing in them (just like any
filesystem).

2. Keeping binary files in version control isn't a big deal because the
whole repo doesn't get copied to everyone's system or use up people's GitHub
storage space.

The SVN repos have been relying on those, but both are false under Git, so I need to do a "modified" conversion, rather than just a straight one.

So, how can I convert an SVN repo to Git, and have the conversion add dummy files to empty directories and exclude specific files? (Also, there are tags and branches to be converted too, in the SVN-standard "tags" and "branches" directories.)

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p6840904.html Sent from the git mailing list archive at Nabble.com.

----- Original Message -----

What requires the empty directories to exist?  The build system?  Can you just let them go away in Git and fix it downstream in the user's working copy?
 

I'd eliminate these on the SVN side before converting to Git.  If you have svnadmin access to the SVN repo svnadmin dump, svndumpfilter, svnadmin load is a pretty easy process.

To do the actual conversion, svn-fe and git fast-import are by far the quickest way to get the data into Git.  Filtering into tags and branches is a bit of a trick though [1].  git-svn has (IMO) a good branching UI, but can be very slow for large repositories.

[1] http://thread.gmane.org/gmane.comp.version-control.git/158940/focus=159151 : note this thread is almost a year out of date now, and even I know much better ways to go about this now -- but the scripts are not written.

HTH, Stephen

Stephen Bash <bash@genarts.com> writes:

Otherwise, you can do it on the Git side with:

  git filter-branch --tree-filter 'rm -f some-large-blob'

On Wed, Sep 28, 2011 at 07:44:24PM +0200, Matthieu Moy wrote:

It's much more efficient to do:

    git filter-branch --index-filter \
      'git rm --cached --ignore-unmatch some-large-blob'

which avoids checking out the large files over and over[1], just to delete them. Of course, it's even more efficient not to import them from svn in the first place. :)

-Peff

[1] This is straight from the filter-branch manpage, btw.

Yes.

working copy?

That won't work retroactively for already-existing revisions. I'd rather not break the build system for older revisions just for such a goofy reason as empty dirs being deemed unnecessary.

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p6841234.html Sent from the git mailing list archive at Nabble.com.

I got busy with other things and haven't had a chance to get back to this until now. I very much appreciate the help but I'm still unclear on this part:

repositories.

http://thread.gmane.org/gmane.comp.version-control.git/158940/focus=159151 : note this thread is almost a year out of date now, and even I know much better ways to go about this now -- but the scripts are not written.

I don't really understand that. So...how do I convert from svn to git while keeping branches and tags intact? If it's automated, I don't care about speed: I can just start it and move on to other things while it's working.

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7162793.html Sent from the git mailing list archive at Nabble.com.

On Wed, Sep 28, 2011 at 09:37:29AM -0700, Abscissa wrote:

The usual technique is to put an empty .gitignore file into the empty directory, which will make sure it always exists. If you import with git-svn, it supports "--preserve-empty-dirs", which will do this for you automatically.

git-svn --ignore-paths will handle this for you.

Side note on the GitHub thing: all of the forks will share objects, so it's not much as space as you might think. And all of the paid plans count repos, not bytes. Of course, the painful part is probably cloning the big objects to everybody's workstation. :)

git svn --stdlayout ?

-Peff

I came across this, and it seems to mostly work: http://john.albin.net/git/git-svn-migrate

Except it fails to do the "--preserve-empty-dirs". It does say that it passes any options it doesn't recognize directly to git-svn, so I gave it that, but just got:

- Cloning repository...
Unknown option: preserve-empty-dirs

Now, I realize this isn't a forum for that particular "git-svn-migrate" tool, and I don't expect anyone to go digging though it on my account, so I'm willing to try to add it into the script if I can just find out the proper way to use "--preserve-empty-dirs".

However, I suspect that script may not be my problem at all: If I do this:

git svn help | grep preserve

I get absolutely nothing. There doesn't seem to be a "--preserve-empty-dirs".

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7163706.html Sent from the git mailing list archive at Nabble.com.

On Sat, Jan 07, 2012 at 09:03:51PM -0800, Abscissa wrote:

What version of git are you using? That option was added in git 1.7.7.

-Peff

Ok, I see. It's reporting 1.7.0, so that would explain it. One "sudo apt-get upgrade git" and...erm...well, it seems to be upgrading my whole damn computer, but I'll see how it all works out. Thanks!

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7163737.html Sent from the git mailing list archive at Nabble.com.

Well that's strange, it finished "upgrading", but now git is still just reporting 1.7.0.4, which is *exactly* the same version it said before. The git-svn package should already be up-to-date because I just installed it today. So I don't know what's up with that.

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7163752.html Sent from the git mailing list archive at Nabble.com.

On Sat, Jan 07, 2012 at 09:25:27PM -0800, Abscissa wrote:

Nothing odd about that. apt-get upgrade means "upgrade my system". If you want to get a newer version of package X, you do apt-get install X and it will install the latest version of that package.

Your OS seems ancient, you might be better off installing from source.

   cmn

Carlos Martín Nieto <cmn@elego.de> writes:

If apt-get upgrade doesn't get you a newer version then apt-get install won't help you either. Both use the same installation source.

Andreas.

On Sun, Jan 08, 2012 at 11:47:52AM +0100, Andreas Schwab wrote:

No, this is true only if none of packages involved uses a new library (including new sonames). "apt-get upgrade" is forbidden to add or remove packages, and thus will skip upgrades that need a new dependency. The message at the end will mention "## not upgraded" though.

"apt-get install X" or "apt-get dist-upgrade" have no such restrictions.

I see. That's strange, I've been using "apt-get upgrade xxx" for at least a couple years, I can't believe I never knew that! Thanks for the detailed info, everyone (and for the patience!)

In answer to someone's question, yea, it's Ubuntu (Kubuntu 10.04, and yea, I know that's old, but it's not my primary system and I haven't had a chance yet to upgrade it and get everything set back up again). I also tried it on a Debian 6 Live/Persistent system and got the same results...apparently for the same reason.

I did manage to get 1.7.8 installed (on both systems) by downloading and installing the source (it was kind of a pain figuring out the names of some of the dependent packages, but I managed to get it.)

So hopefully this should all work now...

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7165979.html Sent from the git mailing list archive at Nabble.com.

Damn. That 'git-svn-migrate' tool is just giving me errors now. It just gets through little over 10% of the revisions, and gives:

----------------------------- Failed to strip path 'bin/lang/.gitignore' ((?-xism:^trunk(/|$)))

- Converting svn:ignore properties into a .gitignore file...
fatal: ambiguous argument 'HEAD': unknown revision or path not in the
working tree.
Use '--' to separate paths from revisions
log --no-color --no-decorate --first-parent --pretty=medium HEAD: command
returned error: 128
-----------------------------

And then it keeps going and gives:

-----------------------------
To /var/git/Goldie.git
 ! [rejected]        master -> trunk (non-fast-forward)
error: failed to push some refs to '/var/git/Goldie.git'
To prevent you from losing history, non-fast-forward updates were rejected
-----------------------------

I guess I'll have to try some other approach. :/

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7166084.html Sent from the git mailing list archive at Nabble.com.

I'm trying to do it manually with git-svn, but I'm getting the same problem. I have two different SVN repos I want to do (turns out neither have branches, I had thought they did, but one of them does have tags). Doing this:

git svn clone {the url to the SVN repo} --prefix=svn/ --preserve-empty-dirs --authors-file={already prepared authors file} --trunk=trunk --tags=tags

It gets partway through, and then gives me:

Failed to strip path '{some path}/.gitignore' ((?-xism:^trunk(/|$)))

Where {some path} is an empty dir in the trunk of one repo, and a completely non-existent path in the other. (In both cases I'm looking at the revision git-svn had gotten to when bailing, plus the revision before and the revision after).

I don't have a clue what that error message is trying to tell me or what is going wrong. :(

If it helps, these are the repos I'm trying to convert:

http://svn.dsource.org/projects/semitwist   (git svn clone fails at r46 (out
of 242) on 'src/nonFatalAssertTest/.gitignore')
http://svn.dsource.org/projects/goldie   (git svn clone fails at r85 (out of
557) on 'bin/lang/.gitignore')

(Yes, they do have a non-standard top-level "downloads" directory, but that's just how that host does file downloads, and I *don't* need to preserve it)

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7181897.html Sent from the git mailing list archive at Nabble.com.

Someone on a different message board noticed that it seemed like it was trying to strip and already-stripped path in git-svn.perl, so I went in there, changed the "die" to "print", and after a "make && sudo make install" it seems to work fine now. Probably not the proper way to fix it, but it seems to work for me.

It does now fail to delete directories once they actually *are* deleted in the SVN repo, which is kinda sloppy, but it doesn't hurt anything, so I can live with that.

-- View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7186699.html Sent from the git mailing list archive at Nabble.com.

On 01/08/2012 11:28 PM, Abscissa wrote:

For Ubuntu users: there is a git PPA [1] which usually has quite up-to-date git packages for recent Ubuntu releases. If you configure your system to use this PPA (instructions are on the page) then you can stay up-to-date with minimal effort. Currently they have version 1.7.8.2 available for hardy, lucid, maverick, natty, and oneiric and older versions for some other Ubuntu releases.

Caveat: I am not affiliated with the PPA and cannot vouch for its integrity.

Michael

[1] https://launchpad.net/~git-core/+archive/ppa

Abscissa wrote:

What kind of distribution do you use?

"apt-get" sounds like Debian or Ubuntu, but those all have at least git 1.7.1:

| Debian versions:
| stable      1:1.7.2.5-3 
| stable-bpo  1:1.7.7.3-1~bpo60+2 
| testing     1:1.7.7.3-1 
| unstable    1:1.7.8.2-1 
| exp         1:1.7.8~rc3-1 

| Ubuntu versions:
| Precise Pangolin  1:1.7.7.3-1
| Oneiric Ocelot    1:1.7.5.4-1
| Natty Narwhal     1:1.7.4.1-3
| Maverick Meerkat  1:1.7.1-1.1ubuntu0.1

-thh

Abscissa schrieb:

"apt-get upgrade" will upgrade _all_ packages and doesn't take a
parameter:
| upgrade
|    upgrade is used to install the newest versions of all packages
|    currently installed on the system from the sources enumerated in
|    /etc/apt/sources.list. 

-thh