Reconstructing git-svn metadata after a git clone

I maintain a couple of git-svn exports [1][2] on GitHub. I'd like to instruct my users on how they should clone those projects and then commit back to Subversion. But in some cases I can't figure out how to reconstruct the Subversion metadata.

    1. http://github.com/openstreetmap
    2. http://github.com/mediawiki

The git-svn documentation is lacking here. I'd like to send a patch for it, but first I have to know what to write. I can add Subversion metadata to some repositories, but not others.

I can e.g. clone the WebKit repository (http://trac.webkit.org/wiki/UsingGitWithWebKit#Checkout):

    git clone git://git.webkit.org/WebKit.git WebKit
    cd WebKit
    git svn init -T trunk http://svn.webkit.org/repository/webkit
    git update-ref refs/remotes/trunk origin/master

That works just fine and gives me:

    [...]
    r59025 = 542777d30c645a095d5416a74fdf60b2f3446685
    r59026 = 9afc07910ceabcc065fcd4fcd1c68382955d0d7e
    r59027 = c7c8bfaf28e19eab90f68f53dc7523dd07ef794f
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.268f45cc-cd09-0410-ab3c-d52691b4dbfc

However, when I do that with some other repositories I can't make it do the right thing:

    # Made with `git svn clone
http://svn.openstreetmap.org/applications/utils/tirex/`
    git clone git://github.com/openstreetmap/tirex.git
    git svn init http://svn.openstreetmap.org/applications/utils/tirex/

So far so good, but this doesn't work:

   git update-ref refs/remotes/trunk origin/master
   # or this
   git update-ref refs/remotes/master origin/master
   # or this
   git update-ref refs/remotes/origin/master origin/master

On git-svn fetch I just get:

    $ git svn fetch
    W: Ignoring error from SVN, path probably does not exist:
(160013): Filesystem has no item:
'/!svn/bc/100/applications/utils/tirex' path not found
    W: Do not be alarmed at the above message git-svn is just
searching aggressively for old history.
    This may take a while on large repositories
    Checked through r14400

Followed by it downloading all the revisions again. I get similar results with the MediaWiki repository:

    git clone git://github.com/mediawiki/mediawiki-svn.git
    # It was cloned with --stdlayout from this path (well, actually a
mirror, but..)
    git svn init --stdlayout svn+ssh://svn.wikimedia.org/svnroot/mediawiki

The branches look like this:

    $ git branch -l -a | head -n 20
    * trunk
      [...]
      remotes/origin/FundraiserPortal-multi@54613
      remotes/origin/HEAD -> origin/trunk
      remotes/origin/LuceneSearch-ajax
      remotes/origin/MWSearch-2.0@21624
      [...]

Trying to update-ref appropriately:

    for i in $(git branch -l -a|ack '^  remotes/origin/(\S+)$'
--output '$1'); do
        git update-ref "refs/remotes/$i" "origin/$i";
    done

Running a subsequent `git svn fetch` fails:

    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
...
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
...
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
...
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
...
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
    Index mismatch: fc65f221da86f97a43691fb203aed3da67fb2336 !=
4b825dc642cb6eb9a060e54bf8d69288fbee4904
    rereading d226d034df045bfae1804da6c23f75fc821c4d3f
            D       CommonsBrowser/CommonsBrowser.sln
            D       CommonsBrowser/CommonsBrowser.suo
            D       CommonsBrowser/CommonsBrowser/CommonsBrowser.cspr
            [..tries to delete everything in the repo..]
    W: -empty_dir: trunk/zedler
    a.sql
            D       scavenger/tables_pg.sql
            D       scavenger/
    Last fetched revision of refs/remotes/trunk was r66073, but we are
about to fetch: r1!

However, this evil hack works fine for me:

    # Exactly like before
    git clone git://github.com/mediawiki/mediawiki-svn.git
    cd mediawiki-svn
    git svn init --stdlayout svn+ssh://svn.wikimedia.org/svnroot/mediawiki

    # Download a tarball of the .git/svn content from the original
    # repository:

    cd .git
    wget http://git.anyonecanedit.org/~avar/svn.tar.bz2
    tar xjvf svn.tar.bz2
    cd ..

    # Now git svn fetch works!

    $ git svn fetch
            D       extensions/Storyboard/jquery/jquery.ajaxscroll.min.js
            M       extensions/Storyboard/jquery/jquery.ajaxscroll.js
            M       extensions/Storyboard/README
            M       extensions/Storyboard/tags/Storyboard/storyboard.js
            M       extensions/Storyboard/storyboard.js
            M       extensions/Storyboard/INSTALL
            M       extensions/Storyboard/RELEASE-NOTES
    W: -empty_dir: trunk/extensions/Storyboard/jquery/jquery.ajaxscroll.min.js
    r66071 = f33f81701b06abde87ed11d140cfe3ce4e073bf5 (refs/remotes/trunk)
            M       extensions/Storyboard/Storyboard.i18n.php
            M
extensions/Storyboard/tags/Storysubmission/Storysubmission_body.php
            M       extensions/Storyboard/specials/Story/Story_body.php
            M       extensions/Storyboard/storyboard.js
            M       extensions/Storyboard/Storyboard.php
    r66072 = 954997aef3bbadb4c2a3215e7dd031572a6c3d09 (refs/remotes/trunk)
            M       extensions/SemanticMaps/GeoCoords/SM_GeoCoordsValue.php
            M
extensions/SemanticMaps/GeoCoords/SM_GeoCoordsValueDescription.php
    r66073 = 1902743c893d9107a3fe4e1ce4ed109c4c5ca0de (refs/remotes/trunk)
    $ git svn fetch
    $

So:

 * Am I doing something wrong? If so I can't see what it is.

 * Is there something that works for the general case, i.e. you only
   have to know the original `git svn init` options. If there is I'd
   like to document that & submit a patch.

 * Depending on the above; can git-svn itself be friendlier here?
   Maybe by having a `git svn bootstrap` command. E.g.:

  git clone git://$some_url
  # Does all the work of setting up metadata/refs
  git svn bootstrap --stdlayout $remote_svn_url

On 8 May 2010 18:58, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

No

In my repo I have a branch with no anestors which has a config file, setup.sh & fetch.sh

I instruct to clone repo, checkout "utils" branch, run setup.sh (it overrides .git/config with config file committed to utils branch and after it copied config it runs git svn init URL), fetch.sh just runs git svn fetch =)

I have to do this because I have two svn remotes and both are not standard layout. The only way for me to represet "same init options" is by editing .git/config cause it's impossible for me to supply git-init options on the command line =)

It would be nice to have the git init info propogate with the git clone. But this won't work. You are cloning *all* branches and providing a git mirror, where as I want to to git init just my svn branch or a subset of them. To achieve that I will clone just the branches I need modify my .git/config and get the result I want.

So imho git-svn is ok here.

If you are committing to svn regularly you are better of with bzr-svn in my opinion. Because launchpad can run automatic imports for you (webkit is already running btw) and the whole bootstrapping thing is done the way you are expecting it.

#create repository to store revisions efficiently $ bzr init-repo .

#on the first ever run it will rebuild meta-data #subsequent runs just fetch missing revisions $ bzr branch svn://path.to.any.branch

And you can commit from that =) and bzr can operate on your svn checkouts. And every single clone done by bzr-svn is identical (unlike git where everyone has to follow the same git-svn mirror to get same revision-ids).

You have two options when commiting with bzr-svn. Regular bzr ci will store bzr merge information in revision properties on svn server or you can use bzr dpush which is like git svn dcommit.

IMHO bzr-svn is the best when you need to commit back to svn and painlessly commit parts of the feature branch and merge other bits later.

On Sat, May 8, 2010 at 18:20, Dmitrijs Ledkovs <dmitrij.ledkov@ubuntu.com> wrote:

If I can't resolve this I'll do something similar. Just provide a tarball of .git/svn for each repo that is.

I still don't see why it can't work. Maybe I'm just hopelessly naïve to git-svn internals, but:

   * Every commit message in the Git repo has info on the svn
     branch/svn commit.

   * The Git repo has a list of branches that map to SVN branches.

   * If I supply the same `git svn init` options that made the repo,
     it should be able to bootstrap just using the above.

Why isn't that the case? What info is in .git/svn that can't be inferred from the above?

That's informative. But from having used Bazaar a bit my experience with it was that I might as well be using Subversion.

On 10 May 2010 22:43, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Always an option.

Your local one does. But not the cloned one.

Yes, but it takes a while..... especially if the svn repo is huge.

Is there a git versions mismatch between machine where the first import was made and the machine where you are testing bootstrapping by any chance?

on my machine I sometimes see "upgrading metadata" when I'm updating some of my acient git-svn clones.

On Tue, May 11, 2010 at 04:27, Dmitrijs Ledkovs <dmitrij.ledkov@ubuntu.com> wrote:

No, they were both running 1.7.1 from Debian.