{"thread":{"id":"23748","subject":"Reconstructing git-svn metadata after a git clone","startedAt":"2010-05-08T17:58:07Z","lastAt":"2010-05-11T13:04:17Z","messageCount":5,"participants":["Ævar Arnfjörð Bjarmason","Dmitrijs Ledkovs"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"141257","messageId":"AANLkTikR5EpZhwFXmRfmE9jSNtiuFoGwTgGbpcwUBxnq@mail.gmail.com","threadId":"23748","inReplyTo":null,"subject":"Reconstructing git-svn metadata after a git clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-05-08T17:58:07Z","receivedAt":"2010-05-08T17:58:07Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I maintain a couple of git-svn exports [1][2] on GitHub. I'd like to\ninstruct my users on how they should clone those projects and then\ncommit back to Subversion. But in some cases I can't figure out how to\nreconstruct the Subversion metadata.\n\n    1. http://github.com/openstreetmap\n    2. http://github.com/mediawiki\n\nThe git-svn documentation is lacking here. I'd like to send a patch\nfor it, but first I have to know what to write. I can add Subversion\nmetadata to some repositories, but not others.\n\nI can e.g. clone the WebKit repository\n(http://trac.webkit.org/wiki/UsingGitWithWebKit#Checkout):\n\n    git clone git://git.webkit.org/WebKit.git WebKit\n    cd WebKit\n    git svn init -T trunk http://svn.webkit.org/repository/webkit\n    git update-ref refs/remotes/trunk origin/master\n\nThat works just fine and gives me:\n\n    [...]\n    r59025 = 542777d30c645a095d5416a74fdf60b2f3446685\n    r59026 = 9afc07910ceabcc065fcd4fcd1c68382955d0d7e\n    r59027 = c7c8bfaf28e19eab90f68f53dc7523dd07ef794f\n    Done rebuilding\n.git/svn/refs/remotes/trunk/.rev_map.268f45cc-cd09-0410-ab3c-d52691b4dbfc\n\nHowever, when I do that with some other repositories I can't make it\ndo the right thing:\n\n    # Made with `git svn clone\nhttp://svn.openstreetmap.org/applications/utils/tirex/`\n    git clone git://github.com/openstreetmap/tirex.git\n    git svn init http://svn.openstreetmap.org/applications/utils/tirex/\n\nSo far so good, but this doesn't work:\n\n   git update-ref refs/remotes/trunk origin/master\n   # or this\n   git update-ref refs/remotes/master origin/master\n   # or this\n   git update-ref refs/remotes/origin/master origin/master\n\nOn git-svn fetch I just get:\n\n    $ git svn fetch\n    W: Ignoring error from SVN, path probably does not exist:\n(160013): Filesystem has no item:\n'/!svn/bc/100/applications/utils/tirex' path not found\n    W: Do not be alarmed at the above message git-svn is just\nsearching aggressively for old history.\n    This may take a while on large repositories\n    Checked through r14400\n\nFollowed by it downloading all the revisions again. I get similar\nresults with the MediaWiki repository:\n\n    git clone git://github.com/mediawiki/mediawiki-svn.git\n    # It was cloned with --stdlayout from this path (well, actually a\nmirror, but..)\n    git svn init --stdlayout svn+ssh://svn.wikimedia.org/svnroot/mediawiki\n\nThe branches look like this:\n\n    $ git branch -l -a | head -n 20\n    * trunk\n      [...]\n      remotes/origin/FundraiserPortal-multi@54613\n      remotes/origin/HEAD -> origin/trunk\n      remotes/origin/LuceneSearch-ajax\n      remotes/origin/MWSearch-2.0@21624\n      [...]\n\nTrying to update-ref appropriately:\n\n    for i in $(git branch -l -a|ack '^  remotes/origin/(\\S+)$'\n--output '$1'); do\n        git update-ref \"refs/remotes/$i\" \"origin/$i\";\n    done\n\nRunning a subsequent `git svn fetch` fails:\n\n    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6\n...\n    Done rebuilding\n.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6\n    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6\n...\n    Done rebuilding\n.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6\n    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6\n...\n    Done rebuilding\n.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6\n    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6\n...\n    Done rebuilding\n.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6\n    Index mismatch: fc65f221da86f97a43691fb203aed3da67fb2336 !=\n4b825dc642cb6eb9a060e54bf8d69288fbee4904\n    rereading d226d034df045bfae1804da6c23f75fc821c4d3f\n            D       CommonsBrowser/CommonsBrowser.sln\n            D       CommonsBrowser/CommonsBrowser.suo\n            D       CommonsBrowser/CommonsBrowser/CommonsBrowser.cspr\n            [..tries to delete everything in the repo..]\n    W: -empty_dir: trunk/zedler\n    a.sql\n            D       scavenger/tables_pg.sql\n            D       scavenger/\n    Last fetched revision of refs/remotes/trunk was r66073, but we are\nabout to fetch: r1!\n\nHowever, this evil hack works fine for me:\n\n    # Exactly like before\n    git clone git://github.com/mediawiki/mediawiki-svn.git\n    cd mediawiki-svn\n    git svn init --stdlayout svn+ssh://svn.wikimedia.org/svnroot/mediawiki\n\n    # Download a tarball of the .git/svn content from the original\n    # repository:\n\n    cd .git\n    wget http://git.anyonecanedit.org/~avar/svn.tar.bz2\n    tar xjvf svn.tar.bz2\n    cd ..\n\n    # Now git svn fetch works!\n\n    $ git svn fetch\n            D       extensions/Storyboard/jquery/jquery.ajaxscroll.min.js\n            M       extensions/Storyboard/jquery/jquery.ajaxscroll.js\n            M       extensions/Storyboard/README\n            M       extensions/Storyboard/tags/Storyboard/storyboard.js\n            M       extensions/Storyboard/storyboard.js\n            M       extensions/Storyboard/INSTALL\n            M       extensions/Storyboard/RELEASE-NOTES\n    W: -empty_dir: trunk/extensions/Storyboard/jquery/jquery.ajaxscroll.min.js\n    r66071 = f33f81701b06abde87ed11d140cfe3ce4e073bf5 (refs/remotes/trunk)\n            M       extensions/Storyboard/Storyboard.i18n.php\n            M\nextensions/Storyboard/tags/Storysubmission/Storysubmission_body.php\n            M       extensions/Storyboard/specials/Story/Story_body.php\n            M       extensions/Storyboard/storyboard.js\n            M       extensions/Storyboard/Storyboard.php\n    r66072 = 954997aef3bbadb4c2a3215e7dd031572a6c3d09 (refs/remotes/trunk)\n            M       extensions/SemanticMaps/GeoCoords/SM_GeoCoordsValue.php\n            M\nextensions/SemanticMaps/GeoCoords/SM_GeoCoordsValueDescription.php\n    r66073 = 1902743c893d9107a3fe4e1ce4ed109c4c5ca0de (refs/remotes/trunk)\n    $ git svn fetch\n    $\n\nSo:\n\n * Am I doing something wrong? If so I can't see what it is.\n\n * Is there something that works for the general case, i.e. you only\n   have to know the original `git svn init` options. If there is I'd\n   like to document that & submit a patch.\n\n * Depending on the above; can git-svn itself be friendlier here?\n   Maybe by having a `git svn bootstrap` command. E.g.:\n\n  git clone git://$some_url\n  # Does all the work of setting up metadata/refs\n  git svn bootstrap --stdlayout $remote_svn_url\n"},{"id":"141259","messageId":"t2l86ecb3c71005081120lf87fc99bh1566c82a4d21904e@mail.gmail.com","threadId":"23748","inReplyTo":"AANLkTikR5EpZhwFXmRfmE9jSNtiuFoGwTgGbpcwUBxnq@mail.gmail.com","subject":"Re: Reconstructing git-svn metadata after a git clone","fromName":"Dmitrijs Ledkovs","fromEmail":"dmitrij.ledkov@ubuntu.com","sentAt":"2010-05-08T18:20:59Z","receivedAt":"2010-05-08T18:20:59Z","isPatch":false,"sender":{"key":"dmitrij.ledkov@ubuntu.com","avatar":"https://gravatar.com/avatar/79b618c6eb391ddb486c2ee0d3b429014e0e9ba579fd55f2e035f482fc8c24f4?d=mp&s=160"},"body":"On 8 May 2010 18:58, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> So:\n>\n>  * Am I doing something wrong? If so I can't see what it is.\n>\n\nNo\n\n>  * Is there something that works for the general case, i.e. you only\n>   have to know the original `git svn init` options. If there is I'd\n>   like to document that & submit a patch.\n>\n\nIn my repo I have a branch with no anestors which has a config file,\nsetup.sh & fetch.sh\n\nI instruct to clone repo, checkout \"utils\" branch, run setup.sh (it\noverrides .git/config with config file committed to utils branch and\nafter it copied config it runs git svn init URL), fetch.sh just runs\ngit svn fetch =)\n\nI have to do this because I have two svn remotes and both are not\nstandard layout. The only way for me to represet \"same init options\"\nis by editing .git/config cause it's impossible for me to supply\ngit-init options on the command line =)\n\n>  * Depending on the above; can git-svn itself be friendlier here?\n>   Maybe by having a `git svn bootstrap` command. E.g.:\n>\n\nIt would be nice to have the git init info propogate with the git\nclone. But this won't work. You are cloning *all* branches and\nproviding a git mirror, where as I want to to git init just my svn\nbranch or a subset of them. To achieve that I will clone just the\nbranches I need modify my .git/config and get the result I want.\n\nSo imho git-svn is ok here.\n\n>  git clone git://$some_url\n>  # Does all the work of setting up metadata/refs\n>  git svn bootstrap --stdlayout $remote_svn_url\n\n\nIf you are committing to svn regularly you are better of with bzr-svn\nin my opinion. Because launchpad can run automatic imports for you\n(webkit is already running btw) and the whole bootstrapping thing is\ndone the way you are expecting it.\n\n#create repository to store revisions efficiently\n$ bzr init-repo .\n\n#on the first ever run it will rebuild meta-data\n#subsequent runs just fetch missing revisions\n$ bzr branch svn://path.to.any.branch\n\nAnd you can commit from that =) and bzr can operate on your svn\ncheckouts. And every single clone done by bzr-svn is identical (unlike\ngit where everyone has to follow the same git-svn mirror to get same\nrevision-ids).\n\nYou have two options when commiting with bzr-svn. Regular bzr ci will\nstore bzr merge information in revision properties on svn server or\nyou can use bzr dpush which is like git svn dcommit.\n\nIMHO bzr-svn is the best when you need to commit back to svn and\npainlessly commit parts of the feature branch and merge other bits\nlater.\n"},{"id":"141443","messageId":"AANLkTin_hVhyrhjwoM_on_sd1RIZrk6YIFVsYXKm5UKg@mail.gmail.com","threadId":"23748","inReplyTo":"t2l86ecb3c71005081120lf87fc99bh1566c82a4d21904e@mail.gmail.com","subject":"Re: Reconstructing git-svn metadata after a git clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-05-10T21:43:11Z","receivedAt":"2010-05-10T21:43:11Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, May 8, 2010 at 18:20, Dmitrijs Ledkovs\n<dmitrij.ledkov@ubuntu.com> wrote:\n> On 8 May 2010 18:58, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n>> So:\n>>\n>>  * Am I doing something wrong? If so I can't see what it is.\n>>\n>\n> No\n>\n>>  * Is there something that works for the general case, i.e. you only\n>>   have to know the original `git svn init` options. If there is I'd\n>>   like to document that & submit a patch.\n>>\n>\n> In my repo I have a branch with no anestors which has a config file,\n> setup.sh & fetch.sh\n>\n> I instruct to clone repo, checkout \"utils\" branch, run setup.sh (it\n> overrides .git/config with config file committed to utils branch and\n> after it copied config it runs git svn init URL), fetch.sh just runs\n> git svn fetch =)\n>\n> I have to do this because I have two svn remotes and both are not\n> standard layout. The only way for me to represet \"same init options\"\n> is by editing .git/config cause it's impossible for me to supply\n> git-init options on the command line =)\n\nIf I can't resolve this I'll do something similar. Just provide a\ntarball of .git/svn for each repo that is.\n\n>>  * Depending on the above; can git-svn itself be friendlier here?\n>>   Maybe by having a `git svn bootstrap` command. E.g.:\n>>\n>\n> It would be nice to have the git init info propogate with the git\n> clone. But this won't work. You are cloning *all* branches and\n> providing a git mirror, where as I want to to git init just my svn\n> branch or a subset of them. To achieve that I will clone just the\n> branches I need modify my .git/config and get the result I want.\n>\n> So imho git-svn is ok here.\n\nI still don't see why it can't work. Maybe I'm just hopelessly naïve\nto git-svn internals, but:\n\n   * Every commit message in the Git repo has info on the svn\n     branch/svn commit.\n\n   * The Git repo has a list of branches that map to SVN branches.\n\n   * If I supply the same `git svn init` options that made the repo,\n     it should be able to bootstrap just using the above.\n\nWhy isn't that the case? What info is in .git/svn that can't be\ninferred from the above?\n\n>>  git clone git://$some_url\n>>  # Does all the work of setting up metadata/refs\n>>  git svn bootstrap --stdlayout $remote_svn_url\n>\n> If you are committing to svn regularly you are better of with bzr-svn\n> in my opinion. Because launchpad can run automatic imports for you\n> (webkit is already running btw) and the whole bootstrapping thing is\n> done the way you are expecting it.\n>\n> #create repository to store revisions efficiently\n> $ bzr init-repo .\n>\n> #on the first ever run it will rebuild meta-data\n> #subsequent runs just fetch missing revisions\n> $ bzr branch svn://path.to.any.branch\n>\n> And you can commit from that =) and bzr can operate on your svn\n> checkouts. And every single clone done by bzr-svn is identical (unlike\n> git where everyone has to follow the same git-svn mirror to get same\n> revision-ids).\n>\n> You have two options when commiting with bzr-svn. Regular bzr ci will\n> store bzr merge information in revision properties on svn server or\n> you can use bzr dpush which is like git svn dcommit.\n>\n> IMHO bzr-svn is the best when you need to commit back to svn and\n> painlessly commit parts of the feature branch and merge other bits\n> later.\n\nThat's informative. But from having used Bazaar a bit my experience\nwith it was that I might as well be using Subversion.\n"},{"id":"141449","messageId":"AANLkTinMZvBUPPmChwrA1dHr47HXpFldnKH9GO60oIkU@mail.gmail.com","threadId":"23748","inReplyTo":"AANLkTin_hVhyrhjwoM_on_sd1RIZrk6YIFVsYXKm5UKg@mail.gmail.com","subject":"Re: Reconstructing git-svn metadata after a git clone","fromName":"Dmitrijs Ledkovs","fromEmail":"dmitrij.ledkov@ubuntu.com","sentAt":"2010-05-11T04:27:05Z","receivedAt":"2010-05-11T04:27:05Z","isPatch":false,"sender":{"key":"dmitrij.ledkov@ubuntu.com","avatar":"https://gravatar.com/avatar/79b618c6eb391ddb486c2ee0d3b429014e0e9ba579fd55f2e035f482fc8c24f4?d=mp&s=160"},"body":"On 10 May 2010 22:43, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> On Sat, May 8, 2010 at 18:20, Dmitrijs Ledkovs\n> <dmitrij.ledkov@ubuntu.com> wrote:\n>> On 8 May 2010 18:58, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n>>> So:\n>>>\n>>>  * Am I doing something wrong? If so I can't see what it is.\n>>>\n>>\n>> No\n>>\n>>>  * Is there something that works for the general case, i.e. you only\n>>>   have to know the original `git svn init` options. If there is I'd\n>>>   like to document that & submit a patch.\n>>>\n>>\n>> In my repo I have a branch with no anestors which has a config file,\n>> setup.sh & fetch.sh\n>>\n>> I instruct to clone repo, checkout \"utils\" branch, run setup.sh (it\n>> overrides .git/config with config file committed to utils branch and\n>> after it copied config it runs git svn init URL), fetch.sh just runs\n>> git svn fetch =)\n>>\n>> I have to do this because I have two svn remotes and both are not\n>> standard layout. The only way for me to represet \"same init options\"\n>> is by editing .git/config cause it's impossible for me to supply\n>> git-init options on the command line =)\n>\n> If I can't resolve this I'll do something similar. Just provide a\n> tarball of .git/svn for each repo that is.\n>\n\nAlways an option.\n\n>>>  * Depending on the above; can git-svn itself be friendlier here?\n>>>   Maybe by having a `git svn bootstrap` command. E.g.:\n>>>\n>>\n>> It would be nice to have the git init info propogate with the git\n>> clone. But this won't work. You are cloning *all* branches and\n>> providing a git mirror, where as I want to to git init just my svn\n>> branch or a subset of them. To achieve that I will clone just the\n>> branches I need modify my .git/config and get the result I want.\n>>\n>> So imho git-svn is ok here.\n>\n> I still don't see why it can't work. Maybe I'm just hopelessly naïve\n> to git-svn internals, but:\n>\n>   * Every commit message in the Git repo has info on the svn\n>     branch/svn commit.\n>\n>   * The Git repo has a list of branches that map to SVN branches.\n>\n\nYour local one does. But not the cloned one.\n\n>   * If I supply the same `git svn init` options that made the repo,\n>     it should be able to bootstrap just using the above.\n>\n\nYes, but it takes a while..... especially if the svn repo is huge.\n\n> Why isn't that the case? What info is in .git/svn that can't be\n> inferred from the above?\n>\n\nIs there a git versions mismatch between machine where the first\nimport was made and the machine where you are testing bootstrapping by\nany chance?\n\non my machine I sometimes see \"upgrading metadata\" when I'm updating\nsome of my acient git-svn clones.\n\n>>>  git clone git://$some_url\n>>>  # Does all the work of setting up metadata/refs\n>>>  git svn bootstrap --stdlayout $remote_svn_url\n>>\n>> If you are committing to svn regularly you are better of with bzr-svn\n>> in my opinion. Because launchpad can run automatic imports for you\n>> (webkit is already running btw) and the whole bootstrapping thing is\n>> done the way you are expecting it.\n>>\n>> #create repository to store revisions efficiently\n>> $ bzr init-repo .\n>>\n>> #on the first ever run it will rebuild meta-data\n>> #subsequent runs just fetch missing revisions\n>> $ bzr branch svn://path.to.any.branch\n>>\n>> And you can commit from that =) and bzr can operate on your svn\n>> checkouts. And every single clone done by bzr-svn is identical (unlike\n>> git where everyone has to follow the same git-svn mirror to get same\n>> revision-ids).\n>>\n>> You have two options when commiting with bzr-svn. Regular bzr ci will\n>> store bzr merge information in revision properties on svn server or\n>> you can use bzr dpush which is like git svn dcommit.\n>>\n>> IMHO bzr-svn is the best when you need to commit back to svn and\n>> painlessly commit parts of the feature branch and merge other bits\n>> later.\n>\n> That's informative. But from having used Bazaar a bit my experience\n> with it was that I might as well be using Subversion.\n>\n"},{"id":"141477","messageId":"AANLkTinZWtMUFtL-Z9gZQQ2pDrPpMooO95l_ylNuD2A9@mail.gmail.com","threadId":"23748","inReplyTo":"AANLkTinMZvBUPPmChwrA1dHr47HXpFldnKH9GO60oIkU@mail.gmail.com","subject":"Re: Reconstructing git-svn metadata after a git clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-05-11T13:04:17Z","receivedAt":"2010-05-11T13:04:17Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Tue, May 11, 2010 at 04:27, Dmitrijs Ledkovs\n<dmitrij.ledkov@ubuntu.com> wrote:\n> On 10 May 2010 22:43, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n>> Why isn't that the case? What info is in .git/svn that can't be\n>> inferred from the above?\n>>\n>\n> Is there a git versions mismatch between machine where the first\n> import was made and the machine where you are testing bootstrapping by\n> any chance?\n>\n> on my machine I sometimes see \"upgrading metadata\" when I'm updating\n> some of my acient git-svn clones.\n\nNo, they were both running 1.7.1 from Debian.\n"}]}