{"thread":{"id":"21182","subject":"git-svn very slow on fetch (shared git-svn repo)","startedAt":"2009-10-10T09:10:45Z","lastAt":"2009-10-11T09:37:36Z","messageCount":5,"participants":["Pascal Obry","Eric Wong"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"124561","messageId":"4AD04F95.7050603@obry.net","threadId":"21182","inReplyTo":null,"subject":"git-svn very slow on fetch (shared git-svn repo)","fromName":"Pascal Obry","fromEmail":"pascal@obry.net","sentAt":"2009-10-10T09:10:45Z","receivedAt":"2009-10-10T09:10:45Z","isPatch":false,"sender":{"key":"pascal@obry.net","avatar":"https://avatars.githubusercontent.com/u/467069?v=4"},"body":"\nHere is the problem.\n\nDoing a:\n\n    $ git svn fetch\n\nTakes age to update the repository. The long story is that I'm trying to \nhave shared git-svn repositories.\n\nI'm cloning the repository on a server using a standard \"git svn clone\". \nI then let users cloning this repository using the procedure described \nin git-svn help. Copy/paste here:\n\n<<\n        # Do the initial import on a server\n                ssh server \"cd /pub && git svn clone \nhttp://svn.example.com/project\n        # Clone locally - make sure the refs/remotes/ space matches the \nserver\n                mkdir project\n                cd project\n                git init\n                git remote add origin server:/pub/project\n                git config --add remote.origin.fetch \n'+refs/remotes/*:refs/remotes/*'\n                git fetch\n        # Create a local branch from one of the branches just fetched\n                git checkout -b master FETCH_HEAD\n        # Initialize 'git svn' locally (be sure to use the same URL and \n-T/-b/-t options as were used on server)\n                git svn init http://svn.example.com/project\n        # Pull the latest changes from Subversion\n                git svn rebase\n >>\n\nIf you do a \"git svn fetch\" (to get new branches) it takes age if you \nhave imported branches that are not used since a long time.\n\nI've traced this down to the Perl fetch_all procedure. It seems that the \nfetch is looking at the older version in all branches and then read the \nremote repository starting from this revision. As some branches are \nunused since a very long it it re-read most of the history. In my \nexample it start at rev 8200 whereas the last revision on trunk is \n150000 (I put trace in git-svn Perl script).\n\nI have observed that this happen only the first time.\n\nThis can be confirmed by the fact that if you break git-svn fetch \nprocess and restart it it will start to a later revision. So it seems \nthat git-svn is keeping some kind of data about what has already been \nfetched but those data are not properly copied by the procedure above.\n\nIs there a workaround that? Where are those data stored?\n\nAny guidance would be very welcomed.\n\nThanks,\nPascal.\n\n-- \n\n--|------------------------------------------------------\n--| Pascal Obry                           Team-Ada Member\n--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE\n--|------------------------------------------------------\n--|    http://www.obry.net  -  http://v2p.fr.eu.org\n--| \"The best way to travel is by means of imagination\"\n--|\n--| gpg --keyserver keys.gnupg.net --recv-key F949BD3B\n"},{"id":"124589","messageId":"20091011070136.GB16264@dcvr.yhbt.net","threadId":"21182","inReplyTo":"4AD04F95.7050603@obry.net","subject":"Re: git-svn very slow on fetch (shared git-svn repo)","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-10-11T07:01:36Z","receivedAt":"2009-10-11T07:01:36Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Pascal Obry <pascal@obry.net> wrote:\n> Here is the problem.\n>\n> Doing a:\n>\n>    $ git svn fetch\n>\n> Takes age to update the repository. The long story is that I'm trying to  \n> have shared git-svn repositories.\n>\n> I'm cloning the repository on a server using a standard \"git svn clone\".  \n> I then let users cloning this repository using the procedure described  \n> in git-svn help. Copy/paste here:\n>\n> <<\n>        # Do the initial import on a server\n>                ssh server \"cd /pub && git svn clone  \n> http://svn.example.com/project\n>        # Clone locally - make sure the refs/remotes/ space matches the  \n> server\n>                mkdir project\n>                cd project\n>                git init\n>                git remote add origin server:/pub/project\n>                git config --add remote.origin.fetch  \n> '+refs/remotes/*:refs/remotes/*'\n>                git fetch\n>        # Create a local branch from one of the branches just fetched\n>                git checkout -b master FETCH_HEAD\n>        # Initialize 'git svn' locally (be sure to use the same URL and  \n> -T/-b/-t options as were used on server)\n>                git svn init http://svn.example.com/project\n>        # Pull the latest changes from Subversion\n>                git svn rebase\n> >>\n>\n> If you do a \"git svn fetch\" (to get new branches) it takes age if you  \n> have imported branches that are not used since a long time.\n>\n> I've traced this down to the Perl fetch_all procedure. It seems that the  \n> fetch is looking at the older version in all branches and then read the  \n> remote repository starting from this revision. As some branches are  \n> unused since a very long it it re-read most of the history. In my  \n> example it start at rev 8200 whereas the last revision on trunk is  \n> 150000 (I put trace in git-svn Perl script).\n>\n> I have observed that this happen only the first time.\n>\n> This can be confirmed by the fact that if you break git-svn fetch  \n> process and restart it it will start to a later revision. So it seems  \n> that git-svn is keeping some kind of data about what has already been  \n> fetched but those data are not properly copied by the procedure above.\n>\n> Is there a workaround that? Where are those data stored?\n\nHi Pascal,\n\nFor globs (branches and tags) the $GIT_DIR/svn/.metadata\nconfig file, under the svn-remote.svn.{branches,tags}-maxRev keys.\n\nFor explicitly specified refs (e.g. \"trunk\"), then it's in the last\nrecord of the corresponding rev_map (e.g.\n$GIT_DIR/svn/trunk/.rev_map.$UUID) as a 4-byte revision number + 20\nbytes of zeroes.\n\n-- \nEric Wong\n"},{"id":"124593","messageId":"4AD19E9F.8010302@obry.net","threadId":"21182","inReplyTo":"20091011070136.GB16264@dcvr.yhbt.net","subject":"Re: git-svn very slow on fetch (shared git-svn repo)","fromName":"Pascal Obry","fromEmail":"pascal@obry.net","sentAt":"2009-10-11T09:00:15Z","receivedAt":"2009-10-11T09:00:15Z","isPatch":false,"sender":{"key":"pascal@obry.net","avatar":"https://avatars.githubusercontent.com/u/467069?v=4"},"body":"Hi Eric,\n\n> For globs (branches and tags) the $GIT_DIR/svn/.metadata\n> config file, under the svn-remote.svn.{branches,tags}-maxRev keys.\n\nRight. I found out that copying the .metadata solve this problem.\n\n> For explicitly specified refs (e.g. \"trunk\"), then it's in the last\n> record of the corresponding rev_map (e.g.\n> $GIT_DIR/svn/trunk/.rev_map.$UUID) as a 4-byte revision number + 20\n> bytes of zeroes.\n\nSeems like we do not need to copy this one as it is reconstructed by a \nsimple rebase on the trunk branch. This is quite fast as reconstructed \nfrom the embedded git-svn-id if I'm not mistaken.\n\nSo you confirm that copying the .metadata is the solution?\n\nPascal.\n\n-- \n\n--|------------------------------------------------------\n--| Pascal Obry                           Team-Ada Member\n--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE\n--|------------------------------------------------------\n--|    http://www.obry.net  -  http://v2p.fr.eu.org\n--| \"The best way to travel is by means of imagination\"\n--|\n--| gpg --keyserver keys.gnupg.net --recv-key F949BD3B\n"},{"id":"124594","messageId":"20091011090338.GA27480@dcvr.yhbt.net","threadId":"21182","inReplyTo":"4AD19E9F.8010302@obry.net","subject":"Re: git-svn very slow on fetch (shared git-svn repo)","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-10-11T09:03:38Z","receivedAt":"2009-10-11T09:03:38Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Pascal Obry <pascal@obry.net> wrote:\n> Hi Eric,\n>\n>> For globs (branches and tags) the $GIT_DIR/svn/.metadata\n>> config file, under the svn-remote.svn.{branches,tags}-maxRev keys.\n>\n> Right. I found out that copying the .metadata solve this problem.\n>\n>> For explicitly specified refs (e.g. \"trunk\"), then it's in the last\n>> record of the corresponding rev_map (e.g.\n>> $GIT_DIR/svn/trunk/.rev_map.$UUID) as a 4-byte revision number + 20\n>> bytes of zeroes.\n>\n> Seems like we do not need to copy this one as it is reconstructed by a  \n> simple rebase on the trunk branch. This is quite fast as reconstructed  \n> from the embedded git-svn-id if I'm not mistaken.\n>\n> So you confirm that copying the .metadata is the solution?\n\nYes, copying .metadata should be enough.  The .rev_map rebuild should\nbe fast.\n\n-- \nEric Wong\n"},{"id":"124596","messageId":"4AD1A760.9060006@obry.net","threadId":"21182","inReplyTo":"20091011090338.GA27480@dcvr.yhbt.net","subject":"Re: git-svn very slow on fetch (shared git-svn repo)","fromName":"Pascal Obry","fromEmail":"pascal@obry.net","sentAt":"2009-10-11T09:37:36Z","receivedAt":"2009-10-11T09:37:36Z","isPatch":false,"sender":{"key":"pascal@obry.net","avatar":"https://avatars.githubusercontent.com/u/467069?v=4"},"body":"Eric,\n\n> Yes, copying .metadata should be enough.\n\nI've put this in place. Works fine.\n\n > The .rev_map rebuild should be fast.\n\nYep. Thanks for your quick reply.\n\nPascal.\n\n-- \n\n--|------------------------------------------------------\n--| Pascal Obry                           Team-Ada Member\n--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE\n--|------------------------------------------------------\n--|    http://www.obry.net  -  http://v2p.fr.eu.org\n--| \"The best way to travel is by means of imagination\"\n--|\n--| gpg --keyserver keys.gnupg.net --recv-key F949BD3B\n"}]}