{"thread":{"id":"2706","subject":"[PATCH] SVN import: Use one log call","startedAt":"2005-11-29T07:13:04Z","lastAt":"2005-11-30T08:34:39Z","messageCount":7,"participants":["Matthias Urlichs","Junio C Hamano","Nicolas Pitre"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"12922","messageId":"pan.2005.11.29.07.13.02.145977@smurf.noris.de","threadId":"2706","inReplyTo":null,"subject":"[PATCH] SVN import: Use one log call","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-11-29T07:13:04Z","receivedAt":"2005-11-29T07:13:04Z","isPatch":true,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"One \"svn log\" (or its equivalent) per revision adds delay and server load.\nInstead, open two SVN connections -- one for the log, and one for the files.\n\nPositive side effect: Only those log entries which actually contain data\nare committed => no more empty commits.\n\nAlso, change the \"-l\" option to set the maximum revision to be pulled,\nnot the number of revisions.\n\n---\n\n... or pull from http://netz.smurf.noris.de/git/git.git#svnup.\n\n Documentation/git-svnimport.txt | 15 ++++++---------\n git-svnimport.perl | 37 ++++++++++++++++---------------------\n 2 files changed, 22 insertions(+), 30 deletions(-)\n\napplies-to: 744acb3ef516240503d9a13cf1adb6a9670f905e\n81da2a98762ef4965e6410c1c83600f2d80dc67a\ndiff --git a/Documentation/git-svnimport.txt b/Documentation/git-svnimport.txt\nindex fcc79fa..f8dbee7 100644\n--- a/Documentation/git-svnimport.txt\n+++ b/Documentation/git-svnimport.txt\n@@ -10,7 +10,7 @@ git-svnimport - Import a SVN repository \n SYNOPSIS\n --------\n 'git-svnimport' [ -o ] [ -h ] [ -v ] [ -d | -D ]\n-\t\t\t[ -C ] [ -i ] [ -u ] [-l limit_nr_changes]\n+\t\t\t[ -C ] [ -i ] [ -u ] [-l limit_rev]\n \t\t\t[ -b branch_subdir ] [ -t trunk_subdir ] [ -T tag_subdir ]\n \t\t\t[ -s start_chg ] [ -m ] [ -M regex ]\n \t\t\t [ ]\n@@ -71,14 +71,11 @@ When importing incementally, you might n\n \tregex. It can be used with -m to also see the default regexes.\n \tYou must escape forward slashes.\n \n--l ::\n-\tLimit the number of SVN changesets we pull before quitting.\n-\tThis option is necessary because the SVN library has serious memory\n-\tleaks; the recommended value for nontrivial imports is 100.\n-\n-\tgit-svnimport will still exit with a zero exit code. You can check\n-\tthe size of the file \".git/svn2git\" to determine whether to call\n-\tthe importer again.\n+-l ::\n+\tSpecify a maximum revision number to pull.\n+\n+\tFormerly, this option controlled how many revisions to pull, due to\n+\tSVN memory leaks. (These have been worked around.)\n \n -v::\n \tVerbosity: let 'svnimport' report what it is doing.\ndiff --git a/git-svnimport.perl b/git-svnimport.perl\nindex af13fdd..640d675 100755\n--- a/git-svnimport.perl\n+++ b/git-svnimport.perl\n@@ -35,7 +35,7 @@ our($opt_h,$opt_o,$opt_v,$opt_u,$opt_C,$\n sub usage() {\n \tprint STDERR <new($svn);\n $svn = SVNconn->new($svn);\n \n my $lwp_ua;\n@@ -198,7 +199,7 @@ $ENV{GIT_INDEX_FILE} = $git_index;\n my $maxnum = 0;\n my $last_rev = \"\";\n my $last_branch;\n-my $current_rev = $opt_s-1;\n+my $current_rev = $opt_s || 1;\n unless(-d $git_dir) {\n \tsystem(\"git-init-db\");\n \tdie \"Cannot init the GIT db at $git_tree: $?\\n\" if $?;\n@@ -250,7 +251,7 @@ EOM\n \t\tmy($num,$branch,$ref) = split;\n \t\t$branches{$branch}{$num} = $ref;\n \t\t$branches{$branch}{\"LAST\"} = $ref;\n-\t\t$current_rev = $num if $current_rev < $num;\n+\t\t$current_rev = $num+1 if $current_rev <= $num;\n \t}\n \tclose($B);\n }\n@@ -704,17 +705,17 @@ sub commit {\n \tprint \"DONE: $revision $dest $cid\\n\" if $opt_v;\n }\n \n-my ($changed_paths, $revision, $author, $date, $message, $pool) = @_;\n-sub _commit_all {\n-\t($changed_paths, $revision, $author, $date, $message, $pool) = @_;\n+sub commit_all {\n+\t# Recursive use of the SVN connection does not work\n+\tlocal $svn = $svn2;\n+\n+\tmy ($changed_paths, $revision, $author, $date, $message, $pool) = @_;\n \tmy %p;\n \twhile(my($path,$action) = each %$changed_paths) {\n \t\t$p{$path} = [ $action->action,$action->copyfrom_path, $action->copyfrom_rev, $path ];\n \t}\n \t$changed_paths = \\%p;\n-}\n \n-sub commit_all {\n \tmy %done;\n \tmy @col;\n \tmy $pref;\n@@ -730,18 +731,12 @@ sub commit_all {\n \t}\n }\n \n-while(++$current_rev <= $svn->{'maxrev'}) {\n-\tif (defined $opt_l) {\n-\t\t$opt_l--;\n-\t\tif ($opt_l < 0) {\n-\t\t\tlast;\n-\t\t}\n-\t}\n-\tmy $pool=SVN::Pool->new;\n-\t$svn->{'svn'}->get_log(\"/\",$current_rev,$current_rev,1,1,1,\\&_commit_all,$pool);\n-\t$pool->clear;\n-\tcommit_all();\n-}\n+$opt_l = $svn->{'maxrev'} if not defined $opt_l or $opt_l > $svn->{'maxrev'};\n+print \"Fetching from $current_rev to $opt_l ...\\n\" if $opt_v;\n+\n+my $pool=SVN::Pool->new;\n+$svn->{'svn'}->get_log(\"/\",$current_rev,$opt_l,0,1,1,\\&commit_all,$pool);\n+$pool->clear;\n \n \n unlink($git_index);\n---\n\n\n-- \nMatthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de\nDisclaimer: The quote was selected randomly. Really. | http://smurf.noris.de\n - -\nThere was a young peasant named Gorse Who fell madly in love with his horse.\nSaid his wife, \"You rapscallion, That horse is a stallion -- This constitutes\ngrounds for divorce.\"\n"},{"id":"12943","messageId":"7vhd9vdx1o.fsf@assigned-by-dhcp.cox.net","threadId":"2706","inReplyTo":"pan.2005.11.29.07.13.02.145977@smurf.noris.de","subject":"Re: [PATCH] SVN import: Use one log call","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-29T20:22:11Z","receivedAt":"2005-11-29T20:22:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthias Urlichs writes:\n\n> One \"svn log\" (or its equivalent) per revision adds delay and server load.\n> Instead, open two SVN connections -- one for the log, and one for the files.\n\nThanks, applied and pushed out.\n\nBTW, I've never successfully managed to run svnimport from my\nprivate svn repository. Admittedly the repository does not\nfollow the recommended layout and that probably is the major\ncause (it started its life when svn documentation recommended\n{trunk,branches,tags}/{projectA,projectB,projectC} layout.\n{projectA,projectB,projectC}/{trunk,branches,tags} is the layout\nthey recommend these days, I think [*1*]).\n\nIt does not use any branches (it is primarily a random\ncollection of small throwaway scripts). The repository hosts\nmany unrelated pieces (\"http://127.0.0.1/svn/private/\" is the\nroot level of the repository), organized like this:\n\n $ svn ls http://127.0.0.1/svn/private/\n attic/\n main/\n tags/\n\n main/ is the active one (trunk), and it has bunch of\n unrelated subdirectories.\n\n attic/ is where I prepared to \"svn mv\" things from main/ that\n are no longer needed, but is empty.\n\n tags/ have one tree that is a copy of one of the subtrees\n under main/ from distant past.\n\nI wanted to convert one of the subsubdirectory of main to git.\n\n $ svn ls http://127.0.0.1/svn/private/main/sources/photocat\n Makefile\n Notes\n cmdmason.pm\n ...\n\nWhat is the svnimport command line I should give? Luckily, I do\nnot have \"tags\" or \"branches\" under private/main/sources/, so I\ntried to cheat like this, hoping it would mistake \"photocat\" is the trunk of \n\"main/sources\" project in the repository. No such luck.\n\n$ cd /var/tmp && rm -fr try0 && mkdir try0 && cd try0\n$ git svnimport -v -i -t photocat http://127.0.0.1/svn/private main/sources\n1: Unrecognized path: /main/sources\n1: Unrecognized path: /main/in-place\n1: Unrecognized path: /main\n...\n1500: Unrecognized path: /main/sources/photocat/db/catalog.sql\n1501: Unrecognized path: /main/sources/photocat/data/035-maribon-making.yaml\nDONE; creating master branch\ncp: cannot stat `/var/tmp/try0/.git/refs/heads/origin': No such file or directory\nfatal: master: not a valid SHA1\n$ \n\nIf your answer is \"your repository layout is too weird and\nnonstandard, you are screwed\", that is perfectly fine. I do not\nwant you to bend over backwards to butcher the import script to\nsupport it, if it is too nonstandard. I already converted what\nI wanted to convert manually already; history being linear\nwithout branches, that was easy enough.\n\nBut I thought it would never hurt to ask ;-).\n\n[Footnote]\n\n*1*\n\nhttp://svnbook.red-bean.com/en/1.1/svn-book.html#svn-ch-5-sect-6.1\nshows two layouts, one with {trunk,tags,branches} at the top\nlevel of each project, another with nested projects (if you look\nat \"utils\" in the picture as a project with two subcomponents\n\"calc\" and \"calendar\"), with {trunk,tags,branches} under each\nsubproject. I think the current code should import from\n\"calendar\" or \"calc\" level just fine, but I wonder if we want to\nsupport importing from \"utils\" level.\n"},{"id":"12971","messageId":"20051130024610.GX20418@kiste.smurf.noris.de","threadId":"2706","inReplyTo":"7vhd9vdx1o.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] SVN import: Use one log call","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-11-30T02:46:10Z","receivedAt":"2005-11-30T02:46:10Z","isPatch":true,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"Hi,\n\nJunio C Hamano:\n> $ git svnimport -v -i -t photocat http://127.0.0.1/svn/private main/sources\n> ...\n> 1500: Unrecognized path: /main/sources/photocat/db/catalog.sql\n\nHmmm, \"git-svnimport -T main/sources\" should work. Somewhat.\n\n>>/\n>> utils/\n>> calc/\n>> trunk/\n>> tags/\n>> branches/\n>> calendar/\n>> trunk/\n>> tags/\n>> branches/\n>> …\n\n> but I wonder if we want to\n> support importing from \"utils\" level.\n> \nThat'd require somewhat more flexible support for reordering the\nrepository path elements than we have now, I'm afraid, thus the answer\nto that particular question is \"import both separately, then merge\ninto a common git repo if you really need to\". We *can* do that, after\nall. ;-)\n\n-- \nMatthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de\nDisclaimer: The quote was selected randomly. Really. | http://smurf.noris.de\n - -\nYou goddamn cornhuskers are all alike.\n\t\t\t\t\t-- Jim Thompson\n"},{"id":"12974","messageId":"Pine.LNX.4.64.0511292247591.3122@localhost.localdomain","threadId":"2706","inReplyTo":"7vhd9vdx1o.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] SVN import: Use one log call","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2005-11-30T03:54:21Z","receivedAt":"2005-11-30T03:54:21Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 29 Nov 2005, Junio C Hamano wrote:\n\n> BTW, I've never successfully managed to run svnimport from my\n> private svn repository.\n\nSame here, with the following public repo: svn://mielke.cc/main/brltty\n\n> $ git svnimport -v -i -t photocat http://127.0.0.1/svn/private main/sources\n> 1: Unrecognized path: /main/sources\n> 1: Unrecognized path: /main/in-place\n> 1: Unrecognized path: /main\n> ...\n> 1500: Unrecognized path: /main/sources/photocat/db/catalog.sql\n> 1501: Unrecognized path: /main/sources/photocat/data/035-maribon-making.yaml\n> DONE; creating master branch\n> cp: cannot stat `/var/tmp/try0/.git/refs/heads/origin': No such file or directory\n> fatal: master: not a valid SHA1\n> $ \n\nAnd I only get similar results.\n\n> If your answer is \"your repository layout is too weird and\n> nonstandard, you are screwed\", that is perfectly fine. I do not\n> want you to bend over backwards to butcher the import script to\n> support it, if it is too nonstandard.\n\nThing is the above repository is not _that_ weird. And with the real \nsvn it produces a proper source tree of course, without any special \noptions. So I would think git-svnimport should be able to do the same. \nNo?\n\n\nNicolas\n"},{"id":"12980","messageId":"20051130065241.GY20418@kiste.smurf.noris.de","threadId":"2706","inReplyTo":"Pine.LNX.4.64.0511292247591.3122@localhost.localdomain","subject":"Re: [PATCH] SVN import: Use one log call","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-11-30T06:52:42Z","receivedAt":"2005-11-30T06:52:42Z","isPatch":true,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"Hi,\n\nNicolas Pitre:\n> Same here, with the following public repo: svn://mielke.cc/main/brltty\n> \nSame thing -- should work with a \"-T main/brltty\" option.\n\n> Thing is the above repository is not _that_ weird. And with the real \n> svn it produces a proper source tree of course, without any special \n> options. So I would think git-svnimport should be able to do the same. \n> No?\n\nSure. The problem is that svn doesn't have tags, branches, or different\nprojects; it's all just subdirectories with linked files.\n\nSo you need a good mapping from one to the other. In principle that's\nsimple, it's just that nobody has written the code for a more flexible\nmapping from svn paths to tags / branch+subpath yet.\n\n-- \nMatthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de\nDisclaimer: The quote was selected randomly. Really. | http://smurf.noris.de\n - -\n\"Adam may not have been so perfect after the 'fall,\" but he was\n not so big a fool.\"\n [Lemuel K. Washburn, _Is The Bible Worth Reading\n And Other Essays_]\n"},{"id":"12982","messageId":"7vwtiqwoy7.fsf@assigned-by-dhcp.cox.net","threadId":"2706","inReplyTo":"20051130065241.GY20418@kiste.smurf.noris.de","subject":"Re: [PATCH] SVN import: Use one log call","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-30T07:54:24Z","receivedAt":"2005-11-30T07:54:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthias Urlichs writes:\n\n> Nicolas Pitre:\n>> Same here, with the following public repo: svn://mielke.cc/main/brltty\n>> \n> Same thing -- should work with a \"-T main/brltty\" option.\n\nSo would the full command line be like this?\n\n\t$ rm -rf try1; mkdir try1; cd try1\n $ git-svnimport -T main/brltty svn://mielke.cc/main/brltty\n\nI realize my first attempt was incorrectly using \"-t\" (tags)\ninstead of \"-T\" (trunk), which was obviously incorrect, but\nreplacing that lowercase -t with -T did not work well. However,\nas you told me:\n\n\t$ git svnimport -T main/sources \\\n\t http://127.0.0.1/svn/private/main/sources/photocat\n\nseems to be doing something. But I do not understand why. How\ndid you decide \"-T main/sources\" is good for my case while \"-T\nmain/brltty\" is good for Nicolas? I would have guessed you may\nsay \"-T main\" for Nicolas, dropping the last path component.\n"},{"id":"12984","messageId":"20051130083436.GZ20418@kiste.smurf.noris.de","threadId":"2706","inReplyTo":"7vwtiqwoy7.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] SVN import: Use one log call","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-11-30T08:34:39Z","receivedAt":"2005-11-30T08:34:39Z","isPatch":true,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"Hi,\n\nJunio C Hamano:\n> \t$ git svnimport -T main/sources \\\n> \t http://127.0.0.1/svn/private/main/sources/photocat\n> \n> seems to be doing something. But I do not understand why. How\n> did you decide \"-T main/sources\" is good for my case while \"-T\n> main/brltty\" is good for Nicolas? I would have guessed you may\n> say \"-T main\" for Nicolas, dropping the last path component.\n> \nWhat git-svnimport does right now is rather simple. It pulls the log\nfrom the SVN URL, drops everything that does not have a prefix of either\nof the -t tags -T trunk -b branches options. The prefix is then thrown\naway. If it's a trunk, import the rest to the \"origin\" branch; otherwise\nremove the next path component and use that as either the branch name\nor the tag name.\n\nThe point is that the picture in the SVN manual is incomplete. \nThere's not just a branches/ subdirectory alongside the trunk/ one;\ntrunk/ directly contains the project's files, while branches/ contains\nsubdirectories which are named for the branch, and *they* contain the\nfiles.\n\n-- \nMatthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de\nDisclaimer: The quote was selected randomly. Really. | http://smurf.noris.de\n - -\nIf you're looking ahead long-term, and what you see looks like\nscience fiction, it might be wrong. But if it doesn't look like\nscience fiction, it's definitely wrong...\n\t\t-- Christine Peterson,\n"}]}