{"thread":{"id":"27806","subject":"[PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha","startedAt":"2011-07-13T15:26:44Z","lastAt":"2011-07-13T15:26:53Z","messageCount":10,"participants":["Dmitry Ivankov"],"isPatch":true,"patchVersion":2,"patchTotal":9},"messages":[{"id":"171252","messageId":"1310570813-16648-1-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":null,"subject":"[GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:44Z","receivedAt":"2011-07-13T15:26:44Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"This is a second iteration of remote-svn-alpha series [1].\n\nThe main changes are:\n1) more strict errors checking - be ready for helper exit code being checked[3]\n2) saving \"rN\" marks as a commit->svn revision mapping\n3) supporting incremental imports\n\n\"2)\" could be even nicer if there were a way for git-remote add or git-clone to\nsetup a show-notes config option. Another interesting aspect is notes_ref commits\nauthor (vcs-svn <vcs-svn@local>) and timestamps (\"now\"). Timestamps make notes sha1\ndiverge on different repos/importers, so fetching notes from a peer git repo is not\nfast-forward and a bad thing is they are stored in a private namespace - not likely\nplace for user to perform merges. But as the tests show, one may want to trust peer\nnotes and fetch --force svn updated imported by a peer.\n\nBut anyway, there are a few tests in this series and it looks like the helper is\nreally able to import from svn incrementally and exchange/clone imported data\nbetween different git repositories. Same limitations as earlier: only whole svnroot,\nonly import, needs svnrdump (for file:// svnadmin+svnlook would suffice)\n\nFurther improvements for a just import from svn and maybe clone/fetch svn data from\nanother git peers are:\na) mentioned above \"2)\" problems\nb) adding various configuration options (like svn username and password)\nc) add progress indication, either true percentage of # of imported revisions or\nsome adaptive indicator or any other informative one\nd) maybe issue a checkpoint command from time to time or somehow allow to specify\nhow many revisions to fetch - for huge initial imports\ne) import a subdirectory (of svnroot) history\nf) shallow imports\ng) save all revprops/fileprops whether they have git counterparts or not, ideally\nthese are sufficient to convert the imported history back to svn lossless\nh) get rid of bashisms in this helper or rewrite it in C.\n\nThese do not block proceeding to the next topics:\ni) push commits to svn\nii) track svn branches.\n\n\nThe patch base is svn-fe-pu at git://repo.or.cz/git/jrn.git\nalso a commit 7153183171de77d084a4c24ef19d23d6313ded2a can be used\nas an earlier patch base.\n\nFor the last five commits (those just after the \"add a test\") some \nnew svn-fe options[2] are required.\n\nWhole git-remote-svn-alpha and a test script are cited at the bottom\nof this letter as a quick reference for those who don't track svn-fe-pu.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/176617\n[2] http://thread.gmane.org/gmane.comp.version-control.git/177025\n[3] http://thread.gmane.org/gmane.comp.version-control.git/176002/focus=176019\n\nDmitry Ivankov (9):\n  svn-fe: use svnrdump --quiet in remote-svn-alpha\n  svn-fe: avoid error on no-op imports in remote-svn-alpha\n  svn-fe: allow svnadmin instead of svnrdump in remote-svn-alpha\n  svn-fe: add a test for remote-svn-alpha\n  svn-fe: use svn-fe --no-progress in remote-svn-alpha\n  svn-fe: use proper refspec in remote-svn-alpha\n  svn-fe: write svnrev notes in remote-svn-alpha\n  svn-fe: import incrementally in svn-remote-alpha\n  svn-fe: reuse import-marks in remote-svn-alpha\n\n contrib/svn-fe/git-remote-svn-alpha        |  140 +++++++++++-\n contrib/svn-fe/t/.gitignore                |    3 +\n contrib/svn-fe/t/t9010-remote-svn-alpha.sh |  351 ++++++++++++++++++++++++++++\n 3 files changed, 485 insertions(+), 9 deletions(-)\n create mode 100644 contrib/svn-fe/t/.gitignore\n create mode 100755 contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n\n-- \n1.7.3.4\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nnew file mode 100755\nindex 0000000..8096ef2\n--- /dev/null\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -0,0 +1,177 @@\n+#!/bin/bash\n+set -o pipefail\n+set -e\n+\n+die () {\n+\tprintf >&2 'fatal: %s\\n' \"$*\"\n+\texit 128\n+}\n+\n+usage () {\n+\tprintf >&2 'usage: %s\\n' \"$*\"\n+\texit 129\n+}\n+\n+svnrdump_wrap () {\n+\texec 6<&1 &&\n+\n+\tEX=$(\n+\t\t(svnrdump dump --non-interactive --username=Guest --password= \\\n+\t\t\t--quiet \"$1\" \"$2\" >&6) 2>&1; test $? -ne 0 || echo Success\n+\t) &&\n+\tif test \"z$EX\" != \"zSuccess\"; then\n+\t\tif test \"z$EX\" = \"zLOWER cannot be greater than UPPER.\"; then\n+\t\t\treturn 0\n+\t\tfi\n+\t\techo \"$EX\" >&2\n+\t\treturn 1\n+\tfi\n+}\n+\n+try_svnrdump () {\n+\tcommand -v svnrdump >/dev/null &&\n+\techo svnrdump_wrap ||\n+\ttrue\n+}\n+\n+svnadmin_wrap () {\n+\tpath=${1##file://} &&\n+\ttest \"z$path\" != \"z$1\" &&\n+\tlatest=$(svnlook youngest \"$path\") &&\n+\tfuture=$(( $latest + 1 )) &&\n+\tasked=${2#-r} &&\n+\tasked=${asked%:*} &&\n+\ttest \"$asked\" -eq \"$future\" ||\n+\tsvnadmin dump --incremental --deltas --quiet \"$path\" \"$2\"\n+}\n+\n+try_svnadmin () {\n+\tcommand -v svnadmin >/dev/null &&\n+\tcommand -v svnlook >/dev/null &&\n+\techo svnadmin_wrap ||\n+\ttrue\n+}\n+\n+SVNDUMP=\"\"\n+SVNDUMP=${SVNDUMP:-`try_svnrdump`}\n+SVNDUMP=${SVNDUMP:-`try_svnadmin`}\n+test -n \"$SVNDUMP\" || die \"neither svnrdump nor svnadmin & svnlook was found\"\n+\n+git_dir=\"\"\n+marks_dir=\"info/fast-import/svn-alpha/$repo\"\n+\n+do_gen_marks () {\n+\tnotes=$1\n+\ttest \"z$git_dir\" != \"$z\" || die \"we were not told the gitdir\"\n+\tgit --git-dir=\"$git_dir\" ls-tree -r \"$notes\" |\n+\t{\n+\t\twhile read -r mode type sha path\n+\t\tdo\n+\t\t\tdata=$(git --git-dir=\"$git_dir\" cat-file blob $sha | tail -n 1)\n+\t\t\tdata=${data##r}\n+\t\t\tcommit=$(echo $path | tr -d /)\n+\t\t\techo \":$data $commit\"\n+\t\tdone\n+\t}\n+}\n+\n+prepare_marks () {\n+\tdst=$1 notes=$2 rev=$3\n+\ttest \"z$git_dir\" != \"$z\" || die \"we were not told the gitdir\"\n+\tpath=\"$git_dir/$marks_dir\"\n+\tmkdir -p \"$path\"\n+\tpath=\"$path/marks\"\n+\n+\tif test ! -f \"$path\"; then\n+\t\ttouch \"$path\"\n+\tfi\n+\tif test \"$rev\" = \"-1\"; then\n+\t\treturn 0\n+\tfi\n+\tmark_sha=\"\"\n+\t{\n+\t\twhile read -r m sha\n+\t\tdo\n+\t\t\tif test \"$m\" = \":$rev\"; then\n+\t\t\t\tmark_sha=\"$sha\"\n+\t\t\tfi\n+\t\tdone\n+\t} <\"$path\"\n+\tdst_sha=$( git rev-parse \"$dst\" )\n+\tif test -n \"$mark_sha\"; then\n+\t\ttest \"$mark_sha\" = \"$dst_sha\" || die \"latest mark and note diverge\"\n+\telse\n+\t\tdo_gen_marks \"$notes\" >\"$path\"\n+\tfi\n+}\n+\n+last_imported_rev () {\n+\tdst=$1 notes=$2\n+\ttest \"z$git_dir\" != \"$z\" || die \"we were not told the gitdir\"\n+\tgit --git-dir=\"$git_dir\" show-ref -q --verify $dst || {\n+\t\techo \"-1\"\n+\t\treturn 0\n+\t}\n+\trev=$(git --git-dir=\"$git_dir\" log --show-notes=\"$notes\" -1 --format=%N $dst)\n+\trev=${rev##r}\n+\ttest \"z$rev\" != \"z\" || {\n+\t\tdie \"remote HEAD has no note\"\n+\t}\n+\techo $rev\n+}\n+\n+do_import () {\n+\trevs=$1 url=$2 dst=$3 notes=$4\n+\trev=$(last_imported_rev \"$dst\" \"$notes\")\n+\tstart_rev=$(($rev + 1))\n+\trevs=\"$start_rev:HEAD\"\n+\tprepare_marks $dst $notes $rev\n+\techo \"feature import-marks=$git_dir/$marks_dir/marks\"\n+\techo \"feature export-marks=$git_dir/$marks_dir/marks\"\n+\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --ref=\"$dst\" --notes-ref=\"$notes\" --incremental --no-progress) 3<&0 || die \"FAILURE\"\n+\texec 1>&-\n+}\n+\n+test \"${2+set}\" ||\n+usage 'git remote-svn-alpha <repository> <URL> < commandlist'\n+repo=$1\n+url=$2\n+need_import=\"\"\n+remote_ref=\"refs/heads/master\"\n+private_ref=\"refs/svn-alpha/$repo/SVNHEAD\"\n+remote_notes=\"refs/notes/svnr\"\n+private_notes=\"refs/svn-alpha/$repo/SVNR\"\n+\n+while read -r cmd args\n+do\n+\tcase $cmd in\n+\tcapabilities)\n+\t\techo import\n+\t\techo gitdir\n+\t\techo \"refspec HEAD:$private_ref\"\n+\t\techo \"refspec $remote_ref:$private_ref\"\n+\t\techo \"refspec $remote_notes:$private_notes\"\n+\t\techo\n+\t\t;;\n+\tgitdir)\n+\t\tgit_dir=\"$args\"\n+\t\t;;\n+\tlist)\n+\t\techo \"? HEAD\"\n+\t\techo \"? $remote_ref\"\n+\t\techo\n+\t\t;;\n+\timport)\n+\t\ttest \"$args\" = \"HEAD\" || test \"$args\" = \"$remote_ref\" ||\n+\t\tdie \"remote-svn-alpha: unsupported import ref argument: $args\"\n+\t\tneed_import=\"yes\"\n+\t\t;;\n+\t'')\n+\t\ttest \"$need_import\" = \"yes\" || exit 0\n+\t\tdo_import 0:HEAD \"$url\" \"$private_ref\" \"$private_notes\"\n+\t\tneed_import=\"\"\n+\t\t;;\n+\t*)\n+\t\tdie \"remote-svn-alpha: unsupported command: $cmd $args\"\n+\tesac\n+done\ndiff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\nnew file mode 100755\nindex 0000000..9d9aca0\n--- /dev/null\n+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n@@ -0,0 +1,351 @@\n+#!/bin/sh\n+\n+test_description='check svn-alpha remote helper'\n+\n+PATH=$(pwd)/..:$PATH\n+TEST_DIRECTORY=$(pwd)/../../../t\n+. $TEST_DIRECTORY/test-lib.sh\n+\n+if command -v svnrdump >/dev/null; then\n+\ttest_set_prereq SVNRDUMP\n+fi\n+\n+deinit_git () {\n+\trm -fr .git\n+}\n+\n+reinit_git () {\n+\tdeinit_git &&\n+\tgit init\n+}\n+\n+properties () {\n+\twhile test \"$#\" -ne 0\n+\tdo\n+\t\tproperty=\"$1\" &&\n+\t\tvalue=\"$2\" &&\n+\t\tprintf \"%s\\n\" \"K ${#property}\" &&\n+\t\tprintf \"%s\\n\" \"$property\" &&\n+\t\tprintf \"%s\\n\" \"V ${#value}\" &&\n+\t\tprintf \"%s\\n\" \"$value\" &&\n+\t\tshift 2 ||\n+\t\treturn 1\n+\tdone\n+}\n+\n+text_no_props () {\n+\ttext=\"$1\n+\" &&\n+\tprintf \"%s\\n\" \"Prop-content-length: 10\" &&\n+\tprintf \"%s\\n\" \"Text-content-length: ${#text}\" &&\n+\tprintf \"%s\\n\" \"Content-length: $((${#text} + 10))\" &&\n+\tprintf \"%s\\n\" \"\" \"PROPS-END\" &&\n+\tprintf \"%s\\n\" \"$text\"\n+}\n+\n+dump_to_svnrepo () {\n+\tdump=\"$1\" &&\n+\tpath=\"$2\" &&\n+\tsvnadmin create \"$path\" &&\n+\tsvnadmin load \"$path\" < \"$dump\"\n+}\n+\n+svnurl () {\n+\tprintf \"svn-alpha::file://%s/%s\" \"$(pwd)\" \"$1\"\n+}\n+\n+test_nr_revs () {\n+\tn=$1 &&\n+\trepo=$2 &&\n+\tref=$3 &&\n+\tgit --git-dir=\"$repo\" log --format=oneline \"$ref\" >revs &&\n+\ttest_line_count = \"$n\" revs\n+}\n+\n+test_expect_success 'svnadmin is present' '\n+\tcommand -v svnadmin &&\n+\ttest_set_prereq SVNADMIN\n+'\n+\n+test_expect_success SVNADMIN 'create empty svnrepo' '\n+\techo \"SVN-fs-dump-format-version: 2\" > empty.dump &&\n+\tdump_to_svnrepo empty.dump empty.svn &&\n+\ttest_set_prereq EMPTY_SVN\n+'\n+\n+test_expect_success SVNADMIN 'create tiny svnrepo' '\n+\t{\n+\t\tproperties \\\n+\t\t\tsvn:author author@example.com \\\n+\t\t\tsvn:date \"1999-02-01T00:01:002.000000Z\" \\\n+\t\t\tsvn:log \"add directory with some files in it\" &&\n+\t\techo PROPS-END\n+\t} >props &&\n+\t{\n+\t\tcat <<-EOF &&\n+\t\tSVN-fs-dump-format-version: 3\n+\n+\t\tRevision-number: 1\n+\t\tEOF\n+\t\techo Prop-content-length: $(wc -c <props) &&\n+\t\techo Content-length: $(wc -c <props) &&\n+\t\techo &&\n+\t\tcat props &&\n+\t\tcat <<-\\EOF &&\n+\n+\t\tNode-path: directory\n+\t\tNode-kind: dir\n+\t\tNode-action: add\n+\t\tProp-content-length: 10\n+\t\tContent-length: 10\n+\n+\t\tPROPS-END\n+\t\tNode-path: directory/somefile\n+\t\tNode-kind: file\n+\t\tNode-action: add\n+\t\tEOF\n+\t\ttext_no_props hi\n+\t} >tiny.dump &&\n+\tdump_to_svnrepo tiny.dump tiny.svn &&\n+\ttest_set_prereq TINY_SVN\n+'\n+\n+test_expect_success SVNADMIN 'create small svndump' '\n+\t{\n+\t\tproperties \\\n+\t\t\tsvn:author author@example.com \\\n+\t\t\tsvn:date \"1999-02-01T00:01:002.000000Z\" \\\n+\t\t\tsvn:log \"add directory with some files in it\" &&\n+\t\techo PROPS-END\n+\t} >props &&\n+\t{\n+\t\techo Prop-content-length: $(wc -c <props) &&\n+\t\techo Content-length: $(wc -c <props) &&\n+\t\techo &&\n+\t\tcat props\n+\t} >props_dump &&\n+\tcat >small.dump.r0 <<-EOF &&\n+\tSVN-fs-dump-format-version: 3\n+\tEOF\n+\tfor x in `seq 1 1 10`; do\n+\t\t{\n+\t\t\techo &&\n+\t\t\techo \"Revision-number: $x\" &&\n+\t\t\tcat props_dump &&\n+\n+\t\t\tif test \"$x\" -eq \"1\"; then\n+\t\t\t\tcat <<-\\EOF\n+\n+\t\t\t\tNode-path: directory\n+\t\t\t\tNode-kind: dir\n+\t\t\t\tNode-action: add\n+\t\t\t\tProp-content-length: 10\n+\t\t\t\tContent-length: 10\n+\n+\t\t\t\tPROPS-END\n+\n+\t\t\t\tEOF\n+\t\t\tfi\n+\t\t\techo \"Node-path: directory/somefile$x\" &&\n+\t\t\techo \"Node-kind: file\" &&\n+\t\t\techo \"Node-action: add\" &&\n+\t\t\ttext_no_props hi\n+\t\t} >small.dump.r$x\n+\tdone &&\n+\ttest_set_prereq SMALL_SVNDUMP\n+'\n+\n+test_expect_success SMALL_SVNDUMP 'create small svnrepo' '\n+\trm -rf small.svn* &&\n+\tsvnadmin create small.svn &&\n+\tfor x in `seq 1 1 10`; do\n+\t\tcat small.dump.r0 small.dump.r$x >part.dump &&\n+\t\tsvnadmin load small.svn <part.dump &&\n+\t\tcp -r small.svn small.svn.r0-$x\n+\tdone &&\n+\ttest_set_prereq SMALL_SVN\n+'\n+\n+test_expect_failure EMPTY_SVN 'fetch empty' '\n+\treinit_git &&\n+\turl=$(svnurl empty.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\tgit fetch svn\n+'\n+\n+test_expect_failure TINY_SVN 'clone tiny' '\n+\tdeinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit clone \"$url\" tiny1.git &&\n+\ttest_nr_revs 1 tiny1.git refs/remotes/origin/master\n+'\n+\n+test_expect_success TINY_SVN 'clone --mirror tiny' '\n+\tdeinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit clone --mirror \"$url\" tiny2.git &&\n+\ttest_nr_revs 1 tiny2.git refs/heads/master\n+'\n+\n+test_expect_success TINY_SVN 'clone --bare tiny' '\n+\tdeinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit clone --bare \"$url\" tiny3.git &&\n+\ttest_nr_revs 1 tiny3.git refs/heads/master\n+'\n+\n+test_expect_success TINY_SVN 'clone -b master tiny' '\n+\tdeinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit clone -b master \"$url\" tiny4.git &&\n+\ttest_nr_revs 1 tiny4.git/.git refs/heads/master\n+'\n+\n+test_expect_success SMALL_SVN 'clone -b master small' '\n+\tdeinit_git &&\n+\turl=$(svnurl small.svn) &&\n+\tgit clone -b master \"$url\" small.git &&\n+\ttest_nr_revs 10 small.git/.git refs/heads/master\n+'\n+\n+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path' '\n+\tdeinit_git &&\n+\turl=$(svnurl small.svn)/directory &&\n+\tgit clone -b master \"$url\" small_dir.git &&\n+\ttest_nr_revs 10 small_dir.git/.git refs/heads/master\n+'\n+\n+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path/file' '\n+\tdeinit_git &&\n+\turl=$(svnurl small.svn)/directory/somefile3 &&\n+\tgit clone -b master \"$url\" small_dir_file.git &&\n+\ttest_nr_revs 10 small_dir_file.git/.git refs/heads/master\n+'\n+\n+test_expect_success SMALL_SVN 'fetch each rev of SMALL separately' '\n+\treinit_git &&\n+\turl=$(svnurl small.svn) &&\n+\n+\tfor x in `seq 1 1 10`; do\n+\t\tgit remote add svn_$x \"$url.r0-$x\"\n+\tdone &&\n+\tgit remote update &&\n+\n+\tgit remote add svn \"$url\" &&\n+\tgit fetch svn &&\n+\n+\tgit rev-parse -s remotes/svn_7/master~5 >ref7_2 &&\n+\tgit rev-parse -s remotes/svn/master~8 >ref_2 &&\n+\ttest_cmp ref7_2 ref_2\n+'\n+\n+test_expect_success SMALL_SVN 'fetch updates from SMALL' '\n+\treinit_git &&\n+\turl=$(svnurl link.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\n+\tln -sfn small.svn.r0-5 link.svn &&\n+\tgit fetch svn &&\n+\ttest_nr_revs 5 .git refs/remotes/svn/master &&\n+\n+\tln -sfn small.svn.r0-10 link.svn &&\n+\tgit fetch svn &&\n+\ttest_nr_revs 10 .git refs/remotes/svn/master &&\n+\n+\tgit fetch svn &&\n+\ttest_nr_revs 10 .git refs/remotes/svn/master\n+'\n+\n+test_expect_success TINY_SVN 'fetch TINY does not write to refs/heads/master' '\n+\treinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\tgit fetch svn &&\n+\tgit show-ref --verify refs/remotes/svn/master &&\n+\ttest_must_fail git show-ref --verify refs/heads/master\n+'\n+\n+test_expect_success SMALL_SVN 'fetch SMALL writes revnum notes' '\n+\treinit_git &&\n+\turl=$(svnurl small.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\tgit fetch svn &&\n+\tgit log --show-notes=refs/svn-alpha/svn/SVNR --format=%N -1 refs/remotes/svn/master^ >actual.note &&\n+\techo r9 >expect.note &&\n+\techo >>expect.note &&\n+\ttest_cmp expect.note actual.note\n+'\n+\n+test_expect_success SMALL_SVN 'marks from notes regeneration' '\n+\treinit_git &&\n+\turl=$(svnurl link.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\n+\tln -sfn small.svn.r0-5 link.svn &&\n+\tgit fetch svn &&\n+\ttest_nr_revs 5 .git refs/remotes/svn/master &&\n+\n+\trm -rf .git/info/fast-import/svn-alpha &&\n+\n+\tln -sfn small.svn.r0-10 link.svn &&\n+\tgit fetch svn &&\n+\ttest_nr_revs 10 .git refs/remotes/svn/master\n+'\n+\n+test_expect_success SMALL_SVN 'clone to bootstrap' '\n+\tdeinit_git &&\n+\turl=$(svnurl link.svn) &&\n+\tln -sfn small.svn.r0-5 link.svn &&\n+\tgit clone -b master \"$url\" master.git &&\n+\t{\n+\t\tcd master.git &&\n+\t\tgit branch pub_head refs/svn-alpha/origin/SVNHEAD &&\n+\t\tgit branch pub_notes refs/svn-alpha/origin/SVNR &&\n+\t\tcd ..\n+\t} &&\n+\tln -sfn small.svn.r0-8 link.svn &&\n+\tgit clone master.git slave.git &&\n+\t{\n+\t\tcd slave.git &&\n+\t\tgit remote add svn \"$url\" &&\n+\t\tgit update-ref refs/svn-alpha/svn/SVNHEAD refs/remotes/origin/pub_head &&\n+\t\tgit update-ref refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&\n+\t\tgit fetch svn &&\n+\t\tgit merge-base refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&\n+\t\tcd ..\n+\t} &&\n+\ttest_nr_revs 8 slave.git/.git refs/remotes/svn/master\n+'\n+\n+test_expect_success SMALL_SVN 'clone and exchange' '\n+\tdeinit_git &&\n+\turl=$(svnurl link.svn) &&\n+\tln -sfn small.svn.r0-1 link.svn &&\n+\tgit clone -b master \"$url\" A.git &&\n+\tgit clone -b master \"$url\" B.git &&\n+\ttest_nr_revs 1 A.git/.git origin/master &&\n+\ttest_nr_revs 1 B.git/.git origin/master &&\n+\n+\tgit --git-dir=A.git/.git remote add B B.git &&\n+\tgit --git-dir=B.git/.git remote add A A.git &&\n+\trefspecs=\"+refs/svn-alpha/origin/SVNR:refs/svn-alpha/origin/SVNR\"\n+\trefspecs=\"$refspecs refs/svn-alpha/origin/SVNHEAD:refs/svn-alpha/origin/SVNHEAD\"\n+\n+\tln -sfn small.svn.r0-2 link.svn &&\n+\tgit --git-dir=A.git/.git fetch origin &&\n+\n+\tgit --git-dir=B.git/.git fetch -f A $refspecs &&\n+\n+\tln -sfn small.svn.r0-3 link.svn &&\n+\tgit --git-dir=B.git/.git fetch origin &&\n+\n+\tgit --git-dir=A.git/.git fetch B $refspecs &&\n+\n+\tgit --git-dir=A.git/.git fetch origin &&\n+\tgit --git-dir=B.git/.git fetch origin &&\n+\n+\ttest_nr_revs 3 A.git/.git origin/master &&\n+\ttest_nr_revs 3 B.git/.git origin/master\n+'\n+\n+test_done\n"},{"id":"171251","messageId":"1310570813-16648-2-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:45Z","receivedAt":"2011-07-13T15:26:45Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"svnrdump by default shows \"Dumped revision #n\" lines on stderr.\nsvn-fe does it too on each imported revision, so pass --quiet to\nsvnrdump. Once process indication is really needed it will be\ncustom formatted in either svn-fe or remote-svn-alpha.\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/git-remote-svn-alpha |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nindex b2cca9f..61c9b07 100755\n--- a/contrib/svn-fe/git-remote-svn-alpha\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -15,7 +15,7 @@ usage () {\n do_import () {\n \trevs=$1 url=$2\n \t(svnrdump dump --non-interactive --username=Guest --password= \\\n-\t\t-r\"$revs\" \"$url\" | svn-fe) 3<&0 || die \"FAILURE\"\n+\t\t-r\"$revs\" \"$url\" --quiet | svn-fe) 3<&0 || die \"FAILURE\"\n \texec 1>&-\n }\n \n-- \n1.7.3.4\n"},{"id":"171253","messageId":"1310570813-16648-3-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 2/9] svn-fe: avoid error on no-op imports in remote-svn-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:46Z","receivedAt":"2011-07-13T15:26:46Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"Currently helpers' exit codes are not checked by transport_helper.\nBut a proper helper should check for all possible internal errors\nanyway. svnrdump dump --incremental considers -rX:HEAD range as bad\nif X is greater than the actual HEAD revision. And there is no option\nto change this behavior.\n\nOne way to address this issue is to fire one more svn command to get\nthe HEAD value, but it is one more connection, one more tool (ok,\nwe can svnrdump HEAD but it looks slow) and possibly one more\npassword prompt, and maybe even a race condition (if we talk to a\nsvn servers farm frontend for example).\n\nAnother one is to patch svnrdump to report the revision, because\ninternally it asks svn for it anyway before doing a dump. Longer\nterm it looks nice, moreover it can be used to display percentage\nprogress.\n\nAdd a wrapper around svnrdump that captures stderr and exit code.\nIf stderr matches a hardcoded \"LOWER cannot be greater than UPPER.\"\nand the exit code is non-zero, don't produce any dump and emulate\nexit 0, otherwise the wrapper is transparent. The only side effect\nis dup2-ing stdout to fd=6, ugly but fine as only standard\ndescriptors are used currently.\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/git-remote-svn-alpha |   19 +++++++++++++++++--\n 1 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nindex 61c9b07..2eac7d4 100755\n--- a/contrib/svn-fe/git-remote-svn-alpha\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -12,10 +12,25 @@ usage () {\n \texit 129\n }\n \n+svnrdump_wrap () {\n+\texec 6<&1 &&\n+\n+\tEX=$(\n+\t\t(svnrdump dump --non-interactive --username=Guest --password= \\\n+\t\t\t--quiet \"$1\" \"$2\" >&6) 2>&1; test $? -ne 0 || echo Success\n+\t) &&\n+\tif test \"z$EX\" != \"zSuccess\"; then\n+\t\tif test \"z$EX\" = \"zLOWER cannot be greater than UPPER.\"; then\n+\t\t\treturn 0\n+\t\tfi\n+\t\techo \"$EX\" >&2\n+\t\treturn 1\n+\tfi\n+}\n+\n do_import () {\n \trevs=$1 url=$2\n-\t(svnrdump dump --non-interactive --username=Guest --password= \\\n-\t\t-r\"$revs\" \"$url\" --quiet | svn-fe) 3<&0 || die \"FAILURE\"\n+\t(svnrdump_wrap \"$url\" -r\"$revs\" | svn-fe) 3<&0 || die \"FAILURE\"\n \texec 1>&-\n }\n \n-- \n1.7.3.4\n"},{"id":"171254","messageId":"1310570813-16648-4-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 3/9] svn-fe: allow svnadmin instead of svnrdump in remote-svn-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:47Z","receivedAt":"2011-07-13T15:26:47Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"svnrdump dump and svnadmin dump share a dump format. Their dump\nstreams representation may be different but they should have the\nsame effect on a dump consumer. One more difference is svnrdump being\nable to dump via any svn remote protocol while svnadmin needs access\nto the repository filesystem and also can't produce subdirectory dumps.\nBut svnrdump is a newer tool and may be unavailable on some systems.\n\nTry to use svnadmin dump for file:// repository urls if there is no\nsvnrdump in the PATH. First of all this is to be used in tests, where\nthe repository is indeed local most of the time. Also require svnlook\nutility as a lightweight option to get the youngest repository revision.\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/git-remote-svn-alpha |   31 ++++++++++++++++++++++++++++++-\n 1 files changed, 30 insertions(+), 1 deletions(-)\n\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nindex 2eac7d4..ca9d431 100755\n--- a/contrib/svn-fe/git-remote-svn-alpha\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -28,9 +28,38 @@ svnrdump_wrap () {\n \tfi\n }\n \n+try_svnrdump () {\n+\tcommand -v svnrdump >/dev/null &&\n+\techo svnrdump_wrap ||\n+\ttrue\n+}\n+\n+svnadmin_wrap () {\n+\tpath=${1##file://} &&\n+\ttest \"z$path\" != \"z$1\" &&\n+\tlatest=$(svnlook youngest \"$path\") &&\n+\tfuture=$(( $latest + 1 )) &&\n+\tasked=${2#-r} &&\n+\tasked=${asked%:*} &&\n+\ttest \"$asked\" -eq \"$future\" ||\n+\tsvnadmin dump --incremental --deltas --quiet \"$path\" \"$2\"\n+}\n+\n+try_svnadmin () {\n+\tcommand -v svnadmin >/dev/null &&\n+\tcommand -v svnlook >/dev/null &&\n+\techo svnadmin_wrap ||\n+\ttrue\n+}\n+\n+SVNDUMP=\"\"\n+SVNDUMP=${SVNDUMP:-`try_svnrdump`}\n+SVNDUMP=${SVNDUMP:-`try_svnadmin`}\n+test -n \"$SVNDUMP\" || die \"neither svnrdump nor svnadmin & svnlook was found\"\n+\n do_import () {\n \trevs=$1 url=$2\n-\t(svnrdump_wrap \"$url\" -r\"$revs\" | svn-fe) 3<&0 || die \"FAILURE\"\n+\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe) 3<&0 || die \"FAILURE\"\n \texec 1>&-\n }\n \n-- \n1.7.3.4\n"},{"id":"171260","messageId":"1310570813-16648-5-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 4/9] svn-fe: add a test for remote-svn-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:48Z","receivedAt":"2011-07-13T15:26:48Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"The test creates a few svn repositories from fixed dumps using svnadmin.\nCurrently it checks for crashes on first time import mostly.\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/t/.gitignore                |    3 +\n contrib/svn-fe/t/t9010-remote-svn-alpha.sh |  259 ++++++++++++++++++++++++++++\n 2 files changed, 262 insertions(+), 0 deletions(-)\n create mode 100644 contrib/svn-fe/t/.gitignore\n create mode 100755 contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n\ndiff --git a/contrib/svn-fe/t/.gitignore b/contrib/svn-fe/t/.gitignore\nnew file mode 100644\nindex 0000000..4e731dc\n--- /dev/null\n+++ b/contrib/svn-fe/t/.gitignore\n@@ -0,0 +1,3 @@\n+/trash directory*\n+/test-results\n+/.prove\ndiff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\nnew file mode 100755\nindex 0000000..92bc4b2\n--- /dev/null\n+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n@@ -0,0 +1,259 @@\n+#!/bin/sh\n+\n+test_description='check svn-alpha remote helper'\n+\n+PATH=$(pwd)/..:$PATH\n+TEST_DIRECTORY=$(pwd)/../../../t\n+. $TEST_DIRECTORY/test-lib.sh\n+\n+if command -v svnrdump >/dev/null; then\n+\ttest_set_prereq SVNRDUMP\n+fi\n+\n+deinit_git () {\n+\trm -fr .git\n+}\n+\n+reinit_git () {\n+\tdeinit_git &&\n+\tgit init\n+}\n+\n+properties () {\n+\twhile test \"$#\" -ne 0\n+\tdo\n+\t\tproperty=\"$1\" &&\n+\t\tvalue=\"$2\" &&\n+\t\tprintf \"%s\\n\" \"K ${#property}\" &&\n+\t\tprintf \"%s\\n\" \"$property\" &&\n+\t\tprintf \"%s\\n\" \"V ${#value}\" &&\n+\t\tprintf \"%s\\n\" \"$value\" &&\n+\t\tshift 2 ||\n+\t\treturn 1\n+\tdone\n+}\n+\n+text_no_props () {\n+\ttext=\"$1\n+\" &&\n+\tprintf \"%s\\n\" \"Prop-content-length: 10\" &&\n+\tprintf \"%s\\n\" \"Text-content-length: ${#text}\" &&\n+\tprintf \"%s\\n\" \"Content-length: $((${#text} + 10))\" &&\n+\tprintf \"%s\\n\" \"\" \"PROPS-END\" &&\n+\tprintf \"%s\\n\" \"$text\"\n+}\n+\n+dump_to_svnrepo () {\n+\tdump=\"$1\" &&\n+\tpath=\"$2\" &&\n+\tsvnadmin create \"$path\" &&\n+\tsvnadmin load \"$path\" < \"$dump\"\n+}\n+\n+svnurl () {\n+\tprintf \"svn-alpha::file://%s/%s\" \"$(pwd)\" \"$1\"\n+}\n+\n+test_nr_revs () {\n+\tn=$1 &&\n+\trepo=$2 &&\n+\tref=$3 &&\n+\tgit --git-dir=\"$repo\" log --format=oneline \"$ref\" >revs &&\n+\ttest_line_count = \"$n\" revs\n+}\n+\n+test_expect_success 'svnadmin is present' '\n+\tcommand -v svnadmin &&\n+\ttest_set_prereq SVNADMIN\n+'\n+\n+test_expect_success SVNADMIN 'create empty svnrepo' '\n+\techo \"SVN-fs-dump-format-version: 2\" > empty.dump &&\n+\tdump_to_svnrepo empty.dump empty.svn &&\n+\ttest_set_prereq EMPTY_SVN\n+'\n+\n+test_expect_success SVNADMIN 'create tiny svnrepo' '\n+\t{\n+\t\tproperties \\\n+\t\t\tsvn:author author@example.com \\\n+\t\t\tsvn:date \"1999-02-01T00:01:002.000000Z\" \\\n+\t\t\tsvn:log \"add directory with some files in it\" &&\n+\t\techo PROPS-END\n+\t} >props &&\n+\t{\n+\t\tcat <<-EOF &&\n+\t\tSVN-fs-dump-format-version: 3\n+\n+\t\tRevision-number: 1\n+\t\tEOF\n+\t\techo Prop-content-length: $(wc -c <props) &&\n+\t\techo Content-length: $(wc -c <props) &&\n+\t\techo &&\n+\t\tcat props &&\n+\t\tcat <<-\\EOF &&\n+\n+\t\tNode-path: directory\n+\t\tNode-kind: dir\n+\t\tNode-action: add\n+\t\tProp-content-length: 10\n+\t\tContent-length: 10\n+\n+\t\tPROPS-END\n+\t\tNode-path: directory/somefile\n+\t\tNode-kind: file\n+\t\tNode-action: add\n+\t\tEOF\n+\t\ttext_no_props hi\n+\t} >tiny.dump &&\n+\tdump_to_svnrepo tiny.dump tiny.svn &&\n+\ttest_set_prereq TINY_SVN\n+'\n+\n+test_expect_success SVNADMIN 'create small svndump' '\n+\t{\n+\t\tproperties \\\n+\t\t\tsvn:author author@example.com \\\n+\t\t\tsvn:date \"1999-02-01T00:01:002.000000Z\" \\\n+\t\t\tsvn:log \"add directory with some files in it\" &&\n+\t\techo PROPS-END\n+\t} >props &&\n+\t{\n+\t\techo Prop-content-length: $(wc -c <props) &&\n+\t\techo Content-length: $(wc -c <props) &&\n+\t\techo &&\n+\t\tcat props\n+\t} >props_dump &&\n+\tcat >small.dump.r0 <<-EOF &&\n+\tSVN-fs-dump-format-version: 3\n+\tEOF\n+\tfor x in `seq 1 1 10`; do\n+\t\t{\n+\t\t\techo &&\n+\t\t\techo \"Revision-number: $x\" &&\n+\t\t\tcat props_dump &&\n+\n+\t\t\tif test \"$x\" -eq \"1\"; then\n+\t\t\t\tcat <<-\\EOF\n+\n+\t\t\t\tNode-path: directory\n+\t\t\t\tNode-kind: dir\n+\t\t\t\tNode-action: add\n+\t\t\t\tProp-content-length: 10\n+\t\t\t\tContent-length: 10\n+\n+\t\t\t\tPROPS-END\n+\n+\t\t\t\tEOF\n+\t\t\tfi\n+\t\t\techo \"Node-path: directory/somefile$x\" &&\n+\t\t\techo \"Node-kind: file\" &&\n+\t\t\techo \"Node-action: add\" &&\n+\t\t\ttext_no_props hi\n+\t\t} >small.dump.r$x\n+\tdone &&\n+\ttest_set_prereq SMALL_SVNDUMP\n+'\n+\n+test_expect_success SMALL_SVNDUMP 'create small svnrepo' '\n+\trm -rf small.svn* &&\n+\tsvnadmin create small.svn &&\n+\tfor x in `seq 1 1 10`; do\n+\t\tcat small.dump.r0 small.dump.r$x >part.dump &&\n+\t\tsvnadmin load small.svn <part.dump &&\n+\t\tcp -r small.svn small.svn.r0-$x\n+\tdone &&\n+\ttest_set_prereq SMALL_SVN\n+'\n+\n+test_expect_failure EMPTY_SVN 'fetch empty' '\n+\treinit_git &&\n+\turl=$(svnurl empty.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\tgit fetch svn\n+'\n+\n+test_expect_failure TINY_SVN 'clone tiny' '\n+\tdeinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit clone \"$url\" tiny1.git &&\n+\ttest_nr_revs 1 tiny1.git refs/remotes/origin/master\n+'\n+\n+test_expect_success TINY_SVN 'clone --mirror tiny' '\n+\tdeinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit clone --mirror \"$url\" tiny2.git &&\n+\ttest_nr_revs 1 tiny2.git refs/heads/master\n+'\n+\n+test_expect_success TINY_SVN 'clone --bare tiny' '\n+\tdeinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit clone --bare \"$url\" tiny3.git &&\n+\ttest_nr_revs 1 tiny3.git refs/heads/master\n+'\n+\n+test_expect_success TINY_SVN 'clone -b master tiny' '\n+\tdeinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit clone -b master \"$url\" tiny4.git &&\n+\ttest_nr_revs 1 tiny4.git/.git refs/heads/master\n+'\n+\n+test_expect_success SMALL_SVN 'clone -b master small' '\n+\tdeinit_git &&\n+\turl=$(svnurl small.svn) &&\n+\tgit clone -b master \"$url\" small.git &&\n+\ttest_nr_revs 10 small.git/.git refs/heads/master\n+'\n+\n+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path' '\n+\tdeinit_git &&\n+\turl=$(svnurl small.svn)/directory &&\n+\tgit clone -b master \"$url\" small_dir.git &&\n+\ttest_nr_revs 10 small_dir.git/.git refs/heads/master\n+'\n+\n+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path/file' '\n+\tdeinit_git &&\n+\turl=$(svnurl small.svn)/directory/somefile3 &&\n+\tgit clone -b master \"$url\" small_dir_file.git &&\n+\ttest_nr_revs 10 small_dir_file.git/.git refs/heads/master\n+'\n+\n+test_expect_success SMALL_SVN 'fetch each rev of SMALL separately' '\n+\treinit_git &&\n+\turl=$(svnurl small.svn) &&\n+\n+\tfor x in `seq 1 1 10`; do\n+\t\tgit remote add svn_$x \"$url.r0-$x\"\n+\tdone &&\n+\tgit remote update &&\n+\n+\tgit remote add svn \"$url\" &&\n+\tgit fetch svn &&\n+\n+\tgit rev-parse -s remotes/svn_7/master~5 >ref7_2 &&\n+\tgit rev-parse -s remotes/svn/master~8 >ref_2 &&\n+\ttest_cmp ref7_2 ref_2\n+'\n+\n+test_expect_success SMALL_SVN 'fetch updates from SMALL' '\n+\treinit_git &&\n+\turl=$(svnurl link.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\n+\tln -sfn small.svn.r0-5 link.svn &&\n+\tgit fetch svn &&\n+\ttest_nr_revs 5 .git refs/remotes/svn/master &&\n+\n+\tln -sfn small.svn.r0-10 link.svn &&\n+\tgit fetch svn &&\n+\ttest_nr_revs 10 .git refs/remotes/svn/master &&\n+\n+\tgit fetch svn &&\n+\ttest_nr_revs 10 .git refs/remotes/svn/master\n+'\n+\n+test_done\n-- \n1.7.3.4\n"},{"id":"171256","messageId":"1310570813-16648-6-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 5/9] svn-fe: use svn-fe --no-progress in remote-svn-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:49Z","receivedAt":"2011-07-13T15:26:49Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"svn-fe by default  produces a progress line for each imported revision.\nFor large repos it stresses the terminal and a user, possibly scrolls\naway error messages.\n\nJust disable progress lines for now.\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/git-remote-svn-alpha |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nindex ca9d431..ed8da7b 100755\n--- a/contrib/svn-fe/git-remote-svn-alpha\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -59,7 +59,7 @@ test -n \"$SVNDUMP\" || die \"neither svnrdump nor svnadmin & svnlook was found\"\n \n do_import () {\n \trevs=$1 url=$2\n-\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe) 3<&0 || die \"FAILURE\"\n+\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --no-progress) 3<&0 || die \"FAILURE\"\n \texec 1>&-\n }\n \n-- \n1.7.3.4\n"},{"id":"171258","messageId":"1310570813-16648-7-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 6/9] svn-fe: use proper refspec in remote-svn-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:50Z","receivedAt":"2011-07-13T15:26:50Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"Create a per repository name private refs namespace as suggested by\nthe remote helpers documentation. This allows to have several svn\nremotes at the same time. And also this way we don't overwrite the\nlocal branch master.\n\nThe refpec is:\nHEAD:refs/svn-alpha/$reponame/SVNHEAD\nrefs/heads/master:refs/svn-alpha/$reponame/SVNHEAD\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/git-remote-svn-alpha        |   18 ++++++++++--------\n contrib/svn-fe/t/t9010-remote-svn-alpha.sh |    9 +++++++++\n 2 files changed, 19 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nindex ed8da7b..b410302 100755\n--- a/contrib/svn-fe/git-remote-svn-alpha\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -58,8 +58,8 @@ SVNDUMP=${SVNDUMP:-`try_svnadmin`}\n test -n \"$SVNDUMP\" || die \"neither svnrdump nor svnadmin & svnlook was found\"\n \n do_import () {\n-\trevs=$1 url=$2\n-\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --no-progress) 3<&0 || die \"FAILURE\"\n+\trevs=$1 url=$2 dst=$3\n+\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --ref=\"$dst\" --no-progress) 3<&0 || die \"FAILURE\"\n \texec 1>&-\n }\n \n@@ -68,29 +68,31 @@ usage 'git remote-svn-alpha <repository> <URL> < commandlist'\n repo=$1\n url=$2\n need_import=\"\"\n+remote_ref=\"refs/heads/master\"\n+private_ref=\"refs/svn-alpha/$repo/SVNHEAD\"\n \n while read -r cmd args\n do\n \tcase $cmd in\n \tcapabilities)\n \t\techo import\n-\t\techo \"refspec HEAD:refs/heads/master\"\n-\t\techo \"refspec refs/heads/master:refs/heads/master\"\n+\t\techo \"refspec HEAD:$private_ref\"\n+\t\techo \"refspec $remote_ref:$private_ref\"\n \t\techo\n \t\t;;\n \tlist)\n-\t\techo '? HEAD'\n-\t\techo '? refs/heads/master'\n+\t\techo \"? HEAD\"\n+\t\techo \"? $remote_ref\"\n \t\techo\n \t\t;;\n \timport)\n-\t\ttest \"$args\" = \"HEAD\" || test \"$args\" = \"refs/heads/master\" ||\n+\t\ttest \"$args\" = \"HEAD\" || test \"$args\" = \"$remote_ref\" ||\n \t\tdie \"remote-svn-alpha: unsupported import ref argument: $args\"\n \t\tneed_import=\"yes\"\n \t\t;;\n \t'')\n \t\ttest \"$need_import\" = \"yes\" || exit 0\n-\t\tdo_import 0:HEAD \"$url\"\n+\t\tdo_import 0:HEAD \"$url\" \"$private_ref\"\n \t\tneed_import=\"\"\n \t\t;;\n \t*)\ndiff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\nindex 92bc4b2..b0f41cb 100755\n--- a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n@@ -256,4 +256,13 @@ test_expect_success SMALL_SVN 'fetch updates from SMALL' '\n \ttest_nr_revs 10 .git refs/remotes/svn/master\n '\n \n+test_expect_success TINY_SVN 'fetch TINY does not write to refs/heads/master' '\n+\treinit_git &&\n+\turl=$(svnurl tiny.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\tgit fetch svn &&\n+\tgit show-ref --verify refs/remotes/svn/master &&\n+\ttest_must_fail git show-ref --verify refs/heads/master\n+'\n+\n test_done\n-- \n1.7.3.4\n"},{"id":"171255","messageId":"1310570813-16648-8-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 7/9] svn-fe: write svnrev notes in remote-svn-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:51Z","receivedAt":"2011-07-13T15:26:51Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"For each imported svn commit write \"rN\" note in a private namespace.\nThese can be viewed with somewhat ugly\n$ git log --show-notes=refs/svn-alpha/<reponame>/SVNR\nBut also these notes can be used to import svn history incrementally.\nAdd a simple test for these notes.\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/git-remote-svn-alpha        |    9 ++++++---\n contrib/svn-fe/t/t9010-remote-svn-alpha.sh |   11 +++++++++++\n 2 files changed, 17 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nindex b410302..c2af393 100755\n--- a/contrib/svn-fe/git-remote-svn-alpha\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -58,8 +58,8 @@ SVNDUMP=${SVNDUMP:-`try_svnadmin`}\n test -n \"$SVNDUMP\" || die \"neither svnrdump nor svnadmin & svnlook was found\"\n \n do_import () {\n-\trevs=$1 url=$2 dst=$3\n-\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --ref=\"$dst\" --no-progress) 3<&0 || die \"FAILURE\"\n+\trevs=$1 url=$2 dst=$3 notes=$4\n+\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --ref=\"$dst\" --notes_ref=\"$notes\" --no-progress) 3<&0 || die \"FAILURE\"\n \texec 1>&-\n }\n \n@@ -70,6 +70,8 @@ url=$2\n need_import=\"\"\n remote_ref=\"refs/heads/master\"\n private_ref=\"refs/svn-alpha/$repo/SVNHEAD\"\n+remote_notes=\"refs/notes/svnr\"\n+private_notes=\"refs/svn-alpha/$repo/SVNR\"\n \n while read -r cmd args\n do\n@@ -78,6 +80,7 @@ do\n \t\techo import\n \t\techo \"refspec HEAD:$private_ref\"\n \t\techo \"refspec $remote_ref:$private_ref\"\n+\t\techo \"refspec $remote_notes:$private_notes\"\n \t\techo\n \t\t;;\n \tlist)\n@@ -92,7 +95,7 @@ do\n \t\t;;\n \t'')\n \t\ttest \"$need_import\" = \"yes\" || exit 0\n-\t\tdo_import 0:HEAD \"$url\" \"$private_ref\"\n+\t\tdo_import 0:HEAD \"$url\" \"$private_ref\" \"$private_notes\"\n \t\tneed_import=\"\"\n \t\t;;\n \t*)\ndiff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\nindex b0f41cb..5102428 100755\n--- a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n@@ -265,4 +265,15 @@ test_expect_success TINY_SVN 'fetch TINY does not write to refs/heads/master' '\n \ttest_must_fail git show-ref --verify refs/heads/master\n '\n \n+test_expect_success SMALL_SVN 'fetch SMALL writes revnum notes' '\n+\treinit_git &&\n+\turl=$(svnurl small.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\tgit fetch svn &&\n+\tgit log --show-notes=refs/svn-alpha/svn/SVNR --format=%N -1 refs/remotes/svn/master^ >actual.note &&\n+\techo r9 >expect.note &&\n+\techo >>expect.note &&\n+\ttest_cmp expect.note actual.note\n+'\n+\n test_done\n-- \n1.7.3.4\n"},{"id":"171257","messageId":"1310570813-16648-9-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 8/9] svn-fe: import incrementally in svn-remote-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:52Z","receivedAt":"2011-07-13T15:26:52Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"svn-remote-alpha already maintains commits tree imported from svn\nand a mapping to the original svn revision numbers as notes content.\nsvn-fe --incremental requires import marks from previous revisions.\n\nJust generate import marks from notes on each import. This is cheap\non small repositories, fine for a few tests.\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/git-remote-svn-alpha        |   57 +++++++++++++++++++++-\n contrib/svn-fe/t/t9010-remote-svn-alpha.sh |   72 ++++++++++++++++++++++++++++\n 2 files changed, 128 insertions(+), 1 deletions(-)\n\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nindex c2af393..f4e737d 100755\n--- a/contrib/svn-fe/git-remote-svn-alpha\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -57,9 +57,60 @@ SVNDUMP=${SVNDUMP:-`try_svnrdump`}\n SVNDUMP=${SVNDUMP:-`try_svnadmin`}\n test -n \"$SVNDUMP\" || die \"neither svnrdump nor svnadmin & svnlook was found\"\n \n+git_dir=\"\"\n+marks_dir=\"info/fast-import/svn-alpha/$repo\"\n+\n+do_gen_marks () {\n+\tnotes=$1\n+\ttest \"z$git_dir\" != \"$z\" || die \"we were not told the gitdir\"\n+\tgit --git-dir=\"$git_dir\" ls-tree -r \"$notes\" |\n+\t{\n+\t\twhile read -r mode type sha path\n+\t\tdo\n+\t\t\tdata=$(git --git-dir=\"$git_dir\" cat-file blob $sha | tail -n 1)\n+\t\t\tdata=${data##r}\n+\t\t\tcommit=$(echo $path | tr -d /)\n+\t\t\techo \":$data $commit\"\n+\t\tdone\n+\t}\n+}\n+\n+prepare_marks () {\n+\tdst=$1 notes=$2 rev=$3\n+\ttest \"z$git_dir\" != \"$z\" || die \"we were not told the gitdir\"\n+\tmkdir -p \"$git_dir/$marks_dir\"\n+\tif test ! -f \"$git_dir/$marks_dir/marks\"; then\n+\t\ttouch \"$git_dir/$marks_dir/marks\"\n+\tfi\n+\tif test \"$rev\" = \"-1\"; then\n+\t\treturn 0\n+\tfi\n+\tdo_gen_marks \"$notes\" >\"$git_dir/$marks_dir/marks\"\n+}\n+\n+last_imported_rev () {\n+\tdst=$1 notes=$2\n+\ttest \"z$git_dir\" != \"$z\" || die \"we were not told the gitdir\"\n+\tgit --git-dir=\"$git_dir\" show-ref -q --verify $dst || {\n+\t\techo \"-1\"\n+\t\treturn 0\n+\t}\n+\trev=$(git --git-dir=\"$git_dir\" log --show-notes=\"$notes\" -1 --format=%N $dst)\n+\trev=${rev##r}\n+\ttest \"z$rev\" != \"z\" || {\n+\t\tdie \"remote HEAD has no note\"\n+\t}\n+\techo $rev\n+}\n+\n do_import () {\n \trevs=$1 url=$2 dst=$3 notes=$4\n-\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --ref=\"$dst\" --notes_ref=\"$notes\" --no-progress) 3<&0 || die \"FAILURE\"\n+\trev=$(last_imported_rev \"$dst\" \"$notes\")\n+\tstart_rev=$(($rev + 1))\n+\trevs=\"$start_rev:HEAD\"\n+\tprepare_marks $dst $notes $rev\n+\techo \"feature import-marks=$git_dir/$marks_dir/marks\"\n+\t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --ref=\"$dst\" --notes-ref=\"$notes\" --incremental --no-progress) 3<&0 || die \"FAILURE\"\n \texec 1>&-\n }\n \n@@ -78,11 +129,15 @@ do\n \tcase $cmd in\n \tcapabilities)\n \t\techo import\n+\t\techo gitdir\n \t\techo \"refspec HEAD:$private_ref\"\n \t\techo \"refspec $remote_ref:$private_ref\"\n \t\techo \"refspec $remote_notes:$private_notes\"\n \t\techo\n \t\t;;\n+\tgitdir)\n+\t\tgit_dir=\"$args\"\n+\t\t;;\n \tlist)\n \t\techo \"? HEAD\"\n \t\techo \"? $remote_ref\"\ndiff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\nindex 5102428..9d9aca0 100755\n--- a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh\n@@ -276,4 +276,76 @@ test_expect_success SMALL_SVN 'fetch SMALL writes revnum notes' '\n \ttest_cmp expect.note actual.note\n '\n \n+test_expect_success SMALL_SVN 'marks from notes regeneration' '\n+\treinit_git &&\n+\turl=$(svnurl link.svn) &&\n+\tgit remote add svn \"$url\" &&\n+\n+\tln -sfn small.svn.r0-5 link.svn &&\n+\tgit fetch svn &&\n+\ttest_nr_revs 5 .git refs/remotes/svn/master &&\n+\n+\trm -rf .git/info/fast-import/svn-alpha &&\n+\n+\tln -sfn small.svn.r0-10 link.svn &&\n+\tgit fetch svn &&\n+\ttest_nr_revs 10 .git refs/remotes/svn/master\n+'\n+\n+test_expect_success SMALL_SVN 'clone to bootstrap' '\n+\tdeinit_git &&\n+\turl=$(svnurl link.svn) &&\n+\tln -sfn small.svn.r0-5 link.svn &&\n+\tgit clone -b master \"$url\" master.git &&\n+\t{\n+\t\tcd master.git &&\n+\t\tgit branch pub_head refs/svn-alpha/origin/SVNHEAD &&\n+\t\tgit branch pub_notes refs/svn-alpha/origin/SVNR &&\n+\t\tcd ..\n+\t} &&\n+\tln -sfn small.svn.r0-8 link.svn &&\n+\tgit clone master.git slave.git &&\n+\t{\n+\t\tcd slave.git &&\n+\t\tgit remote add svn \"$url\" &&\n+\t\tgit update-ref refs/svn-alpha/svn/SVNHEAD refs/remotes/origin/pub_head &&\n+\t\tgit update-ref refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&\n+\t\tgit fetch svn &&\n+\t\tgit merge-base refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&\n+\t\tcd ..\n+\t} &&\n+\ttest_nr_revs 8 slave.git/.git refs/remotes/svn/master\n+'\n+\n+test_expect_success SMALL_SVN 'clone and exchange' '\n+\tdeinit_git &&\n+\turl=$(svnurl link.svn) &&\n+\tln -sfn small.svn.r0-1 link.svn &&\n+\tgit clone -b master \"$url\" A.git &&\n+\tgit clone -b master \"$url\" B.git &&\n+\ttest_nr_revs 1 A.git/.git origin/master &&\n+\ttest_nr_revs 1 B.git/.git origin/master &&\n+\n+\tgit --git-dir=A.git/.git remote add B B.git &&\n+\tgit --git-dir=B.git/.git remote add A A.git &&\n+\trefspecs=\"+refs/svn-alpha/origin/SVNR:refs/svn-alpha/origin/SVNR\"\n+\trefspecs=\"$refspecs refs/svn-alpha/origin/SVNHEAD:refs/svn-alpha/origin/SVNHEAD\"\n+\n+\tln -sfn small.svn.r0-2 link.svn &&\n+\tgit --git-dir=A.git/.git fetch origin &&\n+\n+\tgit --git-dir=B.git/.git fetch -f A $refspecs &&\n+\n+\tln -sfn small.svn.r0-3 link.svn &&\n+\tgit --git-dir=B.git/.git fetch origin &&\n+\n+\tgit --git-dir=A.git/.git fetch B $refspecs &&\n+\n+\tgit --git-dir=A.git/.git fetch origin &&\n+\tgit --git-dir=B.git/.git fetch origin &&\n+\n+\ttest_nr_revs 3 A.git/.git origin/master &&\n+\ttest_nr_revs 3 B.git/.git origin/master\n+'\n+\n test_done\n-- \n1.7.3.4\n"},{"id":"171259","messageId":"1310570813-16648-10-git-send-email-divanorama@gmail.com","threadId":"27806","inReplyTo":"1310570813-16648-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v2 9/9] svn-fe: reuse import-marks in remote-svn-alpha","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-07-13T15:26:53Z","receivedAt":"2011-07-13T15:26:53Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"remote-svn-alpha needs to prepare import marks file when doing an\nincremental import. This file can be left after previous imports.\n\nTake the last imported commit and a note with corresponding svn\nrevision number. If the marks file exists and defines a :revision\nmark pointing to the last imported commit, consider it valid.\nIt may accidentally contain newer revisions, they are silently\nignored. In practice the check for :revision last_commit should\nbe sufficient to rely on this file defining any lesser revision\nmarks needed correctly.\n\nSo if the import marks file looks valid, do not regenerate it from\nthe notes tree and have save some performance saved.\n\nSigned-off-by: Dmitry Ivankov <divanorama@gmail.com>\n---\n contrib/svn-fe/git-remote-svn-alpha |   26 ++++++++++++++++++++++----\n 1 files changed, 22 insertions(+), 4 deletions(-)\n\ndiff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha\nindex f4e737d..8096ef2 100755\n--- a/contrib/svn-fe/git-remote-svn-alpha\n+++ b/contrib/svn-fe/git-remote-svn-alpha\n@@ -78,14 +78,31 @@ do_gen_marks () {\n prepare_marks () {\n \tdst=$1 notes=$2 rev=$3\n \ttest \"z$git_dir\" != \"$z\" || die \"we were not told the gitdir\"\n-\tmkdir -p \"$git_dir/$marks_dir\"\n-\tif test ! -f \"$git_dir/$marks_dir/marks\"; then\n-\t\ttouch \"$git_dir/$marks_dir/marks\"\n+\tpath=\"$git_dir/$marks_dir\"\n+\tmkdir -p \"$path\"\n+\tpath=\"$path/marks\"\n+\n+\tif test ! -f \"$path\"; then\n+\t\ttouch \"$path\"\n \tfi\n \tif test \"$rev\" = \"-1\"; then\n \t\treturn 0\n \tfi\n-\tdo_gen_marks \"$notes\" >\"$git_dir/$marks_dir/marks\"\n+\tmark_sha=\"\"\n+\t{\n+\t\twhile read -r m sha\n+\t\tdo\n+\t\t\tif test \"$m\" = \":$rev\"; then\n+\t\t\t\tmark_sha=\"$sha\"\n+\t\t\tfi\n+\t\tdone\n+\t} <\"$path\"\n+\tdst_sha=$( git rev-parse \"$dst\" )\n+\tif test -n \"$mark_sha\"; then\n+\t\ttest \"$mark_sha\" = \"$dst_sha\" || die \"latest mark and note diverge\"\n+\telse\n+\t\tdo_gen_marks \"$notes\" >\"$path\"\n+\tfi\n }\n \n last_imported_rev () {\n@@ -110,6 +127,7 @@ do_import () {\n \trevs=\"$start_rev:HEAD\"\n \tprepare_marks $dst $notes $rev\n \techo \"feature import-marks=$git_dir/$marks_dir/marks\"\n+\techo \"feature export-marks=$git_dir/$marks_dir/marks\"\n \t(eval \"$SVNDUMP \\\"$url\\\" -r\\\"$revs\\\"\" |\tsvn-fe --ref=\"$dst\" --notes-ref=\"$notes\" --incremental --no-progress) 3<&0 || die \"FAILURE\"\n \texec 1>&-\n }\n-- \n1.7.3.4\n"}]}