{"thread":{"id":"27532","subject":"[RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","startedAt":"2011-06-02T09:28:31Z","lastAt":"2011-06-03T03:43:48Z","messageCount":8,"participants":["Arnaud Lacurie","Jeff King","Junio C Hamano","Matthieu Moy"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"169179","messageId":"1307006911-4326-1-git-send-email-arnaud.lacurie@ensimag.imag.fr","threadId":"27532","inReplyTo":null,"subject":"[RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","fromName":"Arnaud Lacurie","fromEmail":"arnaud.lacurie@ensimag.imag.fr","sentAt":"2011-06-02T09:28:31Z","receivedAt":"2011-06-02T09:28:31Z","isPatch":true,"sender":{"key":"arnaud.lacurie@ensimag.imag.fr","avatar":null},"body":"Implements a gate between git and mediawiki, allowing git users to\npush and pull objects from mediawiki just as one would do with a\nclassic git repository thanks to remote-helpers.\n\nCurrently supported commands are :\n       git clone mediawiki::http://onewiki.com\n       git pull\n\nYou need the following packages installed (available on common\nrepositories):\n       libmediawiki-api-perl\n       libdatetime-format-iso8601-perl\n\nRemote helpers have been used in order to be as transparent as possible\nto the git user.\n\nMediawiki revisions are downloaded through the Mediawiki API and then\nfast-imported into git.\n\nMediawiki revisions and git commits are linked thanks to notes bound to\ncommits.\n\nThe import part is done on a refs/mediawiki/<remote> branch before\ncoming to refs/remote/origin/master (Huge thanks to Jonathan Nieder\nfor his help)\n\nFor now, the whole wiki is cloned, but it will be possible to clone only\nsome pages: the clone is based on a list of pages which is now all\npages.\n\nSigned-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>\nSigned-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>\nSigned-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>\nSigned-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>\nSigned-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>\nSigned-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>\n---\n contrib/mw-to-git/git-remote-mediawiki     |  252 ++++++++++++++++++++++++++++\n contrib/mw-to-git/git-remote-mediawiki.txt |    7 +\n 2 files changed, 259 insertions(+), 0 deletions(-)\n create mode 100755 contrib/mw-to-git/git-remote-mediawiki\n create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nnew file mode 100755\nindex 0000000..f20edc4\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -0,0 +1,252 @@\n+#! /usr/bin/perl\n+\n+use strict;\n+use Switch;\n+use MediaWiki::API;\n+use Storable qw(freeze thaw);\n+use DateTime::Format::ISO8601;\n+use Encode qw(encode_utf8);\n+\n+my $remotename = $ARGV[0];\n+my $url = $ARGV[1];\n+\n+print STDERR \"$url\\n\";\n+\n+# commands parser\n+my $loop = 1;\n+my $entry;\n+my @cmd;\n+while ($loop) {\n+\t$| = 1; #flush STDOUT\n+\t$entry = <STDIN>;\n+\tprint STDERR $entry;\n+\tchomp($entry);\n+\t@cmd = undef;\n+\t@cmd = split(/ /,$entry);\n+\tswitch ($cmd[0]) {\n+\t\tcase \"capabilities\" {\n+\t\t\tif ($cmd[1] eq \"\") {\n+\t\t\t\tmw_capabilities();\n+\t\t\t} else {\n+\t\t\t       $loop = 0;\n+\t\t\t}\n+\t\t}\n+\t\tcase \"list\" {\n+\t\t\tif ($cmd[2] eq \"\") {\n+\t\t\t\tmw_list($cmd[1]);\n+\t\t\t} else {\n+\t\t\t       $loop = 0;\n+\t\t\t}\n+\t\t}\n+\t\tcase \"import\" {\n+\t\t\tif ($cmd[1] ne \"\" && $cmd[2] eq \"\") {\n+\t\t\t\tmw_import($url);\n+\t\t\t} else {\n+\t\t\t       $loop = 0;\n+\t\t\t}\n+\t\t}\n+\t\tcase \"option\" {\n+\t\t\tmw_option($cmd[1],$cmd[2]);\n+\t\t}\n+\t\tcase \"push\" {\n+\t\t\t#check the pattern +<src>:<dist>\n+\t\t\tmy @pushargs = split(/:/,$cmd[1]);\n+\t\t\tif ($pushargs[1] ne \"\" && $pushargs[2] eq \"\"\n+\t\t\t&& (substr($pushargs[0],0,1) eq \"+\")) {\n+\t\t\t\tmw_push(substr($pushargs[0],1),$pushargs[1]);\n+\t\t\t} else {\n+\t\t\t       $loop = 0;\n+\t\t\t}\n+\t\t} else {\n+\t\t\t$loop = 0;\n+\t\t}\n+\t}\n+\tclose(FILE);\n+}\n+\n+########################## Functions ##############################\n+\n+sub get_last_local_revision {\n+\t# Get last commit sha1\n+\tmy $commit_sha1 = `git rev-parse refs/mediawiki/$remotename/master 2>/dev/null`;\n+\n+\t# Get note regarding that commit\n+\tchomp($commit_sha1);\n+\tmy $note = `git notes show $commit_sha1 2>/dev/null`;\n+\tmy @note_info = split(/ /, $note);\n+\n+\tmy $lastrevision_number;\n+\tif (!($note_info[0] eq \"mediawiki_revision:\")) {\n+\t\tprint STDERR \"No previous mediawiki revision found, fetching from beginning\\n\";\n+\t\t$lastrevision_number = 0;\n+\t} else {\n+\t\t# Notes are formatted : mediawiki_revision: #number\n+\t\t$lastrevision_number = $note_info[1];\n+\t\tchomp($lastrevision_number);\n+\t\tprint STDERR \"Last mediawiki revision found is $lastrevision_number, fetching from here\\n\";\n+\t}\n+\treturn $lastrevision_number;\n+}\n+\n+sub mw_capabilities {\n+\t# Revisions are imported to the private namespace\n+\t# refs/mediawiki/$remotename/ by the helper and fetched into\n+\t# refs/remotes/$remotename later by fetch.\n+\tprint STDOUT \"refspec refs/heads/*:refs/mediawiki/$remotename/*\\n\";\n+\tprint STDOUT \"import\\n\";\n+\tprint STDOUT \"list\\n\";\n+\tprint STDOUT \"option\\n\";\n+\tprint STDOUT \"push\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_list {\n+\t# MediaWiki do not have branches, we consider one branch arbitrarily\n+\t# called master\n+\tprint STDOUT \"? refs/heads/master\\n\";\n+\tprint STDOUT '@'.\"refs/heads/master HEAD\\n\";\n+\tprint STDOUT \"\\n\";\n+\n+}\n+\n+sub mw_option {\n+\tprint STDERR \"not yet implemented \\n\";\n+\tprint STDOUT \"unsupported\\n\";\n+}\n+\n+sub mw_import {\n+\tmy @wiki_name = split(/:\\/\\//,$url);\n+\tmy $wiki_name = $wiki_name[1];\n+\n+\tmy $mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\n+\tmy $pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\taplimit => 500,\n+\t});\n+\tif ($pages == undef) {\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page\\n\";\n+\t\texit;\n+\t}\n+\n+\tmy @revisions;\n+\tprint STDERR \"Searching revisions...\\n\";\n+\tmy $fetch_from = get_last_local_revision() + 1;\n+\tmy $n = 1;\n+\tforeach my $page (@$pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tprint STDERR \"$n/\", scalar(@$pages), \": $page->{title}\\n\";\n+\t\t$n++;\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\trvdir => 'newer',\n+\t\t\trvstartid => $fetch_from,\n+\t\t\trvlimit => 500,\n+\t\t\tpageids => $page->{pageid},\n+\t\t};\n+\n+\t\tmy $revnum = 0;\n+\t\t# Get 500 revisions at a time due to the mediawiki api limit\n+\t\twhile (1) {\n+\t\t\tmy $result = $mediawiki->api($query);\n+\n+\t\t\t# Parse each of those 500 revisions\n+\t\t\tforeach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n+\t\t\t\tmy $page_rev_ids;\n+\t\t\t\t$page_rev_ids->{pageid} = $page->{pageid};\n+\t\t\t\t$page_rev_ids->{revid} = $revision->{revid};\n+\t\t\t\tpush (@revisions, $page_rev_ids);\n+\t\t\t\t$revnum++;\n+\t\t\t}\n+\t\t\tlast unless $result->{'query-continue'};\n+\t\t\t$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n+\t\t\tprint \"\\n\";\n+\t\t}\n+\t\tprint STDERR \"  Found \", $revnum, \" revision(s).\\n\";\n+\t}\n+\n+\t# Creation of the fast-import stream\n+\tprint STDERR \"Fetching & writing export data...\\n\";\n+\tbinmode STDOUT, ':binary';\n+\t$n = 0;\n+\n+\tforeach my $pagerevids (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n+\t\t#fetch the content of the pages\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'content|timestamp|comment|user|ids',\n+\t\t\trevids => $pagerevids->{revid},\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $rev = pop(@{$result->{query}->{pages}->{$pagerevids->{pageid}}->{revisions}});\n+\n+\t\t$n++;\n+\t\tmy $user = $rev->{user} || 'Anonymous';\n+\t\tmy $dt = DateTime::Format::ISO8601->parse_datetime($rev->{timestamp});\n+\n+\t\tmy $comment = defined $rev->{comment} ? $rev->{comment} : '*Empty MediaWiki Message*';\n+\t\tmy $title = $result->{query}->{pages}->{$pagerevids->{pageid}}->{title};\n+\t\tmy $content = $rev->{'*'};\n+\t\t$title =~ y/ /_/;\n+\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision n°$pagerevids->{revid} of $title\\n\";\n+\n+\t\tprint \"commit refs/mediawiki/$remotename/master\\n\";\n+\t\tprint \"mark :$n\\n\";\n+\t\tprint \"committer $user <$user\\@$wiki_name> \", $dt->epoch, \" +0000\\n\";\n+\t\tprint \"data \", bytes::length(encode_utf8($comment)), \"\\n\", encode_utf8($comment);\n+\t\t# If it's not a clone, needs to know where to start from\n+\t\tif ($fetch_from != 1 && $n == 1) {\n+\t\t\tprint \"from refs/mediawiki/$remotename/master^0\\n\";\n+\t\t}\n+\t\tprint \"M 644 inline $title.wiki\\n\";\n+\t\tprint \"data \", bytes::length(encode_utf8($content)), \"\\n\", encode_utf8($content);\n+\t\tprint \"\\n\\n\";\n+\n+\n+\t\t# mediawiki revision number in the git note\n+\t\tmy $note_comment = encode_utf8(\"note added by git-mediawiki\");\n+\t\tmy $note_comment_length = bytes::length($note_comment);\n+\t\tmy $note_content = encode_utf8(\"mediawiki_revision: \" . $pagerevids->{revid} . \"\\n\");\n+\t\tmy $note_content_length = bytes::length($note_content);\n+\n+\t\tif ($fetch_from == 1 && $n == 1) {\n+\t\t\tprint \"reset refs/notes/commits\\n\";\n+\t\t}\n+\t\tprint \"commit refs/notes/commits\\n\";\n+\t\tprint \"committer $user <user\\@example.com> \", $dt->epoch, \" +0000\\n\";\n+\t\tprint \"data \", $note_comment_length, \"\\n\", $note_comment;\n+\t\tif ($fetch_from != 1 && $n == 1) {\n+\t\t\tprint \"from refs/notes/commits^0\\n\";\n+\t\t}\n+\t\tprint \"N inline :$n\\n\";\n+\t\tprint \"data \", $note_content_length, \"\\n\", $note_content;\n+\t\tprint \"\\n\\n\";\n+\t}\n+\n+\tif ($fetch_from == 1) {\n+\t\tif ($n != 0) {\n+\t\t\tprint \"reset refs/heads/master\\n\";\n+\t\t\tprint \"from :$n\\n\";\n+\t\t} else {\n+\t\t\tprint STDERR \"You appear to have cloned an empty mediawiki\\n\";\n+\t\t\t#What do we have to do here ? If nothing is done, an error is thrown saying that\n+\t\t\t#HEAD is refering to unknown object 0000000000000000000\n+\t\t}\n+\t}\n+\n+}\n+\n+sub mw_push {\n+\tprint STDERR \"not yet implemented \\n\";\n+}\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt\nnew file mode 100644\nindex 0000000..4d211f5\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki.txt\n@@ -0,0 +1,7 @@\n+Git-Mediawiki is a project which aims the creation of a gate\n+between git and mediawiki, allowing git users to push and pull\n+objects from mediawiki just as one would do with a classic git\n+repository thanks to remote-helpers.\n+\n+For more information, visit the wiki at\n+https://github.com/Bibzball/Git-Mediawiki/wiki\n-- \n1.7.5.3.486.gdc5865\n"},{"id":"169206","messageId":"20110602170327.GA2928@sigill.intra.peff.net","threadId":"27532","inReplyTo":"1307006911-4326-1-git-send-email-arnaud.lacurie@ensimag.imag.fr","subject":"Re: [RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-02T17:03:27Z","receivedAt":"2011-06-02T17:03:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 02, 2011 at 11:28:31AM +0200, Arnaud Lacurie wrote:\n\n> +sub mw_import {\n> [...]\n> +\t\t# Get 500 revisions at a time due to the mediawiki api limit\n> +\t\twhile (1) {\n> +\t\t\tmy $result = $mediawiki->api($query);\n> +\n> +\t\t\t# Parse each of those 500 revisions\n> +\t\t\tforeach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n> +\t\t\t\tmy $page_rev_ids;\n> +\t\t\t\t$page_rev_ids->{pageid} = $page->{pageid};\n> +\t\t\t\t$page_rev_ids->{revid} = $revision->{revid};\n> +\t\t\t\tpush (@revisions, $page_rev_ids);\n> +\t\t\t\t$revnum++;\n> +\t\t\t}\n> +\t\t\tlast unless $result->{'query-continue'};\n> +\t\t\t$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n> +\t\t\tprint \"\\n\";\n> +\t\t}\n\nWhat is this newline at the end here for? With it, my import reliably\nfails with:\n\n  fatal: Unsupported command: \n  fast-import: dumping crash report to .git/fast_import_crash_6091\n\nRemoving it seems to make things work.\n\n> +\t\tmy $user = $rev->{user} || 'Anonymous';\n> +\t\tmy $dt = DateTime::Format::ISO8601->parse_datetime($rev->{timestamp});\n> +\n> +\t\tmy $comment = defined $rev->{comment} ? $rev->{comment} : '*Empty MediaWiki Message*';\n\nIn importing the git wiki, I ran into an empty timestamp. This throws an\nexception which kills the whole import:\n\n  $ git clone mediawiki::https://git.wiki.kernel.org/ git-wiki\n  2821/7949: Revision n°4210 of GitSurvey\n  Invalid date format:  at /home/peff/compile/git/contrib/mw-to-git/git-remote-mediawiki line 195\n          main::mw_import('https://git.wiki.kernel.org/') called at /home/peff/compile/git/contrib/mw-to-git/git-remote-mediawiki line 42\n\nAt the very least, we should intercept this and put in some placeholder\ntimestamp. I'm not sure what the best placeholder would be. Maybe use\nthe date from the previous revision, plus one second? Or maybe there is\nsome other bug causing us to have an empty timestamp. I didn't dig\ndeeper yet.\n\n> +\t\t# mediawiki revision number in the git note\n> +\t\tmy $note_comment = encode_utf8(\"note added by git-mediawiki\");\n> +\t\tmy $note_comment_length = bytes::length($note_comment);\n> +\t\tmy $note_content = encode_utf8(\"mediawiki_revision: \" . $pagerevids->{revid} . \"\\n\");\n> +\t\tmy $note_content_length = bytes::length($note_content);\n> +\n> +\t\tif ($fetch_from == 1 && $n == 1) {\n> +\t\t\tprint \"reset refs/notes/commits\\n\";\n> +\t\t}\n> +\t\tprint \"commit refs/notes/commits\\n\";\n\nShould these go in refs/notes/commits? I don't think we have a \"best\npractices\" yet for the notes namespaces, as it is still a relatively new\nconcept. But I always thought \"refs/notes/commits\" would be for the\nuser's \"regular\" notes, and that programmatic things would get their own\nnotes, like \"refs/notes/mediawiki\".\n\nThat wouldn't show them by default, but you could do:\n\n  git log --notes=mediawiki\n\nto see them (and maybe that is a feature, because most of the time you\nwon't care about the mediawiki revision).\n\n> +\t\t} else {\n> +\t\t\tprint STDERR \"You appear to have cloned an empty mediawiki\\n\";\n> +\t\t\t#What do we have to do here ? If nothing is done, an error is thrown saying that\n> +\t\t\t#HEAD is refering to unknown object 0000000000000000000\n> +\t\t}\n\nHmm. We do allow cloning empty git repos. It might be nice for there to\nbe some way for a remote helper to signal \"everything OK, but the result\nis empty\". But I think that is probably something that needs to be added\nto the remote-helper protocol, and so is outside the scope of your\nscript (maybe it is as simple as interpreting the null sha1 as \"empty\";\nI dunno).\n\nOverall, it's looking pretty good. I like that I can resume a\nhalf-finished import via \"git fetch\". Though I do have one complaint:\nrunning \"git fetch\" fetches the metainfo for every revision of every\npage, just as it does for an initial clone. Is there something in the\nmediawiki API to say \"show me revisions since N\" (where N would be the\nmediawiki revision of the tip of what we imported)?\n\n-Peff\n"},{"id":"169210","messageId":"7vy61kcdu8.fsf@alter.siamese.dyndns.org","threadId":"27532","inReplyTo":"1307006911-4326-1-git-send-email-arnaud.lacurie@ensimag.imag.fr","subject":"Re: [RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-02T18:01:19Z","receivedAt":"2011-06-02T18:01:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr> writes:\n\n>  contrib/mw-to-git/git-remote-mediawiki     |  252 ++++++++++++++++++++++++++++\n>  contrib/mw-to-git/git-remote-mediawiki.txt |    7 +\n>  2 files changed, 259 insertions(+), 0 deletions(-)\n\nIt is pleasing to see that a half of a custom backend can be done in just\n250 lines of code.  I understand that this is a work-in-progress with many\nunnecessary lines spitting debugging output to STDERR, whose removal will\nfurther shrink the code?\n\n> +# commands parser\n> +my $loop = 1;\n> +my $entry;\n> +my @cmd;\n> +while ($loop) {\n\nThis is somewhat unusual-looking loop control.\n\nWouldn't \"while (1) { ...; last if (...); if (...) { last; } }\" do?\n\n> +\t$| = 1; #flush STDOUT\n> +\t$entry = <STDIN>;\n> +\tprint STDERR $entry;\n> +\tchomp($entry);\n> +\t@cmd = undef;\n> +\t@cmd = split(/ /,$entry);\n> +\tswitch ($cmd[0]) {\n> +\t\tcase \"capabilities\" {\n> +\t\t\tif ($cmd[1] eq \"\") {\n> +\t\t\t\tmw_capabilities();\n> +\t\t\t} else {\n> +\t\t\t       $loop = 0;\n\nI presume that this is \"We were expecting to read capabilities command but\nfound something unexpected; let's abort\". Don't you want to say something\nto the user here, perhaps on STDERR?\n\n> +\t\t\t}\n> +\t\t}\n> ...\n> +\t\tcase \"option\" {\n> +\t\t\tmw_option($cmd[1],$cmd[2]);\n> +\t\t}\n\nNo error checking only for this one?\n\n> +\t\tcase \"push\" {\n> +\t\t\t#check the pattern +<src>:<dist>\n\nThe latter one is usually spelled <dst> standing for \"destination\".\n\n> +\t\t\tmy @pushargs = split(/:/,$cmd[1]);\n> +\t\t\tif ($pushargs[1] ne \"\" && $pushargs[2] eq \"\"\n> +\t\t\t&& (substr($pushargs[0],0,1) eq \"+\")) {\n> +\t\t\t\tmw_push(substr($pushargs[0],1),$pushargs[1]);\n> +\t\t\t} else {\n> +\t\t\t       $loop = 0;\n> +\t\t\t}\n\nIs \"push\" always forcing?\n\n> +sub mw_import {\n> +\tmy @wiki_name = split(/:\\/\\//,$url);\n> +\tmy $wiki_name = $wiki_name[1];\n> +\n> +\tmy $mediawiki = MediaWiki::API->new;\n> +\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n> +\n> +\tmy $pages = $mediawiki->list({\n> +\t\taction => 'query',\n> +\t\tlist => 'allpages',\n> +\t\taplimit => 500,\n> +\t});\n> +\tif ($pages == undef) {\n> +\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n> +\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page\\n\";\n> +\t\texit;\n> +\t}\n> +\n> +\tmy @revisions;\n> +\tprint STDERR \"Searching revisions...\\n\";\n> +\tmy $fetch_from = get_last_local_revision() + 1;\n> +\tmy $n = 1;\n> +\tforeach my $page (@$pages) {\n> +\t\tmy $id = $page->{pageid};\n> +\n> +\t\tprint STDERR \"$n/\", scalar(@$pages), \": $page->{title}\\n\";\n> +\t\t$n++;\n> +\n> +\t\tmy $query = {\n> +\t\t\taction => 'query',\n> +\t\t\tprop => 'revisions',\n> +\t\t\trvprop => 'ids',\n> +\t\t\trvdir => 'newer',\n> +\t\t\trvstartid => $fetch_from,\n> +\t\t\trvlimit => 500,\n> +\t\t\tpageids => $page->{pageid},\n> +\t\t};\n> +\n> +\t\tmy $revnum = 0;\n> +\t\t# Get 500 revisions at a time due to the mediawiki api limit\n\nIt's nice that you can dig deeper with rvlimit increments. I wonder if\n'allpages' also let's you retrieve more than 500 pages in total by somehow\niterating over the set of pages.\n\n> +\t# Creation of the fast-import stream\n> +\tprint STDERR \"Fetching & writing export data...\\n\";\n> +\tbinmode STDOUT, ':binary';\n> +\t$n = 0;\n> +\n> +\tforeach my $pagerevids (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n> +\t\t#fetch the content of the pages\n> +\t\tmy $query = {\n> +\t\t\taction => 'query',\n> +\t\t\tprop => 'revisions',\n> +\t\t\trvprop => 'content|timestamp|comment|user|ids',\n> +\t\t\trevids => $pagerevids->{revid},\n> +\t\t};\n> +\n> +\t\tmy $result = $mediawiki->api($query);\n> +\n> +\t\tmy $rev = pop(@{$result->{query}->{pages}->{$pagerevids->{pageid}}->{revisions}});\n\nIs the list of per-page revisions guaranteed to be sorted (not a\nrhetorical question; just asking)?\n\n> +\t\tprint \"commit refs/mediawiki/$remotename/master\\n\";\n> +\t\tprint \"mark :$n\\n\";\n> +\t\tprint \"committer $user <$user\\@$wiki_name> \", $dt->epoch, \" +0000\\n\";\n> +\t\tprint \"data \", bytes::length(encode_utf8($comment)), \"\\n\", encode_utf8($comment);\n\nCalling encode_utf8() twice on the same data?  How big is this $comment\ntypically?  Or does encode_utf8() somehow memoize?\n\n> +\t\t# If it's not a clone, needs to know where to start from\n> +\t\tif ($fetch_from != 1 && $n == 1) {\n> +\t\t\tprint \"from refs/mediawiki/$remotename/master^0\\n\";\n> +\t\t}\n> +\t\tprint \"M 644 inline $title.wiki\\n\";\n> +\t\tprint \"data \", bytes::length(encode_utf8($content)), \"\\n\", encode_utf8($content);\n\nSame for $content, which presumably is larger than $comment...\n\nPerhaps a small helper\n\n\tsub literal_data {\n        \tmy ($content) = @_;\n                print \"data \", bytes::length($content), \"\\n\", $content;\n\t}\n\nwould help here, above, and below where you create a \"note\" on this\ncommit?\n\n> +\t\t# mediawiki revision number in the git note\n> +\t\tmy $note_comment = encode_utf8(\"note added by git-mediawiki\");\n> +\t\tmy $note_comment_length = bytes::length($note_comment);\n> +\t\tmy $note_content = encode_utf8(\"mediawiki_revision: \" . $pagerevids->{revid} . \"\\n\");\n> +\t\tmy $note_content_length = bytes::length($note_content);\n> +\n> +\t\tif ($fetch_from == 1 && $n == 1) {\n> +\t\t\tprint \"reset refs/notes/commits\\n\";\n> +\t\t}\n> +\t\tprint \"commit refs/notes/commits\\n\";\n> +\t\tprint \"committer $user <user\\@example.com> \", $dt->epoch, \" +0000\\n\";\n> +\t\tprint \"data \", $note_comment_length, \"\\n\", $note_comment;\n\nWith that, this will become\n\n\tliteral_data(encode_utf8(\"note added by git-mediawiki\"));\n\nand you don't need two extra variables.  Same for $note_content*.\n"},{"id":"169221","messageId":"BANLkTi=eYg3uT1hQZO03i4MLyhRkPzXK6w@mail.gmail.com","threadId":"27532","inReplyTo":"20110602170327.GA2928@sigill.intra.peff.net","subject":"Re: [RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","fromName":"Arnaud Lacurie","fromEmail":"arnaud.lacurie@ensimag.imag.fr","sentAt":"2011-06-02T20:28:33Z","receivedAt":"2011-06-02T20:28:33Z","isPatch":true,"sender":{"key":"arnaud.lacurie@ensimag.imag.fr","avatar":null},"body":"2011/6/2 Jeff King <peff@peff.net>:\n> On Thu, Jun 02, 2011 at 11:28:31AM +0200, Arnaud Lacurie wrote:\n>\n>> +sub mw_import {\n>> [...]\n>> +             # Get 500 revisions at a time due to the mediawiki api limit\n>> +             while (1) {\n>> +                     my $result = $mediawiki->api($query);\n>> +\n>> +                     # Parse each of those 500 revisions\n>> +                     foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n>> +                             my $page_rev_ids;\n>> +                             $page_rev_ids->{pageid} = $page->{pageid};\n>> +                             $page_rev_ids->{revid} = $revision->{revid};\n>> +                             push (@revisions, $page_rev_ids);\n>> +                             $revnum++;\n>> +                     }\n>> +                     last unless $result->{'query-continue'};\n>> +                     $query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n>> +                     print \"\\n\";\n>> +             }\n>\n> What is this newline at the end here for? With it, my import reliably\n> fails with:\n>\n>  fatal: Unsupported command:\n>  fast-import: dumping crash report to .git/fast_import_crash_6091\n>\n> Removing it seems to make things work.\n\n Yes we actually found it today. It slipped as we've never fetched\npages with more than 500 revisions since it got there...\n\n>> +             # mediawiki revision number in the git note\n>> +             my $note_comment = encode_utf8(\"note added by git-mediawiki\");\n>> +             my $note_comment_length = bytes::length($note_comment);\n>> +             my $note_content = encode_utf8(\"mediawiki_revision: \" . $pagerevids->{revid} . \"\\n\");\n>> +             my $note_content_length = bytes::length($note_content);\n>> +\n>> +             if ($fetch_from == 1 && $n == 1) {\n>> +                     print \"reset refs/notes/commits\\n\";\n>> +             }\n>> +             print \"commit refs/notes/commits\\n\";\n>\n> Should these go in refs/notes/commits? I don't think we have a \"best\n> practices\" yet for the notes namespaces, as it is still a relatively new\n> concept. But I always thought \"refs/notes/commits\" would be for the\n> user's \"regular\" notes, and that programmatic things would get their own\n> notes, like \"refs/notes/mediawiki\".\n>\nThat's a good idea, we didn't think notes could actually not go in\nrefs/notes/commits. This will be perfect to distinguish the user notes\nfrom ours.\n>\n>> +             } else {\n>> +                     print STDERR \"You appear to have cloned an empty mediawiki\\n\";\n>> +                     #What do we have to do here ? If nothing is done, an error is thrown saying that\n>> +                     #HEAD is refering to unknown object 0000000000000000000\n>> +             }\n>\n> Hmm. We do allow cloning empty git repos. It might be nice for there to\n> be some way for a remote helper to signal \"everything OK, but the result\n> is empty\". But I think that is probably something that needs to be added\n> to the remote-helper protocol, and so is outside the scope of your\n> script (maybe it is as simple as interpreting the null sha1 as \"empty\";\n> I dunno).\n>\n\nYes, that's a problem we've been running into. We didn't really know\nhow to solve it.\n\n> Overall, it's looking pretty good. I like that I can resume a\n> half-finished import via \"git fetch\". Though I do have one complaint:\n> running \"git fetch\" fetches the metainfo for every revision of every\n> page, just as it does for an initial clone. Is there something in the\n> mediawiki API to say \"show me revisions since N\" (where N would be the\n> mediawiki revision of the tip of what we imported)?\n\nI am not sure I understand your question. Because actually, we are\nsupporting this,\nthanks to git notes. Like when you git fetch after a clone, it checks\nonly the last revisions\n\nThank you very much for your help !\n\nArnaud Lacurie\n"},{"id":"169224","messageId":"BANLkTimCONrGyX=v39RXL4pxGWbkLVROPA@mail.gmail.com","threadId":"27532","inReplyTo":"7vy61kcdu8.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","fromName":"Arnaud Lacurie","fromEmail":"arnaud.lacurie@ensimag.imag.fr","sentAt":"2011-06-02T20:58:44Z","receivedAt":"2011-06-02T20:58:44Z","isPatch":true,"sender":{"key":"arnaud.lacurie@ensimag.imag.fr","avatar":null},"body":"2011/6/2 Junio C Hamano <gitster@pobox.com>:\n> Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr> writes:\n>\n>>  contrib/mw-to-git/git-remote-mediawiki     |  252 ++++++++++++++++++++++++++++\n>>  contrib/mw-to-git/git-remote-mediawiki.txt |    7 +\n>>  2 files changed, 259 insertions(+), 0 deletions(-)\n>\n> It is pleasing to see that a half of a custom backend can be done in just\n> 250 lines of code.  I understand that this is a work-in-progress with many\n> unnecessary lines spitting debugging output to STDERR, whose removal will\n> further shrink the code?\n\nTo give you an idea, the whole thing with the other part is about 400\nlines for now.\nThere are a lot of output to STDERR that will be eventually removed.\nWe still have to\ndecide what \"option verbosity\" should display before removing to many of them.\n\n\n>> +# commands parser\n>> +my $loop = 1;\n>> +my $entry;\n>> +my @cmd;\n>> +while ($loop) {\n>\n> This is somewhat unusual-looking loop control.\n>\n> Wouldn't \"while (1) { ...; last if (...); if (...) { last; } }\" do?\n\nOk for that loop control.\n\n>> +     $| = 1; #flush STDOUT\n>> +     $entry = <STDIN>;\n>> +     print STDERR $entry;\n>> +     chomp($entry);\n>> +     @cmd = undef;\n>> +     @cmd = split(/ /,$entry);\n>> +     switch ($cmd[0]) {\n>> +             case \"capabilities\" {\n>> +                     if ($cmd[1] eq \"\") {\n>> +                             mw_capabilities();\n>> +                     } else {\n>> +                            $loop = 0;\n>\n> I presume that this is \"We were expecting to read capabilities command but\n> found something unexpected; let's abort\". Don't you want to say something\n> to the user here, perhaps on STDERR?\n>\n\nActually, we based that \"no error messages displayed\" policy on the\ngit-remote-http which aborts if something unexpected is found. We could add an\nerror message though.\n\n>> +                     }\n>> +             }\n>> ...\n>> +             case \"option\" {\n>> +                     mw_option($cmd[1],$cmd[2]);\n>> +             }\n>\n> No error checking only for this one?\n\nThere should be one. That's a miss.\n\n\n\n>> +                     my @pushargs = split(/:/,$cmd[1]);\n>> +                     if ($pushargs[1] ne \"\" && $pushargs[2] eq \"\"\n>> +                     && (substr($pushargs[0],0,1) eq \"+\")) {\n>> +                             mw_push(substr($pushargs[0],1),$pushargs[1]);\n>> +                     } else {\n>> +                            $loop = 0;\n>> +                     }\n>\n> Is \"push\" always forcing?\n\nThe tests are done in the mw_push function which is still being implemented.\nThe push will abort if there is something to pull, just like git acts.\n\n>> +sub mw_import {\n>> +     my @wiki_name = split(/:\\/\\//,$url);\n>> +     my $wiki_name = $wiki_name[1];\n>> +\n>> +     my $mediawiki = MediaWiki::API->new;\n>> +     $mediawiki->{config}->{api_url} = \"$url/api.php\";\n>> +\n>> +     my $pages = $mediawiki->list({\n>> +             action => 'query',\n>> +             list => 'allpages',\n>> +             aplimit => 500,\n>> +     });\n>> +     if ($pages == undef) {\n>> +             print STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n>> +             print STDERR \"fatal: make sure '$url/api.php' is a valid page\\n\";\n>> +             exit;\n>> +     }\n>> +\n>> +     my @revisions;\n>> +     print STDERR \"Searching revisions...\\n\";\n>> +     my $fetch_from = get_last_local_revision() + 1;\n>> +     my $n = 1;\n>> +     foreach my $page (@$pages) {\n>> +             my $id = $page->{pageid};\n>> +\n>> +             print STDERR \"$n/\", scalar(@$pages), \": $page->{title}\\n\";\n>> +             $n++;\n>> +\n>> +             my $query = {\n>> +                     action => 'query',\n>> +                     prop => 'revisions',\n>> +                     rvprop => 'ids',\n>> +                     rvdir => 'newer',\n>> +                     rvstartid => $fetch_from,\n>> +                     rvlimit => 500,\n>> +                     pageids => $page->{pageid},\n>> +             };\n>> +\n>> +             my $revnum = 0;\n>> +             # Get 500 revisions at a time due to the mediawiki api limit\n>\n> It's nice that you can dig deeper with rvlimit increments. I wonder if\n> 'allpages' also let's you retrieve more than 500 pages in total by somehow\n> iterating over the set of pages.\n\nYes it is possible. And works with the removal of the \"\\n\" peff\npointed out in the previous message.\n\n>> +     # Creation of the fast-import stream\n>> +     print STDERR \"Fetching & writing export data...\\n\";\n>> +     binmode STDOUT, ':binary';\n>> +     $n = 0;\n>> +\n>> +     foreach my $pagerevids (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n>> +             #fetch the content of the pages\n>> +             my $query = {\n>> +                     action => 'query',\n>> +                     prop => 'revisions',\n>> +                     rvprop => 'content|timestamp|comment|user|ids',\n>> +                     revids => $pagerevids->{revid},\n>> +             };\n>> +\n>> +             my $result = $mediawiki->api($query);\n>> +\n>> +             my $rev = pop(@{$result->{query}->{pages}->{$pagerevids->{pageid}}->{revisions}});\n>\n> Is the list of per-page revisions guaranteed to be sorted (not a\n> rhetorical question; just asking)?\n\nYes it is.\n\n\n>> +             print \"commit refs/mediawiki/$remotename/master\\n\";\n>> +             print \"mark :$n\\n\";\n>> +             print \"committer $user <$user\\@$wiki_name> \", $dt->epoch, \" +0000\\n\";\n>> +             print \"data \", bytes::length(encode_utf8($comment)), \"\\n\", encode_utf8($comment);\n>\n> Calling encode_utf8() twice on the same data?  How big is this $comment\n> typically?  Or does encode_utf8() somehow memoize?\n>\n>> +             # If it's not a clone, needs to know where to start from\n>> +             if ($fetch_from != 1 && $n == 1) {\n>> +                     print \"from refs/mediawiki/$remotename/master^0\\n\";\n>> +             }\n>> +             print \"M 644 inline $title.wiki\\n\";\n>> +             print \"data \", bytes::length(encode_utf8($content)), \"\\n\", encode_utf8($content);\n>\n> Same for $content, which presumably is larger than $comment...\n>\n> Perhaps a small helper\n>\n>        sub literal_data {\n>                my ($content) = @_;\n>                print \"data \", bytes::length($content), \"\\n\", $content;\n>        }\n>\n> would help here, above, and below where you create a \"note\" on this\n> commit?\n>\n>> +             # mediawiki revision number in the git note\n>> +             my $note_comment = encode_utf8(\"note added by git-mediawiki\");\n>> +             my $note_comment_length = bytes::length($note_comment);\n>> +             my $note_content = encode_utf8(\"mediawiki_revision: \" . $pagerevids->{revid} . \"\\n\");\n>> +             my $note_content_length = bytes::length($note_content);\n>> +\n>> +             if ($fetch_from == 1 && $n == 1) {\n>> +                     print \"reset refs/notes/commits\\n\";\n>> +             }\n>> +             print \"commit refs/notes/commits\\n\";\n>> +             print \"committer $user <user\\@example.com> \", $dt->epoch, \" +0000\\n\";\n>> +             print \"data \", $note_comment_length, \"\\n\", $note_comment;\n>\n> With that, this will become\n>\n>        literal_data(encode_utf8(\"note added by git-mediawiki\"));\n>\n> and you don't need two extra variables.  Same for $note_content*.\n\nThank you very much for this helper, it'll help factoring the code and\nreducing the number of variables.\n\n\nThank you very much for your help and comments on that project.\n\nRegards\n\n-- \nArnaud Lacurie\n"},{"id":"169227","messageId":"vpqy61jami7.fsf@bauges.imag.fr","threadId":"27532","inReplyTo":"20110602170327.GA2928@sigill.intra.peff.net","subject":"Re: [RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-06-02T22:37:04Z","receivedAt":"2011-06-02T22:37:04Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jeff King <peff@peff.net> writes:\n\n> Overall, it's looking pretty good. I like that I can resume a\n> half-finished import via \"git fetch\". Though I do have one complaint:\n> running \"git fetch\" fetches the metainfo for every revision of every\n> page, just as it does for an initial clone. Is there something in the\n> mediawiki API to say \"show me revisions since N\" (where N would be the\n> mediawiki revision of the tip of what we imported)?\n\nThe idea is that we ultimately want to be able to import a subset of a\nlarge wiki. In Wikipedia, for example, \"show me revisions since N\" will\nbe very large after a few minutes. OTOH, \"show me revisions touching the\nfew pages I'm following\" should be fast. And at least, it's O(imported\nwiki size), not O(complete wiki size)\n\nIdeally, there could be heuristics like\n\n\"show me how many revisions since N\"\nif (not many) {\n    \"OK, show me them all in details\"\n} else {\n    \"hmm, we'll do it another way, show me revisions touching my pages\"\n}\n\nbut let's not be too ambitious for now: it's a student's project,\ncompleting one week from now, and the goal is to have something clean\nand extensible. Bells and whistles will come later ;-).\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"169229","messageId":"20110602224931.GA28189@sigill.intra.peff.net","threadId":"27532","inReplyTo":"BANLkTi=eYg3uT1hQZO03i4MLyhRkPzXK6w@mail.gmail.com","subject":"Re: [RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-02T22:49:31Z","receivedAt":"2011-06-02T22:49:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 02, 2011 at 10:28:33PM +0200, Arnaud Lacurie wrote:\n\n> > Overall, it's looking pretty good. I like that I can resume a\n> > half-finished import via \"git fetch\". Though I do have one complaint:\n> > running \"git fetch\" fetches the metainfo for every revision of every\n> > page, just as it does for an initial clone. Is there something in the\n> > mediawiki API to say \"show me revisions since N\" (where N would be the\n> > mediawiki revision of the tip of what we imported)?\n> \n> I am not sure I understand your question. Because actually, we are\n> supporting this, thanks to git notes. Like when you git fetch after a\n> clone, it checks only the last revisions\n\nSorry, I was partially wrong in what I wrote above. I was resuming a\nfailed import (because of the bogus timestamp), so the numbers of\nrevisions were still high, and I thought they were the same as in the\noriginal version. I see now doing a fetch on the completed import that\nit properly finds 0 revisions for each page. So that's good.\n\nHowever, it does still take O(number of pages) requests just to find out\nthat there is nothing to fetch. For the git wiki, this takes on the\norder of 1.5 minutes to do an empty fetch. When getting the list of\npages, I wonder if there is a way in the mediawiki API to say \"show me\nonly pages which have been modified since rev N\".\n\n-Peff\n"},{"id":"169239","messageId":"20110603034348.GA1371@sigill.intra.peff.net","threadId":"27532","inReplyTo":"vpqy61jami7.fsf@bauges.imag.fr","subject":"Re: [RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-03T03:43:48Z","receivedAt":"2011-06-03T03:43:48Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 03, 2011 at 12:37:04AM +0200, Matthieu Moy wrote:\n\n> The idea is that we ultimately want to be able to import a subset of a\n> large wiki. In Wikipedia, for example, \"show me revisions since N\" will\n> be very large after a few minutes. OTOH, \"show me revisions touching the\n> few pages I'm following\" should be fast. And at least, it's O(imported\n> wiki size), not O(complete wiki size)\n\nYeah, I think what you want to do is dependent on wiki size. For a small\nwiki, it doesn't matter; all pages is not much. For a large wiki, you\nwant a subset of the pages, and you _never_ want to do any operations on\nthe whole page space. In the middle are medium-sized wikis, where you\nwould like look at the whole page space, but ideally not in O(number of\npages).\n\nBut the point is somewhat moot, because having just read through the\nmediawiki API, I've come to the conclusion (which seems familiar\nfrom the last time I looked at this problem) that there is no way to ask\nfor what I want in a single query. That is, to say \"show me all\nrevisions of all pages matching some subset X, that have been modified\nsince revision N\". Or even \"show me all pages matching some subset X\nthat have been modified since revision N\", and then we could at least\ncull the pages that haven't been touched.\n\nBut AFAICT, none of those is possible. I think we are stuck asking for\neach page's information individually (you can even query multiple pages'\nrevision information simultaneously, but you can get only a single\nrevision from each in that case. There's not even a way to say \"get me\nthe latest revision number for all of these pages).\n\nOne thing we could do to reduce the total run-time is to issue several\nqueries in parallel so that the query latency isn't so prevalent. I\ndon't know what a good level of parallelism is for a server like\nwikipedia, though. I'm sure they don't appreciate users hammering the\nservers too hard. Ideally you want just enough queries outstanding that\nthe remote server is always working on _one_, and the rest are doing\nsomething else (traveling across the network, local processing and\nstorage, etc). But I'm not sure of a good way to measure that.\n\n> but let's not be too ambitious for now: it's a student's project,\n> completing one week from now, and the goal is to have something clean\n> and extensible. Bells and whistles will come later ;-).\n\nYes. I think all of this is outside the scope of a student project. I\njust like to dream. :)\n\n-Peff\n"}]}