{"thread":{"id":"27564","subject":"[PATCHv2] Add a remote helper to interact with mediawiki, pull & clone handled","startedAt":"2011-06-07T09:59:14Z","lastAt":"2011-06-07T11:33:09Z","messageCount":4,"participants":["Jeremie Nikaes","Thomas Adam","Matthieu Moy","Arnaud Lacurie"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"169430","messageId":"1307440754-3697-1-git-send-email-jeremie.nikaes@ensimag.imag.fr","threadId":"27564","inReplyTo":null,"subject":"[PATCHv2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jeremie Nikaes","fromEmail":"jeremie.nikaes@ensimag.imag.fr","sentAt":"2011-06-07T09:59:14Z","receivedAt":"2011-06-07T09:59:14Z","isPatch":false,"sender":{"key":"jeremie.nikaes@ensimag.imag.fr","avatar":null},"body":"Implement a gate between git and mediawiki, allowing git users to\npush and pull objects from mediawiki just as one would do with a\nclassic git repository thanks to remote-helpers.\n\nCurrently supported commands are :\n     git clone mediawiki::http://onewiki.com\n     git pull\n\nYou need the following packages installed (available on common\nrepositories):\n     libmediawiki-api-perl\n     libdatetime-format-iso8601-perl\n\nUse remote helpers in order to be as transparent as possible\nto the git user.\n\nDownload Mediawiki revisions through the Mediawiki API and then\nfast-import into git.\n\nMediawiki revisions and git commits are linked thanks to notes bound to\ncommits.\n\nThe import part is done on a refs/mediawiki/<remote> branch before\ncoming to refs/remote/origin/master (Huge thanks to Jonathan Nieder\nfor his help)\n\nFor now, the whole wiki is cloned, but it will be possible to clone only\nsome pages: the clone is based on a list of pages which is now all\npages.\n\nCode clarified & improved with the help of Jeff King and Junio C Hamano.\n\nWe were not able to reproduce the empty timestamp bug noticed by Jeff\nKing, thus needing some further testing. A placeholder is still\nimplemented just in case.\n\nSigned-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>\nSigned-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>\nSigned-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>\nSigned-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>\nSigned-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>\nSigned-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>\n---\n Changes since v1\n - Better loop control using while(1) and removing use of switch\n - Removed unnecessary \"use storable qw(freeze thaw)\" leftover from Peff's script\n - Removed a few print STDERR lines used for debugging\n - Use of git notes show instead of git log and parsing\n - Added \"use warnings;\" and made the changes that these warnings pointed to\n (mainly uses of uninitialized variables in the parser)\n - Standardization of print STDOUT / print STDERR / print\n\n contrib/mw-to-git/git-remote-mediawiki     |  278 ++++++++++++++++++++++++++++\n contrib/mw-to-git/git-remote-mediawiki.txt |    7 +\n 2 files changed, 285 insertions(+), 0 deletions(-)\n create mode 100755 contrib/mw-to-git/git-remote-mediawiki\n create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nnew file mode 100755\nindex 0000000..34fe5dd\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -0,0 +1,278 @@\n+#! /usr/bin/perl\n+\n+use strict;\n+use MediaWiki::API;\n+use DateTime::Format::ISO8601;\n+use Encode qw(encode_utf8);\n+use warnings;\n+\n+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced\n+my $slash_replacement = \"<slash>\";\n+\n+my $remotename = $ARGV[0];\n+my $url = $ARGV[1];\n+\n+# commands parser\n+my $entry;\n+my @cmd;\n+while (1) {\n+\t$| = 1; #flush STDOUT\n+\t$entry = <STDIN>;\n+\tchomp($entry);\n+\t@cmd = split(/ /,$entry);\n+\tif (defined($cmd[0])) {\n+\t\tif ($cmd[0] eq \"capabilities\") {\n+\t\t\tlast unless (!defined($cmd[1]));\n+\t\t\tmw_capabilities();\n+\t\t} elsif ($cmd[0] eq \"list\") {\n+\t\t\tlast unless (!defined($cmd[2]));\n+\t\t\tmw_list($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"import\") {\n+\t\t\tlast unless ($cmd[1] ne \"\" && !defined($cmd[2]));\n+\t\t\tmw_import($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"option\") {\n+\t\t\tlast unless ($cmd[1] ne \"\" && $cmd[2] ne \"\" && !defined($cmd[3]));\n+\t\t\tmw_option($cmd[1],$cmd[2]);\n+\t\t} elsif ($cmd[0] eq \"push\") {\n+\t\t\t# Check the pattern <src>:<dst>\n+\t\t\tmy @pushargs = split(/:/,$cmd[1]);\n+\t\t\tlast unless ($pushargs[1] ne \"\" && !defined($pushargs[2]));\n+\t\t\tmw_push($pushargs[0],$pushargs[1]);\n+\t\t} else {\n+\t\t\tprint STDERR \"Unknown capability. Aborting...\\n\";\n+\t\t\tlast;\n+\t\t}\n+\t} else {\n+\t\t# End of input\n+\t\tlast;\n+\t}\n+\n+}\n+\n+########################## Functions ##############################\n+\n+sub get_last_local_revision {\n+\t# Get note regarding last mediawiki revision\n+\tmy $note = `git notes --ref=mediawiki show refs/mediawiki/$remotename/master 2>/dev/null`;\n+\tmy @note_info = split(/ /, $note);\n+\n+\tmy $lastrevision_number;\n+\tif (!(defined($note_info[0]) && $note_info[0] eq \"mediawiki_revision:\")) {\n+\t\tprint STDERR \"No previous mediawiki revision found\";\n+\t\t$lastrevision_number = 0;\n+\t} else {\n+\t\t# Notes are formatted : mediawiki_revision: #number\n+\t\t$lastrevision_number = $note_info[1];\n+\t\tchomp($lastrevision_number);\n+\t\tprint STDERR \"Last local mediawiki revision found is $lastrevision_number \";\n+\t}\n+\treturn $lastrevision_number;\n+}\n+\n+sub get_last_remote_revision {\n+\tmy $mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\n+\tmy $pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\taplimit => 500,\n+\t});\n+\n+\tmy $max_rev_num = 0;\n+\n+\tforeach my $page (@$pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\tpageids => $id,\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\t\t\n+\t\tmy $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});\n+\t\t\n+\t\t$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);\n+\t}\n+\n+\tprint STDERR \"Last remote revision found is $max_rev_num\\n\";\n+\treturn $max_rev_num;\n+}\n+\n+sub literal_data {\n+\tmy ($content) = @_;\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+}\n+\n+sub mw_capabilities {\n+\t# Revisions are imported to the private namespace\n+\t# refs/mediawiki/$remotename/ by the helper and fetched into\n+\t# refs/remotes/$remotename later by fetch.\n+\tprint STDOUT \"refspec refs/heads/*:refs/mediawiki/$remotename/*\\n\";\n+\tprint STDOUT \"import\\n\";\n+\tprint STDOUT \"list\\n\";\n+\tprint STDOUT \"option\\n\";\n+\tprint STDOUT \"push\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_list {\n+\t# MediaWiki do not have branches, we consider one branch arbitrarily\n+\t# called master\n+\tprint STDOUT \"? refs/heads/master\\n\";\n+\tprint STDOUT '@'.\"refs/heads/master HEAD\\n\";\n+\tprint STDOUT \"\\n\";\n+\n+}\n+\n+sub mw_option {\n+\tprint STDOUT \"unsupported\\n\";\n+}\n+\n+sub mw_import {\n+\tmy @wiki_name = split(/:\\/\\//,$url);\n+\tmy $wiki_name = $wiki_name[1];\n+\n+\tmy $mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\n+\tmy $pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\taplimit => 500,\n+\t});\n+\t\n+\tif (!defined($pages)) {\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page\\n\";\n+\t\texit;\n+\t}\n+\n+\tmy @revisions;\n+\tprint STDERR \"Searching revisions...\\n\";\n+\tmy $fetch_from = get_last_local_revision() + 1;\n+\tif ($fetch_from == 1) {\n+\t\tprint STDERR \", fetching from beginning\\n\";\n+\t} else {\n+\t\tprint STDERR \", fetching from here\\n\";\n+\t}\n+\tmy $n = 1;\n+\tforeach my $page (@$pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tprint STDERR \"$n/\", scalar(@$pages), \": \". encode_utf8($page->{title}).\"\\n\";\n+\t\t$n++;\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\trvdir => 'newer',\n+\t\t\trvstartid => $fetch_from,\n+\t\t\trvlimit => 500,\n+\t\t\tpageids => $page->{pageid},\n+\t\t};\n+\n+\t\tmy $revnum = 0;\n+\t\t# Get 500 revisions at a time due to the mediawiki api limit\n+\t\twhile (1) {\n+\t\t\tmy $result = $mediawiki->api($query);\n+\n+\t\t\t# Parse each of those 500 revisions\n+\t\t\tforeach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n+\t\t\t\tmy $page_rev_ids;\n+\t\t\t\t$page_rev_ids->{pageid} = $page->{pageid};\n+\t\t\t\t$page_rev_ids->{revid} = $revision->{revid};\n+\t\t\t\tpush (@revisions, $page_rev_ids);\n+\t\t\t\t$revnum++;\n+\t\t\t}\n+\t\t\tlast unless $result->{'query-continue'};\n+\t\t\t$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n+\t\t}\n+\t\tprint STDERR \"  Found \", $revnum, \" revision(s).\\n\";\n+\t}\n+\n+\t# Creation of the fast-import stream\n+\tprint STDERR \"Fetching & writing export data...\\n\";\n+\t\n+\t#binmode STDOUT, ':binary'; # Line from Jeff King that raises a warning. What is this line for ?\n+\t$n = 0;\n+\n+\tforeach my $pagerevids (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n+\t\t#fetch the content of the pages\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'content|timestamp|comment|user|ids',\n+\t\t\trevids => $pagerevids->{revid},\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $rev = pop(@{$result->{query}->{pages}->{$pagerevids->{pageid}}->{revisions}});\n+\n+\t\t$n++;\n+\t\tmy $user = $rev->{user} || 'Anonymous';\n+\t\tmy $dt = DateTime::Format::ISO8601->parse_datetime($rev->{timestamp});\n+\n+\t\tmy $comment = defined $rev->{comment} ? $rev->{comment} : '*Empty MediaWiki Message*';\n+\t\tmy $title = encode_utf8($result->{query}->{pages}->{$pagerevids->{pageid}}->{title});\n+\t\tmy $content = $rev->{'*'};\n+\t\t# This \\n is important. If it's not added, a conflict is going to happen if you change\n+\t\t# the last line of a file, push it and then pull it back from mediawiki.\n+\t\t# Note : it seems like there are still problems with this. To be investigated further\n+\t\t$content .= \"\\n\"; \n+\t\t\n+\t\t$title =~ y/ /_/;\n+\t\t$title =~ s/\\//$slash_replacement/g;\n+\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision n°$pagerevids->{revid} of $title\\n\";\n+\n+\t\tprint STDOUT \"commit refs/mediawiki/$remotename/master\\n\";\n+\t\tprint STDOUT \"mark :$n\\n\";\n+\t\tprint STDOUT \"committer $user <$user\\@$wiki_name> \", $dt->epoch, \" +0000\\n\";\n+\t\tliteral_data(encode_utf8($comment));\n+\t\t# If it's not a clone, needs to know where to start from\n+\t\tif ($fetch_from != 1 && $n == 1) {\n+\t\t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n+\t\t}\n+\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tliteral_data(encode_utf8($content));\n+\t\tprint STDOUT \"\\n\\n\";\n+\n+\n+\t\t# mediawiki revision number in the git note\n+\t\tif ($fetch_from == 1 && $n == 1) {\n+\t\t\tprint STDOUT \"reset refs/notes/mediawiki\\n\";\n+\t\t}\n+\t\tprint STDOUT \"commit refs/notes/mediawiki\\n\";\n+\t\tprint STDOUT \"committer $user <$user\\@$wiki_name> \", $dt->epoch, \" +0000\\n\";\n+\t\tliteral_data(encode_utf8(\"note added by git-mediawiki\"));\n+\t\tif ($fetch_from != 1 && $n == 1) {\n+\t\t\tprint STDOUT \"from refs/notes/mediawiki^0\\n\";\n+\t\t}\n+\t\tprint STDOUT \"N inline :$n\\n\";\n+\t\tliteral_data(encode_utf8(\"mediawiki_revision: \" . $pagerevids->{revid}));\n+\t\tprint STDOUT \"\\n\\n\";\n+\t}\n+\n+\tif ($fetch_from == 1) {\n+\t\tif ($n != 0) {\n+\t\t\tprint STDOUT \"reset $_[0]\\n\"; #$_[0] contains refs/heads/master\n+\t\t\tprint STDOUT \"from :$n\\n\";\n+\t\t} else {\n+\t\t\tprint STDERR \"You appear to have cloned an empty mediawiki\\n\";\n+\t\t\t#Something has to be done remote-helper side. If nothing is done, an error is \n+\t\t\t#thrown saying that HEAD is refering to unknown object 0000000000000000000\n+\t\t}\n+\t}\n+\n+}\n+\n+sub mw_push {\n+\tprint STDERR \"Push not yet implemented\\n\";\n+}\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt\nnew file mode 100644\nindex 0000000..4d211f5\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki.txt\n@@ -0,0 +1,7 @@\n+Git-Mediawiki is a project which aims the creation of a gate\n+between git and mediawiki, allowing git users to push and pull\n+objects from mediawiki just as one would do with a classic git\n+repository thanks to remote-helpers.\n+\n+For more information, visit the wiki at\n+https://github.com/Bibzball/Git-Mediawiki/wiki\n-- \n1.7.4.1\n"},{"id":"169432","messageId":"BANLkTik--BAqZxCYQ-ApUn2yKmcr_-hubg@mail.gmail.com","threadId":"27564","inReplyTo":"1307440754-3697-1-git-send-email-jeremie.nikaes@ensimag.imag.fr","subject":"Re: [PATCHv2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Thomas Adam","fromEmail":"thomas@xteddy.org","sentAt":"2011-06-07T10:16:30Z","receivedAt":"2011-06-07T10:16:30Z","isPatch":false,"sender":{"key":"thomas@xteddy.org","avatar":"https://gravatar.com/avatar/e7256db4738e501e5d2e84f00bb0bd99503165729573848a03330301fc2adc4a?d=mp&s=160"},"body":"Hi,\n\nOn 7 June 2011 10:59, Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr> wrote:\n> +       @cmd = split(/ /,$entry);\n\nHmm.  What guarantees can you make about the scalar value of @cmd here\nonce the split has happened?  Do you not care, or do you have a\nhard-limit for split to use?\n\n> +                       print STDERR \"Unknown capability. Aborting...\\n\";\n\nwarn()?\n\n> +       print STDOUT \"refspec refs/heads/*:refs/mediawiki/$remotename/*\\n\";\n> +       print STDOUT \"import\\n\";\n> +       print STDOUT \"list\\n\";\n> +       print STDOUT \"option\\n\";\n> +       print STDOUT \"push\\n\";\n> +       print STDOUT \"\\n\";\n\nThese explicit calls to \"STDOUT\" here -- why?  It's redundant typing\nand unsightly, and certainly not very perl-y.  Is there some reason\nwhy you're using this style?\n\n> +       print STDOUT \"? refs/heads/master\\n\";\n> +       print STDOUT '@'.\"refs/heads/master HEAD\\n\";\n> +       print STDOUT \"\\n\";\n\nHmm.  Why not just:\n\nprint STDOUT '@'.\"refs/heads/master HEAD\\n\\n\";\n\nThere's certainly a large number of perl-specific clean-ups I'd be\ninclined to do -- and if I get time later, I might show you how.  But\ndon't let this necessarily put off this patch for inclusion or\nanything like that.\n\n-- Thomas Adam\n"},{"id":"169437","messageId":"vpq1uz5khut.fsf@bauges.imag.fr","threadId":"27564","inReplyTo":"BANLkTik--BAqZxCYQ-ApUn2yKmcr_-hubg@mail.gmail.com","subject":"Re: [PATCHv2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-06-07T11:20:58Z","receivedAt":"2011-06-07T11:20:58Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Thomas Adam <thomas@xteddy.org> writes:\n\n>> +       print STDOUT \"? refs/heads/master\\n\";\n>> +       print STDOUT '@'.\"refs/heads/master HEAD\\n\";\n>> +       print STDOUT \"\\n\";\n>\n> Hmm.  Why not just:\n>\n> print STDOUT '@'.\"refs/heads/master HEAD\\n\\n\";\n\nTo replace the last two lines, yes, but the\n\n? refs/heads/master\n\nis still needed to show \"git pull\" that a branch master exists.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"169439","messageId":"BANLkTimTAhi6MZg_COopdyHwNifqmRVy0A@mail.gmail.com","threadId":"27564","inReplyTo":"BANLkTik--BAqZxCYQ-ApUn2yKmcr_-hubg@mail.gmail.com","subject":"Re: [PATCHv2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Arnaud Lacurie","fromEmail":"arnaud.lacurie@ensimag.imag.fr","sentAt":"2011-06-07T11:33:09Z","receivedAt":"2011-06-07T11:33:09Z","isPatch":false,"sender":{"key":"arnaud.lacurie@ensimag.imag.fr","avatar":null},"body":"Hi,\n\n2011/6/7 Thomas Adam <thomas@xteddy.org>:\n> Hmm.  What guarantees can you make about the scalar value of @cmd here\n> once the split has happened?  Do you not care, or do you have a\n> hard-limit for split to use?\n\nThe verifications on the different cmd[x] are made in the few lines\n\"if ... elsif... elsif ... else\" right after the split.\n\n> These explicit calls to \"STDOUT\" here -- why?  It's redundant typing\n> and unsightly, and certainly not very perl-y.  Is there some reason\n> why you're using this style?\n\nWe wanted to highlight the fact that we are sending those lines to STDOUT\nwhich is piped into the caller of the remote helper.\n\n>\n>> +       print STDOUT \"? refs/heads/master\\n\";\n>> +       print STDOUT '@'.\"refs/heads/master HEAD\\n\";\n>> +       print STDOUT \"\\n\";\n>\n> Hmm.  Why not just:\n>\n> print STDOUT '@'.\"refs/heads/master HEAD\\n\\n\";\n\nHere, we wanted to stick to the convention given by the remote-helper\ndocumentation that indicates the list command should end with a blank line.\n\n>\n> There's certainly a large number of perl-specific clean-ups I'd be\n> inclined to do -- and if I get time later, I might show you how.  But\n> don't let this necessarily put off this patch for inclusion or\n> anything like that.\n\nAs said in previous mails, we are relatively new using perl (it's been\nroughly two weeks)\nso any advice on how to get things more perl-ish are welcome.\n\nThanks for your feedback\n-- \nArnaud Lacurie\n"}]}