{"thread":{"id":"27592","subject":"[RFC/PATCH 2/2] Git-remote-mediawiki: Add push support","startedAt":"2011-06-09T13:15:59Z","lastAt":"2011-06-10T07:22:46Z","messageCount":10,"participants":["Jeremie Nikaes","Sverre Rabbelier","Jérémie NIKAES","Junio C Hamano","Jeff King","Arnaud Lacurie"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"169761","messageId":"1307625360-10973-1-git-send-email-jeremie.nikaes@ensimag.imag.fr","threadId":"27592","inReplyTo":null,"subject":"[PATCHv3 1/2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jeremie Nikaes","fromEmail":"jeremie.nikaes@ensimag.imag.fr","sentAt":"2011-06-09T13:15:59Z","receivedAt":"2011-06-09T13:15:59Z","isPatch":false,"sender":{"key":"jeremie.nikaes@ensimag.imag.fr","avatar":null},"body":"Implement a gate between git and mediawiki, allowing git users to\npush and pull objects from mediawiki just as one would do with a\nclassic git repository thanks to remote-helpers.\n\nCurrently supported commands are :\n     git clone mediawiki::http://onewiki.com\n     git pull\n\nYou need the following packages installed (available on common\nrepositories):\n     libmediawiki-api-perl\n     libdatetime-format-iso8601-perl\n\nUse remote helpers in order to be as transparent as possible\nto the git user.\n\nDownload Mediawiki revisions through the Mediawiki API and then\nfast-import into git.\n\nMediawiki revisions and git commits are linked thanks to notes bound to\ncommits.\n\nThe import part is done on a refs/mediawiki/<remote> branch before\ncoming to refs/remote/origin/master (Huge thanks to Jonathan Nieder\nfor his help)\n\nFor now, the whole wiki is cloned, but it will be possible to clone only\nsome pages: the clone is based on a list of pages which is now all\npages.\n\nCode clarified & improved with the help of Jeff King and Junio C Hamano.\n\nWe were not able to reproduce the empty timestamp bug noticed by Jeff\nKing, thus needing some further testing. A placeholder is still\nimplemented just in case. Its value is the value of the last valid\ntimestamp received + 1\n\nWith \"use encoding 'utf-8'\" non-iso characters are now fully supported\nin both file content and filename.\nA small helper run_git is also added to execute any git command, helping\nto also utf-8 encode results from git commands.\nHowever, utf-8 encoding for filenames could raise problems if different\nfile systems handle utf-8 filenames differently. A uri_escape of\nmediawiki filenames could be imaginable, and is still to be discussed\nfurther.\n\nPartial cloning is supported using the following syntax :\n\"git clone mediawiki::http://wikiurl##A_Page##Another_Page\"\nAs always, this url is kept in .git/config, helping to always keep\ntrack of these specific pages\n\nSigned-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>\nSigned-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>\nSigned-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>\nSigned-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>\nSigned-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>\nSigned-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>\n---\n -- diff with v2\n Added the timestamp placeholder\n Utf8 encoding now properly supported\n Partial cloning functionality added\n\n contrib/mw-to-git/git-remote-mediawiki     |  322 ++++++++++++++++++++++++++++\n contrib/mw-to-git/git-remote-mediawiki.txt |    7 +\n 2 files changed, 329 insertions(+), 0 deletions(-)\n create mode 100755 contrib/mw-to-git/git-remote-mediawiki\n create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nnew file mode 100755\nindex 0000000..176ff09\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -0,0 +1,322 @@\n+#! /usr/bin/perl\n+\n+use strict;\n+use MediaWiki::API;\n+use DateTime::Format::ISO8601;\n+use encoding 'utf8';\n+use URI::Escape;\n+use warnings;\n+\n+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced\n+my $slash_replacement = \"%2F\";\n+\n+my $remotename = $ARGV[0];\n+# Current syntax to fetch only a set of pages mediawiki::http://mediawikiurl##A_Page##Another_Page\n+my @pages_titles = split(/##/,$ARGV[1]);\n+my $url = shift (@pages_titles);\n+\n+\n+# commands parser\n+my $entry;\n+my @cmd;\n+while (1) {\n+\t$| = 1; #flush STDOUT\n+\t$entry = <STDIN>;\n+\tchomp($entry);\n+\t@cmd = split(/ /,$entry);\n+\tif (defined($cmd[0])) {\n+\t\tif ($cmd[0] eq \"capabilities\") {\n+\t\t\tlast unless (!defined($cmd[1]));\n+\t\t\tmw_capabilities();\n+\t\t} elsif ($cmd[0] eq \"list\") {\n+\t\t\tlast unless (!defined($cmd[2]));\n+\t\t\tmw_list($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"import\") {\n+\t\t\tlast unless ($cmd[1] ne \"\" && !defined($cmd[2]));\n+\t\t\tmw_import($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"option\") {\n+\t\t\tlast unless ($cmd[1] ne \"\" && $cmd[2] ne \"\" && !defined($cmd[3]));\n+\t\t\tmw_option($cmd[1],$cmd[2]);\n+\t\t} elsif ($cmd[0] eq \"push\") {\n+\t\t\t# Check the pattern <src>:<dst>\n+\t\t\tmy @pushargs = split(/:/,$cmd[1]);\n+\t\t\tlast unless ($pushargs[1] ne \"\" && !defined($pushargs[2]));\n+\t\t\tmw_push($pushargs[0],$pushargs[1]);\n+\t\t} else {\n+\t\t\tprint STDERR \"Unknown capability. Aborting...\\n\";\n+\t\t\tlast;\n+\t\t}\n+\t} else {\n+\t\t# End of input\n+\t\tlast;\n+\t}\n+\n+}\n+\n+########################## Functions ##############################\n+sub get_pages{\n+\tmy $mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\n+\tmy $pages;\n+\tif (!@pages_titles){\n+\t\t$pages = $mediawiki->list({\n+\t\t\taction => 'query',\n+\t\t\tlist => 'allpages',\n+\t\t\taplimit => 500,\n+\t\t});\n+\t\tif (!defined($pages)) {\n+\t\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page\\n\";\n+\t\t\texit 1;\n+\t\t}\n+\t\treturn @$pages;\n+\t} else {\n+\t\t#the list of titles should follow the pattern 'page1|page2|...'\n+\t\tmy $titles = \"\";\n+\t\tforeach my $title (@pages_titles){\n+\t\t\t$titles.=\"$title|\";\n+\t\t}\n+\t\t#supress the last | that is add in the foreach\n+\t\tchop($titles);\n+\n+\t\t$pages = $mediawiki->api({\n+\t\t\taction => 'query',\n+\t\t\ttitles => $titles,\n+\t\t});\n+\t\tif (!defined($pages)) {\n+\t\t\tprint STDERR \"fatal: None of the pages exist \\n\";\n+\t\t\texit 1;\n+\t\t}\n+\t\treturn values (%{$pages->{query}->{pages}});\n+\t}\n+}\n+\n+sub run_git {\n+\topen(my $git, \"-|:encoding(UTF-8)\", \"git \" . $_[0]);\n+\tmy $res = do { local $/; <$git> };\n+\tclose($git);\n+\n+\treturn $res;\n+}\n+\n+\n+sub get_last_local_revision {\n+\t# Get note regarding last mediawiki revision\n+\tmy $note = run_git(\"notes --ref=mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n+\tmy @note_info = split(/ /, $note);\n+\n+\tmy $lastrevision_number;\n+\tif (!(defined($note_info[0]) && $note_info[0] eq \"mediawiki_revision:\")) {\n+\t\tprint STDERR \"No previous mediawiki revision found\";\n+\t\t$lastrevision_number = 0;\n+\t} else {\n+\t\t# Notes are formatted : mediawiki_revision: #number\n+\t\t$lastrevision_number = $note_info[1];\n+\t\tchomp($lastrevision_number);\n+\t\tprint STDERR \"Last local mediawiki revision found is $lastrevision_number \";\n+\t}\n+\treturn $lastrevision_number;\n+}\n+\n+sub get_last_remote_revision {\n+\tmy $mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\n+\tmy @pages = get_pages();\n+\n+\tmy $max_rev_num = 0;\n+\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\tpageids => $id,\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\t\t\n+\t\tmy $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});\n+\t\t\n+\t\t$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);\n+\t}\n+\n+\tprint STDERR \"Last remote revision found is $max_rev_num\\n\";\n+\treturn $max_rev_num;\n+}\n+\n+sub literal_data {\n+\tmy ($content) = @_;\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+}\n+\n+sub mw_capabilities {\n+\t# Revisions are imported to the private namespace\n+\t# refs/mediawiki/$remotename/ by the helper and fetched into\n+\t# refs/remotes/$remotename later by fetch.\n+\tprint STDOUT \"refspec refs/heads/*:refs/mediawiki/$remotename/*\\n\";\n+\tprint STDOUT \"import\\n\";\n+\tprint STDOUT \"list\\n\";\n+\tprint STDOUT \"option\\n\";\n+\tprint STDOUT \"push\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_list {\n+\t# MediaWiki do not have branches, we consider one branch arbitrarily\n+\t# called master\n+\tprint STDOUT \"? refs/heads/master\\n\";\n+\tprint STDOUT '@'.\"refs/heads/master HEAD\\n\";\n+\tprint STDOUT \"\\n\";\n+\n+}\n+\n+sub mw_option {\n+\tprint STDOUT \"unsupported\\n\";\n+}\n+\n+sub mw_import {\n+\tmy @wiki_name = split(/:\\/\\//,$url);\n+\tmy $wiki_name = $wiki_name[1];\n+\n+\tmy $mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\n+\tmy @pages = get_pages();\n+\n+\tmy @revisions;\n+\tprint STDERR \"Searching revisions...\\n\";\n+\tmy $fetch_from = get_last_local_revision() + 1;\n+\tif ($fetch_from == 1) {\n+\t\tprint STDERR \", fetching from beginning\\n\";\n+\t} else {\n+\t\tprint STDERR \", fetching from here\\n\";\n+\t}\n+\tmy $n = 1;\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tprint STDERR \"$n/\", scalar(@pages), \": \". $page->{title}.\"\\n\";\n+\t\t$n++;\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\trvdir => 'newer',\n+\t\t\trvstartid => $fetch_from,\n+\t\t\trvlimit => 500,\n+\t\t\tpageids => $id,\n+\t\t};\n+\n+\t\tmy $revnum = 0;\n+\t\t# Get 500 revisions at a time due to the mediawiki api limit\n+\t\twhile (1) {\n+\t\t\tmy $result = $mediawiki->api($query);\n+\n+\t\t\t# Parse each of those 500 revisions\n+\t\t\tforeach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n+\t\t\t\tmy $page_rev_ids;\n+\t\t\t\t$page_rev_ids->{pageid} = $page->{pageid};\n+\t\t\t\t$page_rev_ids->{revid} = $revision->{revid};\n+\t\t\t\tpush (@revisions, $page_rev_ids);\n+\t\t\t\t$revnum++;\n+\t\t\t}\n+\t\t\tlast unless $result->{'query-continue'};\n+\t\t\t$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n+\t\t}\n+\t\tprint STDERR \"  Found \", $revnum, \" revision(s).\\n\";\n+\t}\n+\n+\t# Creation of the fast-import stream\n+\tprint STDERR \"Fetching & writing export data...\\n\";\n+\t\n+\t$n = 0;\n+\tmy $last_timestamp = 0; #Placeholer in case $rev->timestamp is undefined\n+\n+\tforeach my $pagerevids (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n+\t\t#fetch the content of the pages\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'content|timestamp|comment|user|ids',\n+\t\t\trevids => $pagerevids->{revid},\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $rev = pop(@{$result->{query}->{pages}->{$pagerevids->{pageid}}->{revisions}});\n+\n+\t\t$n++;\n+\t\tmy $user = $rev->{user} || 'Anonymous';\n+\n+\t\tif (!defined($rev->{timestamp})) {\n+\t\t\t$last_timestamp++;\n+\t\t} else {\n+\t\t\t$last_timestamp = $rev->{timestamp};\n+\t\t}\n+\t\tmy $dt = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n+\n+\t\tmy $comment = defined $rev->{comment} ? $rev->{comment} : '*Empty MediaWiki Message*';\n+\t\t\n+\t\tmy $title = $result->{query}->{pages}->{$pagerevids->{pageid}}->{title};\n+\t\t$title =~ y/ /_/;\n+\t\t#$title = uri_escape($title); #It would be nice to use uri_escape to be cross compatible\n+\t\t#on different file systems handling accentuated characters differently\n+\t\t$title =~ s/\\//$slash_replacement/g;\n+\t\t\n+\t\tmy $content = $rev->{'*'};\n+\t\t# This \\n is important. This is due to mediawiki's way to handle end of files.\n+\t\t$content .= \"\\n\";\n+\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevids->{revid} of $title\\n\";\n+\n+\t\tprint STDOUT \"commit refs/mediawiki/$remotename/master\\n\";\n+\t\tprint STDOUT \"mark :$n\\n\";\n+\t\tprint STDOUT \"committer $user <$user\\@$wiki_name> \", $dt->epoch, \" +0000\\n\";\n+\t\tliteral_data($comment);\n+\t\t# If it's not a clone, needs to know where to start from\n+\t\tif ($fetch_from != 1 && $n == 1) {\n+\t\t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n+\t\t}\n+\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tliteral_data($content);\n+\t\tprint STDOUT \"\\n\\n\";\n+\n+\n+\n+\n+\t\t# mediawiki revision number in the git note\n+\t\tif ($fetch_from == 1 && $n == 1) {\n+\t\t\tprint STDOUT \"reset refs/notes/mediawiki\\n\";\n+\t\t}\n+\t\tprint STDOUT \"commit refs/notes/mediawiki\\n\";\n+\t\tprint STDOUT \"committer $user <$user\\@$wiki_name> \", $dt->epoch, \" +0000\\n\";\n+\t\tliteral_data(\"note added by git-mediawiki\");\n+\t\tif ($fetch_from != 1 && $n == 1) {\n+\t\t\tprint STDOUT \"from refs/notes/mediawiki^0\\n\";\n+\t\t}\n+\t\tprint STDOUT \"N inline :$n\\n\";\n+\t\tliteral_data(\"mediawiki_revision: \" . $pagerevids->{revid});\n+\t\tprint STDOUT \"\\n\\n\";\n+\t}\n+\n+\tif ($fetch_from == 1) {\n+\t\tif ($n != 0) {\n+\t\t\tprint STDOUT \"reset $_[0]\\n\"; #$_[0] contains refs/heads/master\n+\t\t\tprint STDOUT \"from :$n\\n\";\n+\t\t} else {\n+\t\t\tprint STDERR \"You appear to have cloned an empty mediawiki\\n\";\n+\t\t\t#Something has to be done remote-helper side. If nothing is done, an error is \n+\t\t\t#thrown saying that HEAD is refering to unknown object 0000000000000000000\n+\t\t}\n+\t}\n+\n+}\n+\n+sub mw_push {\n+\tprint STDERR \"Push not yet implemented\\n\";\n+}\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt\nnew file mode 100644\nindex 0000000..4d211f5\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki.txt\n@@ -0,0 +1,7 @@\n+Git-Mediawiki is a project which aims the creation of a gate\n+between git and mediawiki, allowing git users to push and pull\n+objects from mediawiki just as one would do with a classic git\n+repository thanks to remote-helpers.\n+\n+For more information, visit the wiki at\n+https://github.com/Bibzball/Git-Mediawiki/wiki\n-- \n1.7.4.1\n"},{"id":"169760","messageId":"1307625360-10973-2-git-send-email-jeremie.nikaes@ensimag.imag.fr","threadId":"27592","inReplyTo":"1307625360-10973-1-git-send-email-jeremie.nikaes@ensimag.imag.fr","subject":"[RFC/PATCH 2/2] Git-remote-mediawiki: Add push support","fromName":"Jeremie Nikaes","fromEmail":"jeremie.nikaes@ensimag.imag.fr","sentAt":"2011-06-09T13:16:00Z","receivedAt":"2011-06-09T13:16:00Z","isPatch":true,"sender":{"key":"jeremie.nikaes@ensimag.imag.fr","avatar":null},"body":"Push is now supported by the remote-helper\nThanks to notes metadata, it is possible to compare remote and local\nlast mediawiki revision to warn non fast-forward and everything\nup-to-date.\n\nWhen allowed, push looks for each commit between remotes/origin/master\nand HEAD, catches every blob related to these commit and push them in\nchronological order. To do so, it uses git rev-list --children HEAD and\ntravels the tree from remotes/origin/master to HEAD through children. In\nother words :\n\n\t* Shortest path from remotes/origin/master to HEAD\n\t* For each commit encountered, push blobs related to this commit\n\nAn automatic git pull --rebase is executed after a successful push to\nget metadata back from mediawiki. This is also done to maintain\ncloseness with the form of a mediawiki history. It can be a problem\nsince it also flatens the entire history. (This solution is still\nto be discussed).\n\nTo send files to mediawiki, the mediawiki API is used. A filter is\napplied to the data send because mediawiki pages cannot have blank\ncharacters at the end. The filter is thus more or less a right trim.\n\nSigned-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>\nSigned-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>\nSigned-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>\nSigned-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>\nSigned-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>\nSigned-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>\n---\n contrib/mw-to-git/git-remote-mediawiki |   93 +++++++++++++++++++++++++++++++-\n 1 files changed, 92 insertions(+), 1 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 176ff09..dc1aacf 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -148,6 +148,14 @@ sub get_last_remote_revision {\n \treturn $max_rev_num;\n }\n \n+sub mediawiki_filter($) {\n+\t# Mediawiki does not allow blank space at the end of a page and ends with a single \\n.\n+\t# This function right trims a string and adds a \\n at the end to follow this rule\n+\tmy $string = shift;\n+\t$string =~ s/\\s+$//;\n+\treturn $string.\"\\n\";\n+}\n+\n sub literal_data {\n \tmy ($content) = @_;\n \tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n@@ -175,6 +183,7 @@ sub mw_list {\n }\n \n sub mw_option {\n+\tprint STDERR \"remote-helper capability 'option' not yet implemented \\n\";\n \tprint STDOUT \"unsupported\\n\";\n }\n \n@@ -318,5 +327,87 @@ sub mw_import {\n }\n \n sub mw_push {\n-\tprint STDERR \"Push not yet implemented\\n\";\n+\n+\tsub push_file {\n+\t\t#$_[0] contains a string in this format :\n+\t\t#100644 100644 <sha1_of_blob_before_commit> <sha1_of_blob_now> <status>\\0<filename.mw>\\0\n+\t\t#$_[1] contains the title of the commit message (the only phrase kept in the revision message)\n+\t\tmy @blob_info_split = split(/ |\\t|\\0/, $_[0]);\n+\n+\t\tmy $sha1 = $blob_info_split[3];\n+\t\tmy $complete_file_name = $blob_info_split[5];\n+\t\t# complete_file_name = uri_unescape($complete_file_name); # If we use the uri escape before\n+\t\t# we should unescape here, before anything\n+\t\t\n+\t\tif (substr($complete_file_name,-3) eq \".mw\"){\n+\t\t\tmy $title = substr($complete_file_name,0,-3);\n+\t\t\t$title =~ s/$slash_replacement/\\//g;\n+\n+\t\t\tmy $file_content = run_git(\"cat-file -p $sha1\");\n+\t\t\t\n+\t\t\tmy $mw = MediaWiki::API->new();\n+\t\t\t$mw->{config}->{api_url} = \"$url/api.php\";\n+\n+\t\t\t# log in to the wiki : here should be added a way to push changes with an identity\n+\t\t\t#$mw->login( { lgname => 'login', lgpassword => 'passwd' })\n+\t\t\t#|| die $mw->{error}->{code} . ': ' . $mw->{error}->{details};\n+\t\t\t\n+\t\t\t$mw->edit( {\n+\t\t\t\taction => 'edit',\n+\t\t\t\tsummary => $_[1],\n+\t\t\t\ttitle => $title,\n+\t\t\t\ttext => mediawiki_filter($file_content),\n+\t\t\t}, {\n+\t\t\t\tskip_encoding => 1 # Helps with names with accentuated characters\n+\t\t\t}) || die 'Fatal: Error ' . $mw->{error}->{code} . ' from mediwiki: ' . $mw->{error}->{details};\n+\n+\t\t\tprint STDERR \"Pushed file : $sha1 - $title\\n\";\n+\t\t} else {\n+\t\t\tprint STDERR \"$complete_file_name not a mediawiki file. '(Not pushable on this version)\\n\"\n+\t\t}\n+\t}\n+\t\n+\tmy $last_local_revid = get_last_local_revision();\n+\tmy $last_remote_revid = get_last_remote_revision();\n+\n+\t# Get sha1 of commit pointed by local HEAD\n+\tmy $HEAD_sha1 = run_git(\"rev-parse $_[0] 2>/dev/null\"); chomp($HEAD_sha1);\n+\t# Get sha1 of commit pointed by remotes/origin/master\n+\tmy $remoteorigin_sha1 = run_git(\"rev-parse refs/remotes/origin/master 2>/dev/null\"); chomp($remoteorigin_sha1);\n+\n+\tif ($last_local_revid < $last_remote_revid){\n+\t\tmy $message = \"\\\"To prevent you from losing history, non-fast-forward updates were rejected \\\\n\";\n+\t\t$message .= \"Merge the remote changes (e.g. 'git pull') before pushing again. See the\";\n+\t\t$message .= \" 'Note about fast-forwards' section of 'git push --help' for details.\\\"\";\n+\t\tprint STDOUT \"error $_[0] $message\\n\";\n+\t\tprint STDOUT \"\\n\";\n+\t} elsif ($HEAD_sha1 ne $remoteorigin_sha1) {\n+\t\t# Get every commit in between HEAD and refs/remotes/origin/master,\n+\t\t# including HEAD and refs/remotes/origin/master\n+\t\tmy $parsed_sha1 = $remoteorigin_sha1;\n+\t\twhile ($parsed_sha1 ne $HEAD_sha1) {\n+\t\t\tmy @commit_info =  grep(/^$parsed_sha1/, `git rev-list --children $_[0]`);\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $commit_info[0]);\n+\t\t\t# $commit_info_split[0] is the sha1 of the commit itself\n+\t\t\t# $commit_info_split[1] is the sha1 of its direct child\n+\t\t\tmy $blob_infos = run_git(\"diff --raw --abbrev=40 -z $commit_info_split[0] $commit_info_split[1]\");\n+\t\t\tmy @blob_info_list = split(/\\n/, $blob_infos);\n+\t\t\t# Keep the first line of the commit message as mediawiki comment for the revision\n+\t\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $commit_info_split[1]\")))[0];\n+\t\t\tchomp($commit_msg);\n+\t\t\tforeach my $blob_info (@blob_info_list) {\n+\t\t\t\t# Push every blob\n+\t\t\t\tpush_file($blob_info, $commit_msg);\n+\t\t\t}\n+\t\t\t$parsed_sha1 = $commit_info_split[1];\n+\t\t}\n+\n+\t\tprint STDOUT \"ok $_[1]\\n\";\n+\t\tprint STDOUT \"\\n\";\n+\t\t\n+\t\t# Pulling from mediawiki after pushing in order to keep things synchronized\n+\t\texec(\"git pull --rebase >/dev/null\");\n+\t} else {\n+\t\tprint STDOUT \"\\n\";\n+\t}\n }\n-- \n1.7.4.1\n"},{"id":"169764","messageId":"BANLkTikp6bGwVZqq7XAZ1bKVaMVLJ9XQiw@mail.gmail.com","threadId":"27592","inReplyTo":"1307625360-10973-1-git-send-email-jeremie.nikaes@ensimag.imag.fr","subject":"Re: [PATCHv3 1/2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-06-09T14:03:10Z","receivedAt":"2011-06-09T14:03:10Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Thu, Jun 9, 2011 at 15:15, Jeremie Nikaes\n<jeremie.nikaes@ensimag.imag.fr> wrote:\n> +sub mw_capabilities {\n> +       print STDOUT \"push\\n\";\n\n> +sub mw_push {\n> +       print STDERR \"Push not yet implemented\\n\";\n> +}\n\nIf it's not supported then you shouldn't advertise it above. Is there\nany particular reason you implemented it this way? E.g., did you\nencounter a breakage in remote-helpers if push is not implemented?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"169765","messageId":"BANLkTinduWxp5QR+Ri3REoqVY3hYPc_-rg@mail.gmail.com","threadId":"27592","inReplyTo":"BANLkTikp6bGwVZqq7XAZ1bKVaMVLJ9XQiw@mail.gmail.com","subject":"Re: [PATCHv3 1/2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@ensimag.imag.fr","sentAt":"2011-06-09T14:30:45Z","receivedAt":"2011-06-09T14:30:45Z","isPatch":false,"sender":{"key":"jeremie.nikaes@ensimag.imag.fr","avatar":null},"body":"2011/6/9 Sverre Rabbelier <srabbelier@gmail.com>:\n> Heya,\n>\n> On Thu, Jun 9, 2011 at 15:15, Jeremie Nikaes\n> <jeremie.nikaes@ensimag.imag.fr> wrote:\n>> +sub mw_capabilities {\n>> +       print STDOUT \"push\\n\";\n>\n>> +sub mw_push {\n>> +       print STDERR \"Push not yet implemented\\n\";\n>> +}\n>\n> If it's not supported then you shouldn't advertise it above. Is there\n> any particular reason you implemented it this way? E.g., did you\n> encounter a breakage in remote-helpers if push is not implemented?\n\nWell, if you dont give the capability to push, you get the following message\n\nerror: failed to push some refs to 'mediawiki::http://mediawikiurl'\n\nWhich is less explicit than 'Push not yet implemented'.\n\n\nRegards,\n-- \nJérémie Nikaes\n"},{"id":"169766","messageId":"BANLkTimRZ14NhD+evfZu2stHO+DFH0uuPw@mail.gmail.com","threadId":"27592","inReplyTo":"BANLkTinduWxp5QR+Ri3REoqVY3hYPc_-rg@mail.gmail.com","subject":"Re: [PATCHv3 1/2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-06-09T14:32:58Z","receivedAt":"2011-06-09T14:32:58Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\n2011/6/9 Jérémie NIKAES <jeremie.nikaes@ensimag.imag.fr>:\n> Well, if you dont give the capability to push, you get the following message\n>\n> error: failed to push some refs to 'mediawiki::http://mediawikiurl'\n>\n> Which is less explicit than 'Push not yet implemented'.\n\nI suspected as much. In that case perhaps the transport-helper.c code\nneeds to be updated to give a more helpful message?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"169792","messageId":"7vsjrjx6wp.fsf@alter.siamese.dyndns.org","threadId":"27592","inReplyTo":"1307625360-10973-2-git-send-email-jeremie.nikaes@ensimag.imag.fr","subject":"Re: [RFC/PATCH 2/2] Git-remote-mediawiki: Add push support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-09T17:15:50Z","receivedAt":"2011-06-09T17:15:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr> writes:\n\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\n> index 176ff09..dc1aacf 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki\n> +++ b/contrib/mw-to-git/git-remote-mediawiki\n> @@ -148,6 +148,14 @@ sub get_last_remote_revision {\n>  \treturn $max_rev_num;\n>  }\n>  \n> +sub mediawiki_filter($) {\n\nThe only caller calls this function with a plain vanilla single scalar\nstring, and this is not emulating to replace any Perl built-in. I do not\nsee why you want to confuse the readers with a prototype here.\n\n> @@ -318,5 +327,87 @@ sub mw_import {\n>  }\n>  \n>  sub mw_push {\n> -\tprint STDERR \"Push not yet implemented\\n\";\n> +\n> +\tsub push_file {\n\nThe language lets you to write nested functions, but in this case I do not\nthink it is buying you anything, other than one level unnecessarily deeper\nindentation to make the resulting code harder to read.\n\n> +\t\t#$_[0] contains a string in this format :\n> +\t\t#100644 100644 <sha1_of_blob_before_commit> <sha1_of_blob_now> <status>\\0<filename.mw>\\0\n> +\t\t#$_[1] contains the title of the commit message (the only phrase kept in the revision message)\n> +\t\tmy @blob_info_split = split(/ |\\t|\\0/, $_[0]);\n\nWhat if a filename has space or tab in it?  A code that reads from \"-z\"\noutput should not be using split().  Something like this (untested)?\n\n  # avoid $_[number] unless in a trivial few-liner function. they\n  # are unreadable.\n  my ($raw_diff, $message) = @_;\n  my ($old_mode, $new_mode, $old_sha1, $new_sha1, $status, $path) =\n  ($raw_diff =~ /^:([0-7]+) ([0-7]+) ([0-9a-f]{40}) ([0-9a-f]{40}) (\\S+)\\0(.*?)\\0$/) \n\n> +\t\tif (substr($complete_file_name,-3) eq \".mw\"){\n> +\t\t\tmy $title = substr($complete_file_name,0,-3);\n> +\t\t\t$title =~ s/$slash_replacement/\\//g;\n\nIt is probably more customary to write this like so:\n\t\n\tif (($title = $complete_file_name) =~ s/\\.mw$//) {\n\t\t...\n\n> +\t} elsif ($HEAD_sha1 ne $remoteorigin_sha1) {\n> +\t\t# Get every commit in between HEAD and refs/remotes/origin/master,\n> +\t\t# including HEAD and refs/remotes/origin/master\n> +\t\tmy $parsed_sha1 = $remoteorigin_sha1;\n> +\t\twhile ($parsed_sha1 ne $HEAD_sha1) {\n> +\t\t\tmy @commit_info =  grep(/^$parsed_sha1/, `git rev-list --children $_[0]`);\n\nIt feels extremely wasteful to traverse the whole history with rev-list\nevery time you interate this loop. Can't you do better?\n\n> +\t\t\tmy $blob_infos = run_git(\"diff --raw --abbrev=40 -z $commit_info_split[0] $commit_info_split[1]\");\n> +\t\t\tmy @blob_info_list = split(/\\n/, $blob_infos);\n\nHuh?  Didn't you read from \"-z\" output?\n\n> +\t\t\t# Keep the first line of the commit message as mediawiki comment for the revision\n> +\t\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $commit_info_split[1]\")))[0];\n> +\t\t\tchomp($commit_msg);\n> +\t\t\tforeach my $blob_info (@blob_info_list) {\n> +\t\t\t\t# Push every blob\n> +\t\t\t\tpush_file($blob_info, $commit_msg);\n> +\t\t\t}\n> +\t\t\t$parsed_sha1 = $commit_info_split[1];\n> +\t\t}\n> +\n> +\t\tprint STDOUT \"ok $_[1]\\n\";\n> +\t\tprint STDOUT \"\\n\";\n> +\t\t\n> +\t\t# Pulling from mediawiki after pushing in order to keep things synchronized\n> +\t\texec(\"git pull --rebase >/dev/null\");\n> +\t} else {\n> +\t\tprint STDOUT \"\\n\";\n> +\t}\n>  }\n"},{"id":"169822","messageId":"20110609224424.GA5364@sigill.intra.peff.net","threadId":"27592","inReplyTo":"1307625360-10973-1-git-send-email-jeremie.nikaes@ensimag.imag.fr","subject":"Re: [PATCHv3 1/2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-09T22:44:25Z","receivedAt":"2011-06-09T22:44:25Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 09, 2011 at 03:15:59PM +0200, Jeremie Nikaes wrote:\n\n> For now, the whole wiki is cloned, but it will be possible to clone only\n> some pages: the clone is based on a list of pages which is now all\n> pages.\n\nThis is not really true anymore, is it? Later you say:\n\n> Partial cloning is supported using the following syntax :\n> \"git clone mediawiki::http://wikiurl##A_Page##Another_Page\"\n> As always, this url is kept in .git/config, helping to always keep\n> track of these specific pages\n\nso I think it is not \"will be\" possible any more.\n\n> +sub get_pages{\n> +\tmy $mediawiki = MediaWiki::API->new;\n> +\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n> [...]\n> +\t} else {\n> +\t\t#the list of titles should follow the pattern 'page1|page2|...'\n> +\t\tmy $titles = \"\";\n> +\t\tforeach my $title (@pages_titles){\n> +\t\t\t$titles.=\"$title|\";\n> +\t\t}\n> +\t\t#supress the last | that is add in the foreach\n> +\t\tchop($titles);\n\nThis is usually spelled:\n\n  my $titles = join('|', @pages_titles);\n\n> +\t\t$pages = $mediawiki->api({\n> +\t\t\taction => 'query',\n> +\t\t\ttitles => $titles,\n> +\t\t});\n> +\t\tif (!defined($pages)) {\n> +\t\t\tprint STDERR \"fatal: None of the pages exist \\n\";\n> +\t\t\texit 1;\n> +\t\t}\n\nThat's not an accurate error message. If the pages don't exist, we will\nactually get back a set of pages with negative ids (so you can tell\nwhich ones exist and which ones don't). If $pages is undefined, it's\nactually not a valid mediawiki repo.\n\nAlso, according to the mediawiki API, we can send only 51 titles at a\ntime. So we need to break this into pieces.\n\nHowever, I wonder if this code path is needed at all. We are mapping\ntitles to page ids, so that we can later ask mediawiki for revisions by\npage id. But why not just ask for revisions by title, and skip this\nextra round trip to the server?\n\nSpeaking of round trips, I did have an idea for reducing round-trips in\nthe \"mostly up to date\" case. We can ask for the revisions for multiple\ntitles at once (apparently up to 51, or 501 if you have special bot\nprivileges), but you will only get the latest revision for each. Which\nisn't sufficient for us to do anything except tell whether or not there\nare any revisions to fetch.\n\nSo without the optimization, with N pages we will make N requests for\nnew revisions. But with it, we will make N/51 requests for the latest\nrevisions, and then M (where M <= N) for every page that actually has\nnew content. In other words, it is a good optimization as long as less\nthan 49/51 pages have changed, on average. So it's bad for \"clone\", but\nvery good for a subsequent \"fetch\". The best case is a fetch where\nnothing has changed, which should have 1/51th as many round-trips to\ndetermine that is the case.\n\n-Peff\n"},{"id":"169828","messageId":"20110610002137.GA11585@sigill.intra.peff.net","threadId":"27592","inReplyTo":"1307625360-10973-1-git-send-email-jeremie.nikaes@ensimag.imag.fr","subject":"Re: [PATCHv3 1/2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-10T00:21:38Z","receivedAt":"2011-06-10T00:21:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 09, 2011 at 03:15:59PM +0200, Jeremie Nikaes wrote:\n\n> Partial cloning is supported using the following syntax :\n> \"git clone mediawiki::http://wikiurl##A_Page##Another_Page\"\n> As always, this url is kept in .git/config, helping to always keep\n> track of these specific pages\n\nEarlier today I posted a 10-patch series to allow git to handle\nsomething like:\n\n  git clone \\\n    -c mediawiki.page=GitWorkflows \\\n    -c mediawiki.page=Tig \\\n    https://git.wiki.kernel.org\n\nand set that config into the .git/config file. The patch for the\nmediawiki helper to actually read and use it (instead of the \"##\" syntax\nin the url) is just:\n\n---\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex fd26f87..75c3537 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -11,10 +11,9 @@ use warnings;\n my $slash_replacement = \"%2F\";\n \n my $remotename = $ARGV[0];\n-# Current syntax to fetch only a set of pages mediawiki::http://mediawikiurl##A_Page##Another_Page\n-my @pages_titles = split(/##/,$ARGV[1]);\n-my $url = shift (@pages_titles);\n-\n+my $url = $ARGV[1];\n+my @pages_titles = `git config --get-all mediawiki.page`;\n+chomp @pages_titles;\n \n # commands parser\n my $entry;\n"},{"id":"169833","messageId":"BANLkTimruGZsh0aq7gOKmLUiJFwz3beU3g@mail.gmail.com","threadId":"27592","inReplyTo":"20110610002137.GA11585@sigill.intra.peff.net","subject":"Re: [PATCHv3 1/2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Arnaud Lacurie","fromEmail":"arnaud.lacurie@ensimag.imag.fr","sentAt":"2011-06-10T06:31:41Z","receivedAt":"2011-06-10T06:31:41Z","isPatch":false,"sender":{"key":"arnaud.lacurie@ensimag.imag.fr","avatar":null},"body":"Hi !\n\nWhat you did there is great !\nIt'll help clarify the command.\n\n2011/6/10 Jeff King <peff@peff.net>:\n> On Thu, Jun 09, 2011 at 03:15:59PM +0200, Jeremie Nikaes wrote:\n> Earlier today I posted a 10-patch series to allow git to handle\n> something like:\n>\n>  git clone \\\n>    -c mediawiki.page=GitWorkflows \\\n>    -c mediawiki.page=Tig \\\n>    https://git.wiki.kernel.org\n>\n\nSo we give up on the mediawiki prefix (mediawiki::https://git.wiki.kernel.org)\nwhen it comes to partial cloning ?\n\nRegards\n\n-- \nArnaud Lacurie\n"},{"id":"169834","messageId":"20110610072245.GA3385@sigill.intra.peff.net","threadId":"27592","inReplyTo":"BANLkTimruGZsh0aq7gOKmLUiJFwz3beU3g@mail.gmail.com","subject":"Re: [PATCHv3 1/2] Add a remote helper to interact with mediawiki, pull & clone handled","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-10T07:22:46Z","receivedAt":"2011-06-10T07:22:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 10, 2011 at 08:31:41AM +0200, Arnaud Lacurie wrote:\n\n> >  git clone \\\n> >    -c mediawiki.page=GitWorkflows \\\n> >    -c mediawiki.page=Tig \\\n> >    https://git.wiki.kernel.org\n> >\n> \n> So we give up on the mediawiki prefix (mediawiki::https://git.wiki.kernel.org)\n> when it comes to partial cloning ?\n\nOops, no, I just forgot it. You need the mediawiki prefix so git knows\nto invoke the mediawiki helper.\n\nSorry for the confusion.\n\n-Peff\n"}]}