{"thread":{"id":"30905","subject":"[PATCH 0/5] git-remote-mediawiki: support File: import and export","startedAt":"2012-06-26T16:04:28Z","lastAt":"2012-07-05T07:42:50Z","messageCount":18,"participants":["Matthieu Moy","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"194284","messageId":"1340726673-25496-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":null,"subject":"[PATCH 0/5] git-remote-mediawiki: support File: import and export","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-26T16:04:28Z","receivedAt":"2012-06-26T16:04:28Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"This is a rework of two patch series already discussed here:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/199877\nhttp://thread.gmane.org/gmane.comp.version-control.git/200002\n\nI've split the commits in smaller patches, hopefully they should be\neasier to read and should address remarks already made.\n\nThis is built on top of mm/credential-plumbing which also touches\ngit-remote-mediawiki, but should apply to master too.\n\nMatthieu Moy (3):\n git-remote-mediawiki: don't compute the diff when getting commit\n message\n git-remote-mediawiki: don't \"use encoding 'utf8';\"\n git-remote-mediawiki: split get_mw_pages into smaller functions\n\nNGUYEN Kim Thuat (1):\n git-remote-mediawiki: send \"File:\" attachments to a remote wiki\n\nPavel Volek (1):\n git-remote-mediawiki: import \"File:\" attachments\n\n contrib/mw-to-git/git-remote-mediawiki | 453 ++++++++++++++++++++++++++++-----\n 1 file changed, 390 insertions(+), 63 deletions(-)\n\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194289","messageId":"1340726673-25496-2-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340726673-25496-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 1/5] git-remote-mediawiki: don't compute the diff when getting commit message","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-26T16:04:29Z","receivedAt":"2012-06-26T16:04:29Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"---\n contrib/mw-to-git/git-remote-mediawiki | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex a34f53f..a8e6287 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -874,8 +874,7 @@ sub mw_push_revision {\n \t\t# TODO: for now, it's just a delete+add\n \t\tmy @diff_info_list = split(/\\0/, $diff_infos);\n \t\t# Keep the first line of the commit message as mediawiki comment for the revision\n-\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $sha1_commit\")))[0];\n+\t\tmy $commit_msg = (split(/\\n/, run_git(\"log -1 --format=\\\"%s\\\" $sha1_commit\")))[0];\n \t\tchomp($commit_msg);\n \t\t# Push every blob\n \t\twhile (@diff_info_list) {\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194285","messageId":"1340726673-25496-3-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340726673-25496-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 2/5] git-remote-mediawiki: don't \"use encoding 'utf8';\"","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-26T16:04:30Z","receivedAt":"2012-06-26T16:04:30Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"The use of this statement is generally discouraged, and is too intrusive\nfor us: it forces the HTTP requests made by the API to contain only valid\nUTF-8 characters. This would break the upload of binary files.\n---\n contrib/mw-to-git/git-remote-mediawiki | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex a8e6287..ed06ff7 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -36,11 +36,10 @@\n use strict;\n use MediaWiki::API;\n use DateTime::Format::ISO8601;\n-use encoding 'utf8';\n \n-# use encoding 'utf8' doesn't change STDERROR\n-# but we're going to output UTF-8 filenames to STDERR\n+# By default, use UTF-8 to communicate with Git and the user\n binmode STDERR, \":utf8\";\n+binmode STDOUT, \":utf8\";\n \n use URI::Escape;\n use IPC::Open2;\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194288","messageId":"1340726673-25496-4-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340726673-25496-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 3/5] git-remote-mediawiki: send \"File:\" attachments to a remote wiki","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-26T16:04:31Z","receivedAt":"2012-06-26T16:04:31Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: NGUYEN Kim Thuat \n\nThe current version of the git-remote-mediawiki supports only import and\nexport of plain wiki pages. This patch adds the functionality to export\nfile attachments (i.e. the content of the File: MediaWiki namespace),\nwhich are also exposed by MediaWiki API.\n\nThis requires a recent version of MediaWiki::API (Version 0.37 works.\nVersion 0.34 doesn't).\n\nSigned-off-by: Pavel Volek \nSigned-off-by: NGUYEN Kim Thuat \nSigned-off-by: ROUCHER IGLESIAS Javier \nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 98 +++++++++++++++++++++++++++++++---\n 1 file changed, 90 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex ed06ff7..253b449 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -348,8 +348,12 @@ sub get_mw_pages {\n \treturn values(%pages);\n }\n \n+# usage: $out = run_git(\"command args\");\n+# $out = run_git(\"command args\", \"raw\"); # don't interpret output as UTF-8.\n sub run_git {\n-\topen(my $git, \"-|:encoding(UTF-8)\", \"git \" . $_[0]);\n+\tmy $args = shift;\n+\tmy $encoding = (shift || \"encoding(UTF-8)\");\n+\topen(my $git, \"-|:$encoding\", \"git \" . $args);\n \tmy $res = do { local $/; <$git> };\n \tclose($git);\n \n@@ -705,6 +709,63 @@ sub error_non_fast_forward {\n \treturn 0;\n }\n \n+sub mw_upload_file {\n+\tmy $complete_file_name = shift;\n+\tmy $new_sha1 = shift;\n+\tmy $extension = shift;\n+\tmy $file_deleted = shift;\n+\tmy $summary = shift;\n+\tmy $newrevid;\n+\tmy $path = \"File:\" . $complete_file_name;\n+\tmy %hashFiles = get_allowed_file_extensions();\n+\tif (!exists($hashFiles{$extension})) {\n+\t\tprint STDERR \"$complete_file_name is not a permitted file on this wiki.\\n\";\n+\t\tprint STDERR \"Check the configuration of file uploads in your mediawiki.\\n\";\n+\t\treturn $newrevid;\n+\t}\n+\t# Deleting and uploading a file requires a priviledged user\n+\tif ($file_deleted) {\n+\t\tmw_connect_maybe();\n+\t\tmy $query = {\n+\t\t\taction => 'delete',\n+\t\t\ttitle => $path,\n+\t\t\treason => $summary\n+\t\t};\n+\t\tif (!$mediawiki->edit($query)) {\n+\t\t\tprint STDERR \"Failed to delete file on remote wiki\\n\";\n+\t\t\tprint STDERR \"Check your permissions on the remote site. Error code:\\n\";\n+\t\t\tprint STDERR $mediawiki->{error}->{code} . ':' . $mediawiki->{error}->{details};\n+\t\t\texit 1;\n+\t\t}\n+\t} else {\n+\t\t# Don't let perl try to interpret file content as UTF-8 => use \"raw\"\n+\t\tmy $content = run_git(\"cat-file blob $new_sha1\", \"raw\");\n+\t\tif ($content ne \"\") {\n+\t\t\tmw_connect_maybe();\n+\t\t\t$mediawiki->{config}->{upload_url} =\n+\t\t\t\t\"$url/index.php/Special:Upload\";\n+\t\t\t$mediawiki->edit({\n+\t\t\t\taction => 'upload',\n+\t\t\t\tfilename => $complete_file_name,\n+\t\t\t\tcomment => $summary,\n+\t\t\t\tfile => [undef,\n+\t\t\t\t\t $complete_file_name,\n+\t\t\t\t\t Content => $content],\n+\t\t\t\tignorewarnings => 1,\n+\t\t\t}, {\n+\t\t\t\tskip_encoding => 1\n+\t\t\t} ) || die $mediawiki->{error}->{code} . ':'\n+\t\t\t\t . $mediawiki->{error}->{details};\n+\t\t\tmy $last_file_page = $mediawiki->get_page({title => $path});\n+\t\t\t$newrevid = $last_file_page->{revid};\n+\t\t\tprint STDERR \"Pushed file: $new_sha1 - $complete_file_name.\\n\";\n+\t\t} else {\n+\t\t\tprint STDERR \"Empty file $complete_file_name not pushed.\\n\";\n+\t\t}\n+\t}\n+\treturn $newrevid;\n+}\n+\n sub mw_push_file {\n \tmy $diff_info = shift;\n \t# $diff_info contains a string in this format:\n@@ -717,7 +778,8 @@ sub mw_push_file {\n \tmy $summary = shift;\n \t# MediaWiki revision number. Keep the previous one by default,\n \t# in case there's no edit to perform.\n-\tmy $newrevid = shift;\n+\tmy $oldrevid = shift;\n+\tmy $newrevid;\n \n \tmy $new_sha1 = $diff_info_split[3];\n \tmy $old_sha1 = $diff_info_split[2];\n@@ -725,9 +787,11 @@ sub mw_push_file {\n \tmy $page_deleted = ($new_sha1 eq NULL_SHA1);\n \t$complete_file_name = mediawiki_clean_filename($complete_file_name);\n \n-\tif (substr($complete_file_name,-3) eq \".mw\") {\n-\t\tmy $title = substr($complete_file_name,0,-3);\n-\n+\tmy ($title, $extension) = $complete_file_name =~ /^(.*)\\.([^\\.]*)$/;\n+\tif (!defined($extension)) {\n+\t\t$extension = \"\";\n+\t}\n+\tif ($extension eq \"mw\") {\n \t\tmy $file_content;\n \t\tif ($page_deleted) {\n \t\t\t# Deleting a page usually requires\n@@ -745,7 +809,7 @@ sub mw_push_file {\n \t\t\taction => 'edit',\n \t\t\tsummary => $summary,\n \t\t\ttitle => $title,\n-\t\t\tbasetimestamp => $basetimestamps{$newrevid},\n+\t\t\tbasetimestamp => $basetimestamps{$oldrevid},\n \t\t\ttext => mediawiki_clean($file_content, $page_created),\n \t\t\t\t }, {\n \t\t\t\t\t skip_encoding => 1 # Helps with names with accentuated characters\n@@ -757,7 +821,7 @@ sub mw_push_file {\n \t\t\t\t $mediawiki->{error}->{code} .\n \t\t\t\t ' from mediwiki: ' . $mediawiki->{error}->{details} .\n \t\t\t\t \".\\n\";\n-\t\t\t\treturn ($newrevid, \"non-fast-forward\");\n+\t\t\t\treturn ($oldrevid, \"non-fast-forward\");\n \t\t\t} else {\n \t\t\t\t# Other errors. Shouldn't happen => just die()\n \t\t\t\tdie 'Fatal: Error ' .\n@@ -768,8 +832,11 @@ sub mw_push_file {\n \t\t$newrevid = $result->{edit}->{newrevid};\n \t\tprint STDERR \"Pushed file: $new_sha1 - $title\\n\";\n \t} else {\n-\t\tprint STDERR \"$complete_file_name not a mediawiki file (Not pushable on this version of git-remote-mediawiki).\\n\"\n+\t\t$newrevid = mw_upload_file($complete_file_name, $new_sha1,\n+\t\t\t\t\t $extension, $page_deleted,\n+\t\t\t\t\t $summary);\n \t}\n+\t$newrevid = ($newrevid or $oldrevid);\n \treturn ($newrevid, \"ok\");\n }\n \n@@ -906,3 +973,18 @@ sub mw_push_revision {\n \tprint STDOUT \"ok $remote\\n\";\n \treturn 1;\n }\n+\n+sub get_allowed_file_extensions {\n+\tmw_connect_maybe();\n+\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tmeta => 'siteinfo',\n+\t\tsiprop => 'fileextensions'\n+\t\t};\n+\tmy $result = $mediawiki->api($query);\n+\tmy @file_extensions= map $_->{ext},@{$result->{query}->{fileextensions}};\n+\tmy %hashFile = map {$_ => 1}@file_extensions;\n+\n+\treturn %hashFile;\n+}\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194287","messageId":"1340726673-25496-5-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340726673-25496-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 4/5] git-remote-mediawiki: split get_mw_pages into smaller functions","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-26T16:04:32Z","receivedAt":"2012-06-26T16:04:32Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"---\n contrib/mw-to-git/git-remote-mediawiki | 106 +++++++++++++++++++--------------\n 1 file changed, 62 insertions(+), 44 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 253b449..e175c69 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -258,6 +258,64 @@ sub mw_connect_maybe {\n \t}\n }\n \n+## Functions for listing pages on the remote wiki\n+sub get_mw_tracked_pages {\n+\tmy $pages = shift;\n+\tmy @some_pages = @tracked_pages;\n+\twhile (@some_pages) {\n+\t\tmy $last = 50;\n+\t\tif ($#some_pages < $last) {\n+\t\t\t$last = $#some_pages;\n+\t\t}\n+\t\tmy @slice = @some_pages[0..$last];\n+\t\tget_mw_first_pages(\\@slice, $pages);\n+\t\t@some_pages = @some_pages[51..$#some_pages];\n+\t}\n+}\n+\n+sub get_mw_tracked_categories {\n+\tmy $pages = shift;\n+\tforeach my $category (@tracked_categories) {\n+\t\tif (index($category, ':') < 0) {\n+\t\t\t# Mediawiki requires the Category\n+\t\t\t# prefix, but let's not force the user\n+\t\t\t# to specify it.\n+\t\t\t$category = \"Category:\" . $category;\n+\t\t}\n+\t\tmy $mw_pages = $mediawiki->list( {\n+\t\t\taction => 'query',\n+\t\t\tlist => 'categorymembers',\n+\t\t\tcmtitle => $category,\n+\t\t\tcmlimit => 'max' } )\n+\t\t\t|| die $mediawiki->{error}->{code} . ': '\n+\t\t\t\t. $mediawiki->{error}->{details};\n+\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t$pages->{$page->{title}} = $page;\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_all_pages {\n+\tmy $pages = shift;\n+\t# No user-provided list, get the list of pages from the API.\n+\tmy $mw_pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\taplimit => 'max'\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not get the list of wiki pages.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\tforeach my $page (@{$mw_pages}) {\n+\t\t$pages->{$page->{title}} = $page;\n+\t}\n+}\n+\n+# queries the wiki for a set of pages. Meant to be used within a loop\n+# querying the wiki for slices of page list.\n sub get_mw_first_pages {\n \tmy $some_pages = shift;\n \tmy @some_pages = @{$some_pages};\n@@ -286,6 +344,7 @@ sub get_mw_first_pages {\n \t}\n }\n \n+# Get the list of pages to be fetched according to configuration.\n sub get_mw_pages {\n \tmw_connect_maybe();\n \n@@ -295,55 +354,14 @@ sub get_mw_pages {\n \t\t$user_defined = 1;\n \t\t# The user provided a list of pages titles, but we\n \t\t# still need to query the API to get the page IDs.\n-\n-\t\tmy @some_pages = @tracked_pages;\n-\t\twhile (@some_pages) {\n-\t\t\tmy $last = 50;\n-\t\t\tif ($#some_pages < $last) {\n-\t\t\t\t$last = $#some_pages;\n-\t\t\t}\n-\t\t\tmy @slice = @some_pages[0..$last];\n-\t\t\tget_mw_first_pages(\\@slice, \\%pages);\n-\t\t\t@some_pages = @some_pages[51..$#some_pages];\n-\t\t}\n+\t\tget_mw_tracked_pages(\\%pages);\n \t}\n \tif (@tracked_categories) {\n \t\t$user_defined = 1;\n-\t\tforeach my $category (@tracked_categories) {\n-\t\t\tif (index($category, ':') < 0) {\n-\t\t\t\t# Mediawiki requires the Category\n-\t\t\t\t# prefix, but let's not force the user\n-\t\t\t\t# to specify it.\n-\t\t\t\t$category = \"Category:\" . $category;\n-\t\t\t}\n-\t\t\tmy $mw_pages = $mediawiki->list( {\n-\t\t\t\taction => 'query',\n-\t\t\t\tlist => 'categorymembers',\n-\t\t\t\tcmtitle => $category,\n-\t\t\t\tcmlimit => 'max' } )\n-\t\t\t || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};\n-\t\t\tforeach my $page (@{$mw_pages}) {\n-\t\t\t\t$pages{$page->{title}} = $page;\n-\t\t\t}\n-\t\t}\n+\t\tget_mw_tracked_categories(\\%pages);\n \t}\n \tif (!$user_defined) {\n-\t\t# No user-provided list, get the list of pages from\n-\t\t# the API.\n-\t\tmy $mw_pages = $mediawiki->list({\n-\t\t\taction => 'query',\n-\t\t\tlist => 'allpages',\n-\t\t\taplimit => 500,\n-\t\t});\n-\t\tif (!defined($mw_pages)) {\n-\t\t\tprint STDERR \"fatal: could not get the list of wiki pages.\\n\";\n-\t\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n-\t\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n-\t\t\texit 1;\n-\t\t}\n-\t\tforeach my $page (@{$mw_pages}) {\n-\t\t\t$pages{$page->{title}} = $page;\n-\t\t}\n+\t\tget_mw_all_pages(\\%pages);\n \t}\n \treturn values(%pages);\n }\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194286","messageId":"1340726673-25496-6-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340726673-25496-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 5/5] git-remote-mediawiki: import \"File:\" attachments","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-26T16:04:33Z","receivedAt":"2012-06-26T16:04:33Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: Pavel Volek \n\nAdd the symmetrical feature to the \"File:\" export support in the previous\npatch. Download files from the wiki as needed, and feed them into the\nfast-import stream. Import both the file itself, and the corresponding\ndescription page.\n\nSigned-off-by: Pavel Volek \nSigned-off-by: NGUYEN Kim Thuat \nSigned-off-by: ROUCHER IGLESIAS Javier \nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 244 +++++++++++++++++++++++++++++++--\n 1 file changed, 236 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex e175c69..93c9135 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -13,9 +13,6 @@\n #\n # Known limitations:\n #\n-# - Only wiki pages are managed, no support for [[File:...]]\n-# attachments.\n-#\n # - Poor performance in the best case: it takes forever to check\n # whether we're up-to-date (on fetch or push) or to fetch a few\n # revisions from a large wiki, because we use exclusively a\n@@ -36,6 +33,7 @@\n use strict;\n use MediaWiki::API;\n use DateTime::Format::ISO8601;\n+use FileHandle;\n \n # By default, use UTF-8 to communicate with Git and the user\n binmode STDERR, \":utf8\";\n@@ -72,6 +70,9 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".categories\"));\n chomp(@tracked_categories);\n \n+# Import media files too.\n+my $import_media = run_git(\"config --get --bool remote.\". $remotename .\".mediaimport\");\n+\n my $wiki_login = run_git(\"config --get remote.\". $remotename .\".mwLogin\");\n # TODO: ideally, this should be able to read from keyboard, but we're\n # inside a remote helper, so our stdin is connect to git, not to a\n@@ -87,6 +88,9 @@ my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shall\n chomp($shallow_import);\n $shallow_import = ($shallow_import eq \"true\");\n \n+# Cache for MediaWiki namespace ids.\n+my %namespace_id;\n+\n # Dumb push: don't update notes and mediawiki ref to reflect the last push.\n #\n # Configurable with mediawiki.dumbPush, or per-remote with\n@@ -363,6 +367,13 @@ sub get_mw_pages {\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n+\tif ($import_media) {\n+\t\tif ($user_defined) {\n+\t\t\tget_linked_mediafiles(\\%pages);\n+\t\t} else {\n+\t\t\tget_all_mediafiles(\\%pages);\n+\t\t}\n+\t}\n \treturn values(%pages);\n }\n \n@@ -379,6 +390,152 @@ sub run_git {\n }\n \n \n+sub get_all_mediafiles {\n+\tmy $pages = shift;\n+\t# Attach list of all pages for media files from the API,\n+\t# they are in a different namespace, only one namespace\n+\t# can be queried at the same moment\n+\tmy $mw_pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\tapnamespace => get_mw_namespace_id(\"File\"),\n+\t\taplimit => 'max'\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not get the list of pages for media files.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\tforeach my $page (@{$mw_pages}) {\n+\t\t$pages->{$page->{title}} = $page;\n+\t}\n+}\n+\n+sub get_linked_mediafiles {\n+\tmy $pages = shift;\n+\tmy @titles = map $_->{title}, values(%{$pages});\n+\n+\t# The query is split in small batches because of the MW API limit of\n+\t# the number of links to be returned (500 links max).\n+\tmy $batch = 10;\n+\twhile (@titles) {\n+\t\tif ($#titles < $batch) {\n+\t\t\t$batch = $#titles;\n+\t\t}\n+\t\tmy @slice = @titles[0..$batch];\n+\n+\t\t# pattern 'page1|page2|...' required by the API\n+\t\tmy $mw_titles = join('|', @slice);\n+\n+\t\t# Media files could be included or linked from\n+\t\t# a page, get all related\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'links|images',\n+\t\t\ttitles => $mw_titles,\n+\t\t\tplnamespace => get_mw_namespace_id(\"File\"),\n+\t\t\tpllimit => 'max'\n+\t\t};\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\twhile (my ($id, $page) = each(%{$result->{query}->{pages}})) {\n+\t\t\tmy @titles;\n+\t\t\tif (defined($page->{links})) {\n+\t\t\t\tmy @link_titles = map $_->{title}, @{$page->{links}};\n+\t\t\t\tpush(@titles, @link_titles);\n+\t\t\t}\n+\t\t\tif (defined($page->{images})) {\n+\t\t\t\tmy @image_titles = map $_->{title}, @{$page->{images}};\n+\t\t\t\tpush(@titles, @image_titles);\n+\t\t\t}\n+\t\t\tif (@titles) {\n+\t\t\t\tget_mw_first_pages(\\@titles, \\%{$pages});\n+\t\t\t}\n+\t\t}\n+\n+\t\t@titles = @titles[($batch+1)..$#titles];\n+\t}\n+}\n+\n+sub get_mw_mediafile_for_page_revision {\n+\t# Name of the file on Wiki, with the prefix.\n+\tmy $mw_filename = shift;\n+\tmy $timestamp = shift;\n+\tmy %mediafile;\n+\n+\t# Search if on MediaWiki exists a media file with given\n+\t# timestamp. In that case download the file.\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tprop => 'imageinfo',\n+\t\ttitles => $mw_filename,\n+\t\tiistart => $timestamp,\n+\t\tiiend => $timestamp,\n+\t\tiiprop => 'timestamp|archivename|url',\n+\t\tiilimit => 1\n+\t};\n+\tmy $result = $mediawiki->api($query);\n+\n+\tmy ($fileid, $file) = each ( %{$result->{query}->{pages}} );\n+\t# If not defined it means there is no revision of the file for\n+\t# given timestamp.\n+\tif (defined($file->{imageinfo})) {\n+\t\t# Get real name of media file.\n+\t\tmy $filename;\n+\t\tif (index($mw_filename, 'File:') == 0) {\n+\t\t\t$filename = substr $mw_filename, 5;\n+\t\t} else {\n+\t\t\t$filename = substr $mw_filename, 6;\n+\t\t}\n+\t\t$mediafile{title} = $filename;\n+\n+\t\tmy $fileinfo = pop(@{$file->{imageinfo}});\n+\t\t$mediafile{timestamp} = $fileinfo->{timestamp};\n+\t\t# If this is an old version of the file, the file has to be\n+\t\t# obtained from the archive. Otherwise it can be downloaded\n+\t\t# by MediaWiki API download() function.\n+\t\tif (defined($fileinfo->{archivename})) {\n+\t\t\t$mediafile{content} = download_mw_mediafile_from_archive($fileinfo->{url});\n+\t\t} else {\n+\t\t\t$mediafile{content} = download_mw_mediafile($mw_filename);\n+\t\t}\n+\t}\n+\treturn %mediafile;\n+}\n+\n+sub download_mw_mediafile_from_archive {\n+\tmy $url = shift;\n+\tmy $file;\n+\n+\tmy $ua = LWP::UserAgent->new;\n+\tmy $response = $ua->get($url);\n+\tif ($response->code) {\n+\t\t$file = $response->decoded_content;\n+\t} else {\n+\t\tprint STDERR \"Error downloading a file from archive.\\n\";\n+\t}\n+\n+\treturn $file;\n+}\n+\n+sub download_mw_mediafile {\n+\tmy $filename = shift;\n+\n+\t$mediawiki->{config}->{files_url} = $url;\n+\n+\tmy $file_content = $mediawiki->download( { title => $filename } );\n+\tif (!defined($file_content)) {\n+\t\tprint STDERR \"\\tFile \\'$filename\\' could not be downloaded.\\n\";\n+\t\texit 1;\n+\t} elsif ($file_content eq \"\") {\n+\t\tprint STDERR \"\\tFile \\'$filename\\' does not exist on the wiki.\\n\";\n+\t\texit 1;\n+\t} else {\n+\t\treturn $file_content;\n+\t}\n+}\n+\n sub get_last_local_revision {\n \t# Get note regarding last mediawiki revision\n \tmy $note = run_git(\"notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n@@ -569,6 +726,11 @@ sub import_file_revision {\n \tmy %commit = %{$commit};\n \tmy $full_import = shift;\n \tmy $n = shift;\n+\tmy $mediafile = shift;\n+\tmy %mediafile;\n+\tif ($mediafile) {\n+\t\t%mediafile = %{$mediafile};\n+\t}\n \n \tmy $title = $commit{title};\n \tmy $comment = $commit{comment};\n@@ -588,6 +750,10 @@ sub import_file_revision {\n \tif ($content ne DELETED_CONTENT) {\n \t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n \t\tliteral_data($content);\n+\t\tif (%mediafile) {\n+\t\t\tprint STDOUT \"M 644 inline $mediafile{title}\\n\";\n+\t\t\tliteral_data($mediafile{content});\n+\t\t}\n \t\tprint STDOUT \"\\n\\n\";\n \t} else {\n \t\tprint STDOUT \"D $title.mw\\n\";\n@@ -683,12 +849,11 @@ sub mw_import_ref {\n \n \t\t$n++;\n \n+\t\tmy $page_title = $result->{query}->{pages}->{$pagerevid->{pageid}}->{title};\n \t\tmy %commit;\n \t\t$commit{author} = $rev->{user} || 'Anonymous';\n \t\t$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';\n-\t\t$commit{title} = mediawiki_smudge_filename(\n-\t\t\t$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}\n-\t\t );\n+\t\t$commit{title} = mediawiki_smudge_filename($page_title);\n \t\t$commit{mw_revision} = $pagerevid->{revid};\n \t\t$commit{content} = mediawiki_smudge($rev->{'*'});\n \n@@ -699,9 +864,25 @@ sub mw_import_ref {\n \t\t}\n \t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n \n-\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\t\t# Differentiates classic pages and media files.\n+\t\tmy @prefix = split(\":\", $page_title);\n \n-\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t\tmy %mediafile;\n+\t\tif ($prefix[0] eq \"File\" || $prefix[0] eq \"Image\") {\n+\t\t\t# The name of the file is the same as the media page.\n+\t\t\tmy $filename = $page_title;\n+\t\t\t%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});\n+\t\t}\n+\t\t# If this is a revision of the media page for new version\n+\t\t# of a file do one common commit for both file and media page.\n+\t\t# Else do commit only for that page.\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\t\tif (%mediafile) {\n+\t\t\tprint STDERR \"\\tDownloading file $mediafile{title}, version $mediafile{timestamp}\\n\";\n+\t\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n, \\%mediafile);\n+\t\t} else {\n+\t\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t\t}\n \t}\n \n \tif ($fetch_from == 1 && $n == 0) {\n@@ -1006,3 +1187,50 @@ sub get_allowed_file_extensions {\n \n \treturn %hashFile;\n }\n+\n+# Return MediaWiki id for a canonical namespace name.\n+# Ex.: \"File\", \"Project\".\n+# Looks for the namespace id in the local configuration\n+# variables, if it is not found asks MW API.\n+sub get_mw_namespace_id {\n+\tmw_connect_maybe();\n+\tmy $name = shift;\n+\n+\tif (!exists $namespace_id{$name}) {\n+\t\t# Look at configuration file, if the record for that namespace is\n+\t\t# already stored. Namespaces are stored in form:\n+\t\t# \"Name_of_namespace:Id_namespace\", ex.: \"File:6\".\n+\t\tmy @temp = split(/[ \\n]/, run_git(\"config --get-all remote.\"\n+\t\t\t\t\t\t. $remotename .\".namespaces\"));\n+\t\tchomp(@temp);\n+\t\tforeach my $ns (@temp) {\n+\t\t\tmy ($n, $s) = split(/:/, $ns);\n+\t\t\t$namespace_id{$n} = $s;\n+\t\t}\n+\t}\n+\n+\tif (!exists $namespace_id{$name}) {\n+\t\t# NS not found => get namespace id from MW and store it in\n+\t # configuration file.\n+\t my $query = {\n+\t action => 'query',\n+\t meta => 'siteinfo',\n+\t siprop => 'namespaces'\n+\t };\n+\t my $result = $mediawiki->api($query);\n+\n+\t while (my ($id, $ns) = each(%{$result->{query}->{namespaces}})) {\n+\t if (defined($ns->{canonical}) && ($ns->{canonical} eq $name)) {\n+\t run_git(\"config --add remote.\". $remotename\n+\t\t\t\t\t.\".namespaces \". $name .\":\". $ns->{id});\n+\t\t\t\t$namespace_id{$name} = $ns->{id};\n+\t }\n+\t }\n+\t}\n+\n+\tif (exists $namespace_id{$name}) {\n+\t\treturn $namespace_id{$name};\n+\t} else {\n+\t\tdie \"No such namespace $name on MediaWiki.\";\n+\t}\n+}\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194293","messageId":"7v8vfac6dn.fsf@alter.siamese.dyndns.org","threadId":"30905","inReplyTo":"1340726673-25496-2-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH 1/5] git-remote-mediawiki: don't compute the diff when getting commit message","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-06-26T17:47:32Z","receivedAt":"2012-06-26T17:47:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> ---\n\nNot signed off yet?\n\n> contrib/mw-to-git/git-remote-mediawiki | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\n> index a34f53f..a8e6287 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki\n> +++ b/contrib/mw-to-git/git-remote-mediawiki\n> @@ -874,8 +874,7 @@ sub mw_push_revision {\n\nCurious. The hunk replaces an old line with a new one, but somehow\nclaims to reduce 8 to 7 by one???\n\n> \t\t# TODO: for now, it's just a delete+add\n> \t\tmy @diff_info_list = split(/\\0/, $diff_infos);\n> \t\t# Keep the first line of the commit message as mediawiki comment for the revision\n> -\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $sha1_commit\")))[0];\n> +\t\tmy $commit_msg = (split(/\\n/, run_git(\"log -1 --format=\\\"%s\\\" $sha1_commit\")))[0];\n> \t\tchomp($commit_msg);\n> \t\t# Push every blob\n> \t\twhile (@diff_info_list) {\n"},{"id":"194335","messageId":"vpqfw9h9mtw.fsf@bauges.imag.fr","threadId":"30905","inReplyTo":"7v8vfac6dn.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 1/5] git-remote-mediawiki: don't compute the diff when getting commit message","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-06-27T08:32:43Z","receivedAt":"2012-06-27T08:32:43Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> Matthieu Moy writes:\n>\n>> ---\n>\n> Not signed off yet?\n\nMy bad, I broke my email-sending alias.\n\n>> contrib/mw-to-git/git-remote-mediawiki | 2 +-\n>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\n>> index a34f53f..a8e6287 100755\n>> --- a/contrib/mw-to-git/git-remote-mediawiki\n>> +++ b/contrib/mw-to-git/git-remote-mediawiki\n>> @@ -874,8 +874,7 @@ sub mw_push_revision {\n>\n> Curious. The hunk replaces an old line with a new one, but somehow\n> claims to reduce 8 to 7 by one???\n\nI (mis-)tweaked the patch manually. Anyway, a better/simpler one is\ncomming ;-).\n\nSorry for the noise.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"194338","messageId":"1340788220-10084-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340726673-25496-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 0/5 v2] git-remote-mediawiki: support File: import and export","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-27T09:10:15Z","receivedAt":"2012-06-27T09:10:15Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Changes since v1:\n- Added forgotten Signed-Off-By\n- UTF-8 related fixes\n- Fix parsing of mediaimport configuration variable\n\nMatthieu Moy (3):\n git-remote-mediawiki: don't compute the diff when getting commit\n message\n git-remote-mediawiki: don't \"use encoding 'utf8';\"\n git-remote-mediawiki: split get_mw_pages into smaller functions\n\nNGUYEN Kim Thuat (1):\n git-remote-mediawiki: send \"File:\" attachments to a remote wiki\n\nPavel Volek (1):\n git-remote-mediawiki: import \"File:\" attachments\n\n contrib/mw-to-git/git-remote-mediawiki | 466 ++++++++++++++++++++++++++++-----\n 1 file changed, 402 insertions(+), 64 deletions(-)\n\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194341","messageId":"1340788220-10084-2-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340788220-10084-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 1/5] git-remote-mediawiki: don't compute the diff when getting commit message","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-27T09:10:16Z","receivedAt":"2012-06-27T09:10:16Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"While we're there, simplify the code a bit: since log --format=%s anyway\nshows the subject line as a single line, no need to split to take the\nfirst line.\n\nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex a34f53f..781dfe2 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -873,8 +873,8 @@ sub mw_push_revision {\n \t\t# TODO: we could detect rename, and encode them with a #redirect on the wiki.\n \t\t# TODO: for now, it's just a delete+add\n \t\tmy @diff_info_list = split(/\\0/, $diff_infos);\n-\t\t# Keep the first line of the commit message as mediawiki comment for the revision\n-\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $sha1_commit\")))[0];\n+\t\t# Keep the subject line of the commit message as mediawiki comment for the revision\n+\t\tmy $commit_msg = run_git(\"log --no-walk --format=\\\"%s\\\" $sha1_commit\");\n \t\tchomp($commit_msg);\n \t\t# Push every blob\n \t\twhile (@diff_info_list) {\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194340","messageId":"1340788220-10084-3-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340788220-10084-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 2/5] git-remote-mediawiki: don't \"use encoding 'utf8';\"","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-27T09:10:17Z","receivedAt":"2012-06-27T09:10:17Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"The use of this statement is generally discouraged, and is too intrusive\nfor us: it forces the HTTP requests made by the API to contain only valid\nUTF-8 characters. This would break the upload of binary files.\n\nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 781dfe2..f2e841e 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -36,11 +36,10 @@\n use strict;\n use MediaWiki::API;\n use DateTime::Format::ISO8601;\n-use encoding 'utf8';\n \n-# use encoding 'utf8' doesn't change STDERROR\n-# but we're going to output UTF-8 filenames to STDERR\n+# By default, use UTF-8 to communicate with Git and the user\n binmode STDERR, \":utf8\";\n+binmode STDOUT, \":utf8\";\n \n use URI::Escape;\n use IPC::Open2;\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194336","messageId":"1340788220-10084-4-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340788220-10084-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 3/5] git-remote-mediawiki: send \"File:\" attachments to a remote wiki","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-27T09:10:18Z","receivedAt":"2012-06-27T09:10:18Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: NGUYEN Kim Thuat \n\nThe current version of the git-remote-mediawiki supports only import and\nexport of plain wiki pages. This patch adds the functionality to export\nfile attachments (i.e. the content of the File: MediaWiki namespace),\nwhich are also exposed by MediaWiki API.\n\nThis requires a recent version of MediaWiki::API (Version 0.37 works.\nVersion 0.34 doesn't).\n\nSigned-off-by: Pavel Volek \nSigned-off-by: NGUYEN Kim Thuat \nSigned-off-by: ROUCHER IGLESIAS Javier \nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 98 +++++++++++++++++++++++++++++++---\n 1 file changed, 90 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex f2e841e..3405772 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -348,8 +348,12 @@ sub get_mw_pages {\n \treturn values(%pages);\n }\n \n+# usage: $out = run_git(\"command args\");\n+# $out = run_git(\"command args\", \"raw\"); # don't interpret output as UTF-8.\n sub run_git {\n-\topen(my $git, \"-|:encoding(UTF-8)\", \"git \" . $_[0]);\n+\tmy $args = shift;\n+\tmy $encoding = (shift || \"encoding(UTF-8)\");\n+\topen(my $git, \"-|:$encoding\", \"git \" . $args);\n \tmy $res = do { local $/; <$git> };\n \tclose($git);\n \n@@ -705,6 +709,63 @@ sub error_non_fast_forward {\n \treturn 0;\n }\n \n+sub mw_upload_file {\n+\tmy $complete_file_name = shift;\n+\tmy $new_sha1 = shift;\n+\tmy $extension = shift;\n+\tmy $file_deleted = shift;\n+\tmy $summary = shift;\n+\tmy $newrevid;\n+\tmy $path = \"File:\" . $complete_file_name;\n+\tmy %hashFiles = get_allowed_file_extensions();\n+\tif (!exists($hashFiles{$extension})) {\n+\t\tprint STDERR \"$complete_file_name is not a permitted file on this wiki.\\n\";\n+\t\tprint STDERR \"Check the configuration of file uploads in your mediawiki.\\n\";\n+\t\treturn $newrevid;\n+\t}\n+\t# Deleting and uploading a file requires a priviledged user\n+\tif ($file_deleted) {\n+\t\tmw_connect_maybe();\n+\t\tmy $query = {\n+\t\t\taction => 'delete',\n+\t\t\ttitle => $path,\n+\t\t\treason => $summary\n+\t\t};\n+\t\tif (!$mediawiki->edit($query)) {\n+\t\t\tprint STDERR \"Failed to delete file on remote wiki\\n\";\n+\t\t\tprint STDERR \"Check your permissions on the remote site. Error code:\\n\";\n+\t\t\tprint STDERR $mediawiki->{error}->{code} . ':' . $mediawiki->{error}->{details};\n+\t\t\texit 1;\n+\t\t}\n+\t} else {\n+\t\t# Don't let perl try to interpret file content as UTF-8 => use \"raw\"\n+\t\tmy $content = run_git(\"cat-file blob $new_sha1\", \"raw\");\n+\t\tif ($content ne \"\") {\n+\t\t\tmw_connect_maybe();\n+\t\t\t$mediawiki->{config}->{upload_url} =\n+\t\t\t\t\"$url/index.php/Special:Upload\";\n+\t\t\t$mediawiki->edit({\n+\t\t\t\taction => 'upload',\n+\t\t\t\tfilename => $complete_file_name,\n+\t\t\t\tcomment => $summary,\n+\t\t\t\tfile => [undef,\n+\t\t\t\t\t $complete_file_name,\n+\t\t\t\t\t Content => $content],\n+\t\t\t\tignorewarnings => 1,\n+\t\t\t}, {\n+\t\t\t\tskip_encoding => 1\n+\t\t\t} ) || die $mediawiki->{error}->{code} . ':'\n+\t\t\t\t . $mediawiki->{error}->{details};\n+\t\t\tmy $last_file_page = $mediawiki->get_page({title => $path});\n+\t\t\t$newrevid = $last_file_page->{revid};\n+\t\t\tprint STDERR \"Pushed file: $new_sha1 - $complete_file_name.\\n\";\n+\t\t} else {\n+\t\t\tprint STDERR \"Empty file $complete_file_name not pushed.\\n\";\n+\t\t}\n+\t}\n+\treturn $newrevid;\n+}\n+\n sub mw_push_file {\n \tmy $diff_info = shift;\n \t# $diff_info contains a string in this format:\n@@ -717,7 +778,8 @@ sub mw_push_file {\n \tmy $summary = shift;\n \t# MediaWiki revision number. Keep the previous one by default,\n \t# in case there's no edit to perform.\n-\tmy $newrevid = shift;\n+\tmy $oldrevid = shift;\n+\tmy $newrevid;\n \n \tmy $new_sha1 = $diff_info_split[3];\n \tmy $old_sha1 = $diff_info_split[2];\n@@ -725,9 +787,11 @@ sub mw_push_file {\n \tmy $page_deleted = ($new_sha1 eq NULL_SHA1);\n \t$complete_file_name = mediawiki_clean_filename($complete_file_name);\n \n-\tif (substr($complete_file_name,-3) eq \".mw\") {\n-\t\tmy $title = substr($complete_file_name,0,-3);\n-\n+\tmy ($title, $extension) = $complete_file_name =~ /^(.*)\\.([^\\.]*)$/;\n+\tif (!defined($extension)) {\n+\t\t$extension = \"\";\n+\t}\n+\tif ($extension eq \"mw\") {\n \t\tmy $file_content;\n \t\tif ($page_deleted) {\n \t\t\t# Deleting a page usually requires\n@@ -745,7 +809,7 @@ sub mw_push_file {\n \t\t\taction => 'edit',\n \t\t\tsummary => $summary,\n \t\t\ttitle => $title,\n-\t\t\tbasetimestamp => $basetimestamps{$newrevid},\n+\t\t\tbasetimestamp => $basetimestamps{$oldrevid},\n \t\t\ttext => mediawiki_clean($file_content, $page_created),\n \t\t\t\t }, {\n \t\t\t\t\t skip_encoding => 1 # Helps with names with accentuated characters\n@@ -757,7 +821,7 @@ sub mw_push_file {\n \t\t\t\t $mediawiki->{error}->{code} .\n \t\t\t\t ' from mediwiki: ' . $mediawiki->{error}->{details} .\n \t\t\t\t \".\\n\";\n-\t\t\t\treturn ($newrevid, \"non-fast-forward\");\n+\t\t\t\treturn ($oldrevid, \"non-fast-forward\");\n \t\t\t} else {\n \t\t\t\t# Other errors. Shouldn't happen => just die()\n \t\t\t\tdie 'Fatal: Error ' .\n@@ -768,8 +832,11 @@ sub mw_push_file {\n \t\t$newrevid = $result->{edit}->{newrevid};\n \t\tprint STDERR \"Pushed file: $new_sha1 - $title\\n\";\n \t} else {\n-\t\tprint STDERR \"$complete_file_name not a mediawiki file (Not pushable on this version of git-remote-mediawiki).\\n\"\n+\t\t$newrevid = mw_upload_file($complete_file_name, $new_sha1,\n+\t\t\t\t\t $extension, $page_deleted,\n+\t\t\t\t\t $summary);\n \t}\n+\t$newrevid = ($newrevid or $oldrevid);\n \treturn ($newrevid, \"ok\");\n }\n \n@@ -906,3 +973,18 @@ sub mw_push_revision {\n \tprint STDOUT \"ok $remote\\n\";\n \treturn 1;\n }\n+\n+sub get_allowed_file_extensions {\n+\tmw_connect_maybe();\n+\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tmeta => 'siteinfo',\n+\t\tsiprop => 'fileextensions'\n+\t\t};\n+\tmy $result = $mediawiki->api($query);\n+\tmy @file_extensions= map $_->{ext},@{$result->{query}->{fileextensions}};\n+\tmy %hashFile = map {$_ => 1}@file_extensions;\n+\n+\treturn %hashFile;\n+}\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194339","messageId":"1340788220-10084-5-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340788220-10084-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 4/5] git-remote-mediawiki: split get_mw_pages into smaller functions","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-27T09:10:19Z","receivedAt":"2012-06-27T09:10:19Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"\nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 106 +++++++++++++++++++--------------\n 1 file changed, 62 insertions(+), 44 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 3405772..76d8824 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -258,6 +258,64 @@ sub mw_connect_maybe {\n \t}\n }\n \n+## Functions for listing pages on the remote wiki\n+sub get_mw_tracked_pages {\n+\tmy $pages = shift;\n+\tmy @some_pages = @tracked_pages;\n+\twhile (@some_pages) {\n+\t\tmy $last = 50;\n+\t\tif ($#some_pages < $last) {\n+\t\t\t$last = $#some_pages;\n+\t\t}\n+\t\tmy @slice = @some_pages[0..$last];\n+\t\tget_mw_first_pages(\\@slice, $pages);\n+\t\t@some_pages = @some_pages[51..$#some_pages];\n+\t}\n+}\n+\n+sub get_mw_tracked_categories {\n+\tmy $pages = shift;\n+\tforeach my $category (@tracked_categories) {\n+\t\tif (index($category, ':') < 0) {\n+\t\t\t# Mediawiki requires the Category\n+\t\t\t# prefix, but let's not force the user\n+\t\t\t# to specify it.\n+\t\t\t$category = \"Category:\" . $category;\n+\t\t}\n+\t\tmy $mw_pages = $mediawiki->list( {\n+\t\t\taction => 'query',\n+\t\t\tlist => 'categorymembers',\n+\t\t\tcmtitle => $category,\n+\t\t\tcmlimit => 'max' } )\n+\t\t\t|| die $mediawiki->{error}->{code} . ': '\n+\t\t\t\t. $mediawiki->{error}->{details};\n+\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t$pages->{$page->{title}} = $page;\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_all_pages {\n+\tmy $pages = shift;\n+\t# No user-provided list, get the list of pages from the API.\n+\tmy $mw_pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\taplimit => 'max'\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not get the list of wiki pages.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\tforeach my $page (@{$mw_pages}) {\n+\t\t$pages->{$page->{title}} = $page;\n+\t}\n+}\n+\n+# queries the wiki for a set of pages. Meant to be used within a loop\n+# querying the wiki for slices of page list.\n sub get_mw_first_pages {\n \tmy $some_pages = shift;\n \tmy @some_pages = @{$some_pages};\n@@ -286,6 +344,7 @@ sub get_mw_first_pages {\n \t}\n }\n \n+# Get the list of pages to be fetched according to configuration.\n sub get_mw_pages {\n \tmw_connect_maybe();\n \n@@ -295,55 +354,14 @@ sub get_mw_pages {\n \t\t$user_defined = 1;\n \t\t# The user provided a list of pages titles, but we\n \t\t# still need to query the API to get the page IDs.\n-\n-\t\tmy @some_pages = @tracked_pages;\n-\t\twhile (@some_pages) {\n-\t\t\tmy $last = 50;\n-\t\t\tif ($#some_pages < $last) {\n-\t\t\t\t$last = $#some_pages;\n-\t\t\t}\n-\t\t\tmy @slice = @some_pages[0..$last];\n-\t\t\tget_mw_first_pages(\\@slice, \\%pages);\n-\t\t\t@some_pages = @some_pages[51..$#some_pages];\n-\t\t}\n+\t\tget_mw_tracked_pages(\\%pages);\n \t}\n \tif (@tracked_categories) {\n \t\t$user_defined = 1;\n-\t\tforeach my $category (@tracked_categories) {\n-\t\t\tif (index($category, ':') < 0) {\n-\t\t\t\t# Mediawiki requires the Category\n-\t\t\t\t# prefix, but let's not force the user\n-\t\t\t\t# to specify it.\n-\t\t\t\t$category = \"Category:\" . $category;\n-\t\t\t}\n-\t\t\tmy $mw_pages = $mediawiki->list( {\n-\t\t\t\taction => 'query',\n-\t\t\t\tlist => 'categorymembers',\n-\t\t\t\tcmtitle => $category,\n-\t\t\t\tcmlimit => 'max' } )\n-\t\t\t || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};\n-\t\t\tforeach my $page (@{$mw_pages}) {\n-\t\t\t\t$pages{$page->{title}} = $page;\n-\t\t\t}\n-\t\t}\n+\t\tget_mw_tracked_categories(\\%pages);\n \t}\n \tif (!$user_defined) {\n-\t\t# No user-provided list, get the list of pages from\n-\t\t# the API.\n-\t\tmy $mw_pages = $mediawiki->list({\n-\t\t\taction => 'query',\n-\t\t\tlist => 'allpages',\n-\t\t\taplimit => 500,\n-\t\t});\n-\t\tif (!defined($mw_pages)) {\n-\t\t\tprint STDERR \"fatal: could not get the list of wiki pages.\\n\";\n-\t\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n-\t\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n-\t\t\texit 1;\n-\t\t}\n-\t\tforeach my $page (@{$mw_pages}) {\n-\t\t\t$pages{$page->{title}} = $page;\n-\t\t}\n+\t\tget_mw_all_pages(\\%pages);\n \t}\n \treturn values(%pages);\n }\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194337","messageId":"1340788220-10084-6-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340788220-10084-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 5/5] git-remote-mediawiki: import \"File:\" attachments","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-27T09:10:20Z","receivedAt":"2012-06-27T09:10:20Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: Pavel Volek \n\nAdd the symmetrical feature to the \"File:\" export support in the previous\npatch. Download files from the wiki as needed, and feed them into the\nfast-import stream. Import both the file itself, and the corresponding\ndescription page.\n\nSigned-off-by: Pavel Volek \nSigned-off-by: NGUYEN Kim Thuat \nSigned-off-by: ROUCHER IGLESIAS Javier \nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 255 +++++++++++++++++++++++++++++++--\n 1 file changed, 247 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 76d8824..15a55ba 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -13,9 +13,6 @@\n #\n # Known limitations:\n #\n-# - Only wiki pages are managed, no support for [[File:...]]\n-# attachments.\n-#\n # - Poor performance in the best case: it takes forever to check\n # whether we're up-to-date (on fetch or push) or to fetch a few\n # revisions from a large wiki, because we use exclusively a\n@@ -36,6 +33,7 @@\n use strict;\n use MediaWiki::API;\n use DateTime::Format::ISO8601;\n+use FileHandle;\n \n # By default, use UTF-8 to communicate with Git and the user\n binmode STDERR, \":utf8\";\n@@ -72,6 +70,11 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".categories\"));\n chomp(@tracked_categories);\n \n+# Import media files too.\n+my $import_media = run_git(\"config --get --bool remote.\". $remotename .\".mediaimport\");\n+chomp($import_media);\n+$import_media = ($import_media eq \"true\");\n+\n my $wiki_login = run_git(\"config --get remote.\". $remotename .\".mwLogin\");\n # TODO: ideally, this should be able to read from keyboard, but we're\n # inside a remote helper, so our stdin is connect to git, not to a\n@@ -87,6 +90,9 @@ my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shall\n chomp($shallow_import);\n $shallow_import = ($shallow_import eq \"true\");\n \n+# Cache for MediaWiki namespace ids.\n+my %namespace_id;\n+\n # Dumb push: don't update notes and mediawiki ref to reflect the last push.\n #\n # Configurable with mediawiki.dumbPush, or per-remote with\n@@ -363,6 +369,14 @@ sub get_mw_pages {\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n+\tif ($import_media) {\n+\t\tprint STDERR \"Getting media files for selected pages...\\n\";\n+\t\tif ($user_defined) {\n+\t\t\tget_linked_mediafiles(\\%pages);\n+\t\t} else {\n+\t\t\tget_all_mediafiles(\\%pages);\n+\t\t}\n+\t}\n \treturn values(%pages);\n }\n \n@@ -379,6 +393,152 @@ sub run_git {\n }\n \n \n+sub get_all_mediafiles {\n+\tmy $pages = shift;\n+\t# Attach list of all pages for media files from the API,\n+\t# they are in a different namespace, only one namespace\n+\t# can be queried at the same moment\n+\tmy $mw_pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\tapnamespace => get_mw_namespace_id(\"File\"),\n+\t\taplimit => 'max'\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not get the list of pages for media files.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\tforeach my $page (@{$mw_pages}) {\n+\t\t$pages->{$page->{title}} = $page;\n+\t}\n+}\n+\n+sub get_linked_mediafiles {\n+\tmy $pages = shift;\n+\tmy @titles = map $_->{title}, values(%{$pages});\n+\n+\t# The query is split in small batches because of the MW API limit of\n+\t# the number of links to be returned (500 links max).\n+\tmy $batch = 10;\n+\twhile (@titles) {\n+\t\tif ($#titles < $batch) {\n+\t\t\t$batch = $#titles;\n+\t\t}\n+\t\tmy @slice = @titles[0..$batch];\n+\n+\t\t# pattern 'page1|page2|...' required by the API\n+\t\tmy $mw_titles = join('|', @slice);\n+\n+\t\t# Media files could be included or linked from\n+\t\t# a page, get all related\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'links|images',\n+\t\t\ttitles => $mw_titles,\n+\t\t\tplnamespace => get_mw_namespace_id(\"File\"),\n+\t\t\tpllimit => 'max'\n+\t\t};\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\twhile (my ($id, $page) = each(%{$result->{query}->{pages}})) {\n+\t\t\tmy @titles;\n+\t\t\tif (defined($page->{links})) {\n+\t\t\t\tmy @link_titles = map $_->{title}, @{$page->{links}};\n+\t\t\t\tpush(@titles, @link_titles);\n+\t\t\t}\n+\t\t\tif (defined($page->{images})) {\n+\t\t\t\tmy @image_titles = map $_->{title}, @{$page->{images}};\n+\t\t\t\tpush(@titles, @image_titles);\n+\t\t\t}\n+\t\t\tif (@titles) {\n+\t\t\t\tget_mw_first_pages(\\@titles, \\%{$pages});\n+\t\t\t}\n+\t\t}\n+\n+\t\t@titles = @titles[($batch+1)..$#titles];\n+\t}\n+}\n+\n+sub get_mw_mediafile_for_page_revision {\n+\t# Name of the file on Wiki, with the prefix.\n+\tmy $mw_filename = shift;\n+\tmy $timestamp = shift;\n+\tmy %mediafile;\n+\n+\t# Search if on MediaWiki exists a media file with given\n+\t# timestamp. In that case download the file.\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tprop => 'imageinfo',\n+\t\ttitles => $mw_filename,\n+\t\tiistart => $timestamp,\n+\t\tiiend => $timestamp,\n+\t\tiiprop => 'timestamp|archivename|url',\n+\t\tiilimit => 1\n+\t};\n+\tmy $result = $mediawiki->api($query);\n+\n+\tmy ($fileid, $file) = each ( %{$result->{query}->{pages}} );\n+\t# If not defined it means there is no revision of the file for\n+\t# given timestamp.\n+\tif (defined($file->{imageinfo})) {\n+\t\t# Get real name of media file.\n+\t\tmy $filename;\n+\t\tif (index($mw_filename, 'File:') == 0) {\n+\t\t\t$filename = substr $mw_filename, 5;\n+\t\t} else {\n+\t\t\t$filename = substr $mw_filename, 6;\n+\t\t}\n+\t\t$mediafile{title} = $filename;\n+\n+\t\tmy $fileinfo = pop(@{$file->{imageinfo}});\n+\t\t$mediafile{timestamp} = $fileinfo->{timestamp};\n+\t\t# If this is an old version of the file, the file has to be\n+\t\t# obtained from the archive. Otherwise it can be downloaded\n+\t\t# by MediaWiki API download() function.\n+\t\tif (defined($fileinfo->{archivename})) {\n+\t\t\t$mediafile{content} = download_mw_mediafile_from_archive($fileinfo->{url});\n+\t\t} else {\n+\t\t\t$mediafile{content} = download_mw_mediafile($mw_filename);\n+\t\t}\n+\t}\n+\treturn %mediafile;\n+}\n+\n+sub download_mw_mediafile_from_archive {\n+\tmy $url = shift;\n+\tmy $file;\n+\n+\tmy $ua = LWP::UserAgent->new;\n+\tmy $response = $ua->get($url);\n+\tif ($response->code) {\n+\t\t$file = $response->decoded_content;\n+\t} else {\n+\t\tprint STDERR \"Error downloading a file from archive.\\n\";\n+\t}\n+\n+\treturn $file;\n+}\n+\n+sub download_mw_mediafile {\n+\tmy $filename = shift;\n+\n+\t$mediawiki->{config}->{files_url} = $url;\n+\n+\tmy $file_content = $mediawiki->download( { title => $filename } );\n+\tif (!defined($file_content)) {\n+\t\tprint STDERR \"\\tFile \\'$filename\\' could not be downloaded.\\n\";\n+\t\texit 1;\n+\t} elsif ($file_content eq \"\") {\n+\t\tprint STDERR \"\\tFile \\'$filename\\' does not exist on the wiki.\\n\";\n+\t\texit 1;\n+\t} else {\n+\t\treturn $file_content;\n+\t}\n+}\n+\n sub get_last_local_revision {\n \t# Get note regarding last mediawiki revision\n \tmy $note = run_git(\"notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n@@ -482,6 +642,14 @@ sub literal_data {\n \tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n }\n \n+sub literal_data_raw {\n+\t# Output possibly binary content.\n+\tmy ($content) = @_;\n+\tbinmode STDOUT, \":raw\";\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+\tbinmode STDOUT, \":utf8\";\n+}\n+\n sub mw_capabilities {\n \t# Revisions are imported to the private namespace\n \t# refs/mediawiki/$remotename/ by the helper and fetched into\n@@ -569,6 +737,11 @@ sub import_file_revision {\n \tmy %commit = %{$commit};\n \tmy $full_import = shift;\n \tmy $n = shift;\n+\tmy $mediafile = shift;\n+\tmy %mediafile;\n+\tif ($mediafile) {\n+\t\t%mediafile = %{$mediafile};\n+\t}\n \n \tmy $title = $commit{title};\n \tmy $comment = $commit{comment};\n@@ -588,6 +761,10 @@ sub import_file_revision {\n \tif ($content ne DELETED_CONTENT) {\n \t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n \t\tliteral_data($content);\n+\t\tif (%mediafile) {\n+\t\t\tprint STDOUT \"M 644 inline $mediafile{title}\\n\";\n+\t\t\tliteral_data_raw($mediafile{content});\n+\t\t}\n \t\tprint STDOUT \"\\n\\n\";\n \t} else {\n \t\tprint STDOUT \"D $title.mw\\n\";\n@@ -683,12 +860,11 @@ sub mw_import_ref {\n \n \t\t$n++;\n \n+\t\tmy $page_title = $result->{query}->{pages}->{$pagerevid->{pageid}}->{title};\n \t\tmy %commit;\n \t\t$commit{author} = $rev->{user} || 'Anonymous';\n \t\t$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';\n-\t\t$commit{title} = mediawiki_smudge_filename(\n-\t\t\t$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}\n-\t\t );\n+\t\t$commit{title} = mediawiki_smudge_filename($page_title);\n \t\t$commit{mw_revision} = $pagerevid->{revid};\n \t\t$commit{content} = mediawiki_smudge($rev->{'*'});\n \n@@ -699,9 +875,25 @@ sub mw_import_ref {\n \t\t}\n \t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n \n-\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\t\t# Differentiates classic pages and media files.\n+\t\tmy @prefix = split(\":\", $page_title);\n \n-\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t\tmy %mediafile;\n+\t\tif ($prefix[0] eq \"File\" || $prefix[0] eq \"Image\") {\n+\t\t\t# The name of the file is the same as the media page.\n+\t\t\tmy $filename = $page_title;\n+\t\t\t%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});\n+\t\t}\n+\t\t# If this is a revision of the media page for new version\n+\t\t# of a file do one common commit for both file and media page.\n+\t\t# Else do commit only for that page.\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\t\tif (%mediafile) {\n+\t\t\tprint STDERR \"\\tDownloading file $mediafile{title}, version $mediafile{timestamp}\\n\";\n+\t\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n, \\%mediafile);\n+\t\t} else {\n+\t\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t\t}\n \t}\n \n \tif ($fetch_from == 1 && $n == 0) {\n@@ -1006,3 +1198,50 @@ sub get_allowed_file_extensions {\n \n \treturn %hashFile;\n }\n+\n+# Return MediaWiki id for a canonical namespace name.\n+# Ex.: \"File\", \"Project\".\n+# Looks for the namespace id in the local configuration\n+# variables, if it is not found asks MW API.\n+sub get_mw_namespace_id {\n+\tmw_connect_maybe();\n+\tmy $name = shift;\n+\n+\tif (!exists $namespace_id{$name}) {\n+\t\t# Look at configuration file, if the record for that namespace is\n+\t\t# already stored. Namespaces are stored in form:\n+\t\t# \"Name_of_namespace:Id_namespace\", ex.: \"File:6\".\n+\t\tmy @temp = split(/[ \\n]/, run_git(\"config --get-all remote.\"\n+\t\t\t\t\t\t. $remotename .\".namespaces\"));\n+\t\tchomp(@temp);\n+\t\tforeach my $ns (@temp) {\n+\t\t\tmy ($n, $s) = split(/:/, $ns);\n+\t\t\t$namespace_id{$n} = $s;\n+\t\t}\n+\t}\n+\n+\tif (!exists $namespace_id{$name}) {\n+\t\t# NS not found => get namespace id from MW and store it in\n+\t # configuration file.\n+\t my $query = {\n+\t action => 'query',\n+\t meta => 'siteinfo',\n+\t siprop => 'namespaces'\n+\t };\n+\t my $result = $mediawiki->api($query);\n+\n+\t while (my ($id, $ns) = each(%{$result->{query}->{namespaces}})) {\n+\t if (defined($ns->{canonical}) && ($ns->{canonical} eq $name)) {\n+\t run_git(\"config --add remote.\". $remotename\n+\t\t\t\t\t.\".namespaces \". $name .\":\". $ns->{id});\n+\t\t\t\t$namespace_id{$name} = $ns->{id};\n+\t }\n+\t }\n+\t}\n+\n+\tif (exists $namespace_id{$name}) {\n+\t\treturn $namespace_id{$name};\n+\t} else {\n+\t\tdie \"No such namespace $name on MediaWiki.\";\n+\t}\n+}\n-- \n1.7.11.5.g0c7e058.dirty\n"},{"id":"194354","messageId":"1340806889-23081-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340788220-10084-6-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 5/5 v3] git-remote-mediawiki: import \"File:\" attachments","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-06-27T14:21:29Z","receivedAt":"2012-06-27T14:21:29Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: Pavel Volek \n\nAdd the symmetrical feature to the \"File:\" export support in the previous\npatch. Download files from the wiki as needed, and feed them into the\nfast-import stream. Import both the file itself, and the corresponding\ndescription page.\n\nSigned-off-by: Pavel Volek \nSigned-off-by: NGUYEN Kim Thuat \nSigned-off-by: ROUCHER IGLESIAS Javier \nSigned-off-by: Matthieu Moy \n---\nSorry, I tested the code on another machine (latest Ubuntu, while I\nwas testing on Debian stable), and found a bug. Junio, can you take\nthis version instead? The diff with the previous version is just this:\n\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -645,6 +645,8 @@ sub literal_data {\n sub literal_data_raw {\n # Output possibly binary content.\n my ($content) = @_;\n+ # Avoid confusion between size in bytes and in characters\n+ utf8::downgrade($content);\n binmode STDOUT, \":raw\";\n print STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n binmode STDOUT, \":utf8\";\n\n contrib/mw-to-git/git-remote-mediawiki | 257 ++++++++++++++++++++++++++++++++-\n 1 file changed, 249 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 361dbb1..76b78bc 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -13,9 +13,6 @@\n #\n # Known limitations:\n #\n-# - Only wiki pages are managed, no support for [[File:...]]\n-# attachments.\n-#\n # - Poor performance in the best case: it takes forever to check\n # whether we're up-to-date (on fetch or push) or to fetch a few\n # revisions from a large wiki, because we use exclusively a\n@@ -36,6 +33,7 @@\n use strict;\n use MediaWiki::API;\n use DateTime::Format::ISO8601;\n+use FileHandle;\n \n # By default, use UTF-8 to communicate with Git and the user\n binmode STDERR, \":utf8\";\n@@ -72,6 +70,11 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".categories\"));\n chomp(@tracked_categories);\n \n+# Import media files too.\n+my $import_media = run_git(\"config --get --bool remote.\". $remotename .\".mediaimport\");\n+chomp($import_media);\n+$import_media = ($import_media eq \"true\");\n+\n my $wiki_login = run_git(\"config --get remote.\". $remotename .\".mwLogin\");\n # TODO: ideally, this should be able to read from keyboard, but we're\n # inside a remote helper, so our stdin is connect to git, not to a\n@@ -87,6 +90,9 @@ my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shall\n chomp($shallow_import);\n $shallow_import = ($shallow_import eq \"true\");\n \n+# Cache for MediaWiki namespace ids.\n+my %namespace_id;\n+\n # Dumb push: don't update notes and mediawiki ref to reflect the last push.\n #\n # Configurable with mediawiki.dumbPush, or per-remote with\n@@ -363,6 +369,14 @@ sub get_mw_pages {\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n+\tif ($import_media) {\n+\t\tprint STDERR \"Getting media files for selected pages...\\n\";\n+\t\tif ($user_defined) {\n+\t\t\tget_linked_mediafiles(\\%pages);\n+\t\t} else {\n+\t\t\tget_all_mediafiles(\\%pages);\n+\t\t}\n+\t}\n \treturn values(%pages);\n }\n \n@@ -379,6 +393,152 @@ sub run_git {\n }\n \n \n+sub get_all_mediafiles {\n+\tmy $pages = shift;\n+\t# Attach list of all pages for media files from the API,\n+\t# they are in a different namespace, only one namespace\n+\t# can be queried at the same moment\n+\tmy $mw_pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\tapnamespace => get_mw_namespace_id(\"File\"),\n+\t\taplimit => 'max'\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not get the list of pages for media files.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\tforeach my $page (@{$mw_pages}) {\n+\t\t$pages->{$page->{title}} = $page;\n+\t}\n+}\n+\n+sub get_linked_mediafiles {\n+\tmy $pages = shift;\n+\tmy @titles = map $_->{title}, values(%{$pages});\n+\n+\t# The query is split in small batches because of the MW API limit of\n+\t# the number of links to be returned (500 links max).\n+\tmy $batch = 10;\n+\twhile (@titles) {\n+\t\tif ($#titles < $batch) {\n+\t\t\t$batch = $#titles;\n+\t\t}\n+\t\tmy @slice = @titles[0..$batch];\n+\n+\t\t# pattern 'page1|page2|...' required by the API\n+\t\tmy $mw_titles = join('|', @slice);\n+\n+\t\t# Media files could be included or linked from\n+\t\t# a page, get all related\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'links|images',\n+\t\t\ttitles => $mw_titles,\n+\t\t\tplnamespace => get_mw_namespace_id(\"File\"),\n+\t\t\tpllimit => 'max'\n+\t\t};\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\twhile (my ($id, $page) = each(%{$result->{query}->{pages}})) {\n+\t\t\tmy @titles;\n+\t\t\tif (defined($page->{links})) {\n+\t\t\t\tmy @link_titles = map $_->{title}, @{$page->{links}};\n+\t\t\t\tpush(@titles, @link_titles);\n+\t\t\t}\n+\t\t\tif (defined($page->{images})) {\n+\t\t\t\tmy @image_titles = map $_->{title}, @{$page->{images}};\n+\t\t\t\tpush(@titles, @image_titles);\n+\t\t\t}\n+\t\t\tif (@titles) {\n+\t\t\t\tget_mw_first_pages(\\@titles, \\%{$pages});\n+\t\t\t}\n+\t\t}\n+\n+\t\t@titles = @titles[($batch+1)..$#titles];\n+\t}\n+}\n+\n+sub get_mw_mediafile_for_page_revision {\n+\t# Name of the file on Wiki, with the prefix.\n+\tmy $mw_filename = shift;\n+\tmy $timestamp = shift;\n+\tmy %mediafile;\n+\n+\t# Search if on MediaWiki exists a media file with given\n+\t# timestamp. In that case download the file.\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tprop => 'imageinfo',\n+\t\ttitles => $mw_filename,\n+\t\tiistart => $timestamp,\n+\t\tiiend => $timestamp,\n+\t\tiiprop => 'timestamp|archivename|url',\n+\t\tiilimit => 1\n+\t};\n+\tmy $result = $mediawiki->api($query);\n+\n+\tmy ($fileid, $file) = each ( %{$result->{query}->{pages}} );\n+\t# If not defined it means there is no revision of the file for\n+\t# given timestamp.\n+\tif (defined($file->{imageinfo})) {\n+\t\t# Get real name of media file.\n+\t\tmy $filename;\n+\t\tif (index($mw_filename, 'File:') == 0) {\n+\t\t\t$filename = substr $mw_filename, 5;\n+\t\t} else {\n+\t\t\t$filename = substr $mw_filename, 6;\n+\t\t}\n+\t\t$mediafile{title} = $filename;\n+\n+\t\tmy $fileinfo = pop(@{$file->{imageinfo}});\n+\t\t$mediafile{timestamp} = $fileinfo->{timestamp};\n+\t\t# If this is an old version of the file, the file has to be\n+\t\t# obtained from the archive. Otherwise it can be downloaded\n+\t\t# by MediaWiki API download() function.\n+\t\tif (defined($fileinfo->{archivename})) {\n+\t\t\t$mediafile{content} = download_mw_mediafile_from_archive($fileinfo->{url});\n+\t\t} else {\n+\t\t\t$mediafile{content} = download_mw_mediafile($mw_filename);\n+\t\t}\n+\t}\n+\treturn %mediafile;\n+}\n+\n+sub download_mw_mediafile_from_archive {\n+\tmy $url = shift;\n+\tmy $file;\n+\n+\tmy $ua = LWP::UserAgent->new;\n+\tmy $response = $ua->get($url);\n+\tif ($response->code) {\n+\t\t$file = $response->decoded_content;\n+\t} else {\n+\t\tprint STDERR \"Error downloading a file from archive.\\n\";\n+\t}\n+\n+\treturn $file;\n+}\n+\n+sub download_mw_mediafile {\n+\tmy $filename = shift;\n+\n+\t$mediawiki->{config}->{files_url} = $url;\n+\n+\tmy $file_content = $mediawiki->download( { title => $filename } );\n+\tif (!defined($file_content)) {\n+\t\tprint STDERR \"\\tFile \\'$filename\\' could not be downloaded.\\n\";\n+\t\texit 1;\n+\t} elsif ($file_content eq \"\") {\n+\t\tprint STDERR \"\\tFile \\'$filename\\' does not exist on the wiki.\\n\";\n+\t\texit 1;\n+\t} else {\n+\t\treturn $file_content;\n+\t}\n+}\n+\n sub get_last_local_revision {\n \t# Get note regarding last mediawiki revision\n \tmy $note = run_git(\"notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n@@ -482,6 +642,16 @@ sub literal_data {\n \tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n }\n \n+sub literal_data_raw {\n+\t# Output possibly binary content.\n+\tmy ($content) = @_;\n+\t# Avoid confusion between size in bytes and in characters\n+\tutf8::downgrade($content);\n+\tbinmode STDOUT, \":raw\";\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+\tbinmode STDOUT, \":utf8\";\n+}\n+\n sub mw_capabilities {\n \t# Revisions are imported to the private namespace\n \t# refs/mediawiki/$remotename/ by the helper and fetched into\n@@ -569,6 +739,11 @@ sub import_file_revision {\n \tmy %commit = %{$commit};\n \tmy $full_import = shift;\n \tmy $n = shift;\n+\tmy $mediafile = shift;\n+\tmy %mediafile;\n+\tif ($mediafile) {\n+\t\t%mediafile = %{$mediafile};\n+\t}\n \n \tmy $title = $commit{title};\n \tmy $comment = $commit{comment};\n@@ -588,6 +763,10 @@ sub import_file_revision {\n \tif ($content ne DELETED_CONTENT) {\n \t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n \t\tliteral_data($content);\n+\t\tif (%mediafile) {\n+\t\t\tprint STDOUT \"M 644 inline $mediafile{title}\\n\";\n+\t\t\tliteral_data_raw($mediafile{content});\n+\t\t}\n \t\tprint STDOUT \"\\n\\n\";\n \t} else {\n \t\tprint STDOUT \"D $title.mw\\n\";\n@@ -683,12 +862,11 @@ sub mw_import_ref {\n \n \t\t$n++;\n \n+\t\tmy $page_title = $result->{query}->{pages}->{$pagerevid->{pageid}}->{title};\n \t\tmy %commit;\n \t\t$commit{author} = $rev->{user} || 'Anonymous';\n \t\t$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';\n-\t\t$commit{title} = mediawiki_smudge_filename(\n-\t\t\t$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}\n-\t\t );\n+\t\t$commit{title} = mediawiki_smudge_filename($page_title);\n \t\t$commit{mw_revision} = $pagerevid->{revid};\n \t\t$commit{content} = mediawiki_smudge($rev->{'*'});\n \n@@ -699,9 +877,25 @@ sub mw_import_ref {\n \t\t}\n \t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n \n-\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\t\t# Differentiates classic pages and media files.\n+\t\tmy @prefix = split(\":\", $page_title);\n \n-\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t\tmy %mediafile;\n+\t\tif ($prefix[0] eq \"File\" || $prefix[0] eq \"Image\") {\n+\t\t\t# The name of the file is the same as the media page.\n+\t\t\tmy $filename = $page_title;\n+\t\t\t%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});\n+\t\t}\n+\t\t# If this is a revision of the media page for new version\n+\t\t# of a file do one common commit for both file and media page.\n+\t\t# Else do commit only for that page.\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\t\tif (%mediafile) {\n+\t\t\tprint STDERR \"\\tDownloading file $mediafile{title}, version $mediafile{timestamp}\\n\";\n+\t\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n, \\%mediafile);\n+\t\t} else {\n+\t\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t\t}\n \t}\n \n \tif ($fetch_from == 1 && $n == 0) {\n@@ -1006,3 +1200,50 @@ sub get_allowed_file_extensions {\n \n \treturn %hashFile;\n }\n+\n+# Return MediaWiki id for a canonical namespace name.\n+# Ex.: \"File\", \"Project\".\n+# Looks for the namespace id in the local configuration\n+# variables, if it is not found asks MW API.\n+sub get_mw_namespace_id {\n+\tmw_connect_maybe();\n+\tmy $name = shift;\n+\n+\tif (!exists $namespace_id{$name}) {\n+\t\t# Look at configuration file, if the record for that namespace is\n+\t\t# already stored. Namespaces are stored in form:\n+\t\t# \"Name_of_namespace:Id_namespace\", ex.: \"File:6\".\n+\t\tmy @temp = split(/[ \\n]/, run_git(\"config --get-all remote.\"\n+\t\t\t\t\t\t. $remotename .\".namespaces\"));\n+\t\tchomp(@temp);\n+\t\tforeach my $ns (@temp) {\n+\t\t\tmy ($n, $s) = split(/:/, $ns);\n+\t\t\t$namespace_id{$n} = $s;\n+\t\t}\n+\t}\n+\n+\tif (!exists $namespace_id{$name}) {\n+\t\t# NS not found => get namespace id from MW and store it in\n+\t # configuration file.\n+\t my $query = {\n+\t action => 'query',\n+\t meta => 'siteinfo',\n+\t siprop => 'namespaces'\n+\t };\n+\t my $result = $mediawiki->api($query);\n+\n+\t while (my ($id, $ns) = each(%{$result->{query}->{namespaces}})) {\n+\t if (defined($ns->{canonical}) && ($ns->{canonical} eq $name)) {\n+\t run_git(\"config --add remote.\". $remotename\n+\t\t\t\t\t.\".namespaces \". $name .\":\". $ns->{id});\n+\t\t\t\t$namespace_id{$name} = $ns->{id};\n+\t }\n+\t }\n+\t}\n+\n+\tif (exists $namespace_id{$name}) {\n+\t\treturn $namespace_id{$name};\n+\t} else {\n+\t\tdie \"No such namespace $name on MediaWiki.\";\n+\t}\n+}\n-- \n1.7.11.1.147.g47a574d\n"},{"id":"194615","messageId":"1341406416-1251-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"30905","inReplyTo":"1340806889-23081-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH v4] git-remote-mediawiki: import \"File:\" attachments","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-07-04T12:53:36Z","receivedAt":"2012-07-04T12:53:36Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: Pavel Volek \n\nAdd the symmetrical feature to the \"File:\" export support in the previous\npatch. Download files from the wiki as needed, and feed them into the\nfast-import stream. Import both the file itself, and the corresponding\ndescription page.\n\nSigned-off-by: Pavel Volek \nSigned-off-by: NGUYEN Kim Thuat \nSigned-off-by: ROUCHER IGLESIAS Javier \nSigned-off-by: Matthieu Moy \nSigned-off-by: Junio C Hamano \n---\nThis is meant to replace commit 6a9e55b0fc5df40 in branch\nmm/mediawiki-file-attachments in pu.\n\nThe main problem I fixed is the support for non-english wikis, where\nmediafiles can live in namespaces with names different from Image and\nFile. While I was there, I reworked the code in various places, the\nnew code should be simpler and easier to read.\n\n contrib/mw-to-git/git-remote-mediawiki | 241 +++++++++++++++++++++++++++++++--\n 1 file changed, 232 insertions(+), 9 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 361dbb1..063a978 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -13,9 +13,6 @@\n #\n # Known limitations:\n #\n-# - Only wiki pages are managed, no support for [[File:...]]\n-# attachments.\n-#\n # - Poor performance in the best case: it takes forever to check\n # whether we're up-to-date (on fetch or push) or to fetch a few\n # revisions from a large wiki, because we use exclusively a\n@@ -72,6 +69,11 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".categories\"));\n chomp(@tracked_categories);\n \n+# Import media files too.\n+my $import_media = run_git(\"config --get --bool remote.\". $remotename .\".mediaimport\");\n+chomp($import_media);\n+$import_media = ($import_media eq \"true\");\n+\n my $wiki_login = run_git(\"config --get remote.\". $remotename .\".mwLogin\");\n # TODO: ideally, this should be able to read from keyboard, but we're\n # inside a remote helper, so our stdin is connect to git, not to a\n@@ -261,7 +263,13 @@ sub mw_connect_maybe {\n ## Functions for listing pages on the remote wiki\n sub get_mw_tracked_pages {\n \tmy $pages = shift;\n-\tmy @some_pages = @tracked_pages;\n+\tget_mw_page_list(\\@tracked_pages, $pages);\n+}\n+\n+sub get_mw_page_list {\n+\tmy $page_list = shift;\n+\tmy $pages = shift;\n+\tmy @some_pages = @$page_list;\n \twhile (@some_pages) {\n \t\tmy $last = 50;\n \t\tif ($#some_pages < $last) {\n@@ -363,6 +371,14 @@ sub get_mw_pages {\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n+\tif ($import_media) {\n+\t\tprint STDERR \"Getting media files for selected pages...\\n\";\n+\t\tif ($user_defined) {\n+\t\t\tget_linked_mediafiles(\\%pages);\n+\t\t} else {\n+\t\t\tget_all_mediafiles(\\%pages);\n+\t\t}\n+\t}\n \treturn values(%pages);\n }\n \n@@ -379,6 +395,123 @@ sub run_git {\n }\n \n \n+sub get_all_mediafiles {\n+\tmy $pages = shift;\n+\t# Attach list of all pages for media files from the API,\n+\t# they are in a different namespace, only one namespace\n+\t# can be queried at the same moment\n+\tmy $mw_pages = $mediawiki->list({\n+\t\taction => 'query',\n+\t\tlist => 'allpages',\n+\t\tapnamespace => get_mw_namespace_id(\"File\"),\n+\t\taplimit => 'max'\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not get the list of pages for media files.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\tforeach my $page (@{$mw_pages}) {\n+\t\t$pages->{$page->{title}} = $page;\n+\t}\n+}\n+\n+sub get_linked_mediafiles {\n+\tmy $pages = shift;\n+\tmy @titles = map $_->{title}, values(%{$pages});\n+\n+\t# The query is split in small batches because of the MW API limit of\n+\t# the number of links to be returned (500 links max).\n+\tmy $batch = 10;\n+\twhile (@titles) {\n+\t\tif ($#titles < $batch) {\n+\t\t\t$batch = $#titles;\n+\t\t}\n+\t\tmy @slice = @titles[0..$batch];\n+\n+\t\t# pattern 'page1|page2|...' required by the API\n+\t\tmy $mw_titles = join('|', @slice);\n+\n+\t\t# Media files could be included or linked from\n+\t\t# a page, get all related\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'links|images',\n+\t\t\ttitles => $mw_titles,\n+\t\t\tplnamespace => get_mw_namespace_id(\"File\"),\n+\t\t\tpllimit => 'max'\n+\t\t};\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\twhile (my ($id, $page) = each(%{$result->{query}->{pages}})) {\n+\t\t\tmy @media_titles;\n+\t\t\tif (defined($page->{links})) {\n+\t\t\t\tmy @link_titles = map $_->{title}, @{$page->{links}};\n+\t\t\t\tpush(@media_titles, @link_titles);\n+\t\t\t}\n+\t\t\tif (defined($page->{images})) {\n+\t\t\t\tmy @image_titles = map $_->{title}, @{$page->{images}};\n+\t\t\t\tpush(@media_titles, @image_titles);\n+\t\t\t}\n+\t\t\tif (@media_titles) {\n+\t\t\t\tget_mw_page_list(\\@media_titles, $pages);\n+\t\t\t}\n+\t\t}\n+\n+\t\t@titles = @titles[($batch+1)..$#titles];\n+\t}\n+}\n+\n+sub get_mw_mediafile_for_page_revision {\n+\t# Name of the file on Wiki, with the prefix.\n+\tmy $filename = shift;\n+\tmy $timestamp = shift;\n+\tmy %mediafile;\n+\n+\t# Search if on a media file with given timestamp exists on\n+\t# MediaWiki. In that case download the file.\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tprop => 'imageinfo',\n+\t\ttitles => \"File:\" . $filename,\n+\t\tiistart => $timestamp,\n+\t\tiiend => $timestamp,\n+\t\tiiprop => 'timestamp|archivename|url',\n+\t\tiilimit => 1\n+\t};\n+\tmy $result = $mediawiki->api($query);\n+\n+\tmy ($fileid, $file) = each( %{$result->{query}->{pages}} );\n+\t# If not defined it means there is no revision of the file for\n+\t# given timestamp.\n+\tif (defined($file->{imageinfo})) {\n+\t\t$mediafile{title} = $filename;\n+\n+\t\tmy $fileinfo = pop(@{$file->{imageinfo}});\n+\t\t$mediafile{timestamp} = $fileinfo->{timestamp};\n+\t\t# Mediawiki::API's download function doesn't support https URLs\n+\t\t# and can't download old versions of files.\n+\t\tprint STDERR \"\\tDownloading file $mediafile{title}, version $mediafile{timestamp}\\n\";\n+\t\t$mediafile{content} = download_mw_mediafile($fileinfo->{url});\n+\t}\n+\treturn %mediafile;\n+}\n+\n+sub download_mw_mediafile {\n+\tmy $url = shift;\n+\n+\tmy $response = $mediawiki->{ua}->get($url);\n+\tif ($response->code == 200) {\n+\t\treturn $response->decoded_content;\n+\t} else {\n+\t\tprint STDERR \"Error downloading mediafile from :\\n\";\n+\t\tprint STDERR \"URL: $url\\n\";\n+\t\tprint STDERR \"Server response: \" . $response->code . \" \" . $response->message . \"\\n\";\n+\t\texit 1;\n+\t}\n+}\n+\n sub get_last_local_revision {\n \t# Get note regarding last mediawiki revision\n \tmy $note = run_git(\"notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n@@ -482,6 +615,16 @@ sub literal_data {\n \tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n }\n \n+sub literal_data_raw {\n+\t# Output possibly binary content.\n+\tmy ($content) = @_;\n+\t# Avoid confusion between size in bytes and in characters\n+\tutf8::downgrade($content);\n+\tbinmode STDOUT, \":raw\";\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+\tbinmode STDOUT, \":utf8\";\n+}\n+\n sub mw_capabilities {\n \t# Revisions are imported to the private namespace\n \t# refs/mediawiki/$remotename/ by the helper and fetched into\n@@ -569,6 +712,11 @@ sub import_file_revision {\n \tmy %commit = %{$commit};\n \tmy $full_import = shift;\n \tmy $n = shift;\n+\tmy $mediafile = shift;\n+\tmy %mediafile;\n+\tif ($mediafile) {\n+\t\t%mediafile = %{$mediafile};\n+\t}\n \n \tmy $title = $commit{title};\n \tmy $comment = $commit{comment};\n@@ -588,6 +736,10 @@ sub import_file_revision {\n \tif ($content ne DELETED_CONTENT) {\n \t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n \t\tliteral_data($content);\n+\t\tif (%mediafile) {\n+\t\t\tprint STDOUT \"M 644 inline $mediafile{title}\\n\";\n+\t\t\tliteral_data_raw($mediafile{content});\n+\t\t}\n \t\tprint STDOUT \"\\n\\n\";\n \t} else {\n \t\tprint STDOUT \"D $title.mw\\n\";\n@@ -683,12 +835,11 @@ sub mw_import_ref {\n \n \t\t$n++;\n \n+\t\tmy $page_title = $result->{query}->{pages}->{$pagerevid->{pageid}}->{title};\n \t\tmy %commit;\n \t\t$commit{author} = $rev->{user} || 'Anonymous';\n \t\t$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';\n-\t\t$commit{title} = mediawiki_smudge_filename(\n-\t\t\t$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}\n-\t\t );\n+\t\t$commit{title} = mediawiki_smudge_filename($page_title);\n \t\t$commit{mw_revision} = $pagerevid->{revid};\n \t\t$commit{content} = mediawiki_smudge($rev->{'*'});\n \n@@ -699,9 +850,17 @@ sub mw_import_ref {\n \t\t}\n \t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n \n+\t\t# Differentiates classic pages and media files.\n+\t\tmy ($namespace, $filename) = $page_title =~ /^([^:]*):(.*)$/;\n+\t\tmy %mediafile;\n+\t\tif ($namespace && get_mw_namespace_id($namespace) == get_mw_namespace_id(\"File\")) {\n+\t\t\t%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});\n+\t\t}\n+\t\t# If this is a revision of the media page for new version\n+\t\t# of a file do one common commit for both file and media page.\n+\t\t# Else do commit only for that page.\n \t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n-\n-\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n, \\%mediafile);\n \t}\n \n \tif ($fetch_from == 1 && $n == 0) {\n@@ -1006,3 +1165,67 @@ sub get_allowed_file_extensions {\n \n \treturn %hashFile;\n }\n+\n+# In memory cache for MediaWiki namespace ids.\n+my %namespace_id;\n+\n+# Namespaces whose id is cached in the configuration file\n+# (to avoid duplicates)\n+my %cached_mw_namespace_id;\n+\n+# Return MediaWiki id for a canonical namespace name.\n+# Ex.: \"File\", \"Project\".\n+sub get_mw_namespace_id {\n+\tmw_connect_maybe();\n+\tmy $name = shift;\n+\n+\tif (!exists $namespace_id{$name}) {\n+\t\t# Look at configuration file, if the record for that namespace is\n+\t\t# already cached. Namespaces are stored in form:\n+\t\t# \"Name_of_namespace:Id_namespace\", ex.: \"File:6\".\n+\t\tmy @temp = split(/[ \\n]/, run_git(\"config --get-all remote.\"\n+\t\t\t\t\t\t. $remotename .\".namespaceCache\"));\n+\t\tchomp(@temp);\n+\t\tforeach my $ns (@temp) {\n+\t\t\tmy ($n, $id) = split(/:/, $ns);\n+\t\t\t$namespace_id{$n} = $id;\n+\t\t\t$cached_mw_namespace_id{$n} = 1;\n+\t\t}\n+\t}\n+\n+\tif (!exists $namespace_id{$name}) {\n+\t\tprint STDERR \"Namespace $name not found in cache, querying the wiki ...\\n\";\n+\t\t# NS not found => get namespace id from MW and store it in\n+\t # configuration file.\n+\t my $query = {\n+\t action => 'query',\n+\t meta => 'siteinfo',\n+\t siprop => 'namespaces'\n+\t };\n+\t my $result = $mediawiki->api($query);\n+\n+\t while (my ($id, $ns) = each(%{$result->{query}->{namespaces}})) {\n+\t if (defined($ns->{id}) && defined($ns->{canonical})) {\n+\t\t\t\t$namespace_id{$ns->{canonical}} = $ns->{id};\n+\t\t\t\tif ($ns->{'*'}) {\n+\t\t\t\t\t# alias (e.g. french Fichier: as alias for canonical File:)\n+\t\t\t\t\t$namespace_id{$ns->{'*'}} = $ns->{id};\n+\t\t\t\t}\n+\t\t\t}\n+\t }\n+\t}\n+\n+\tmy $id = $namespace_id{$name};\n+\n+\tif (defined $id) {\n+\t\t# Store explicitely requested namespaces on disk\n+\t\tif (!exists $cached_mw_namespace_id{$name}) {\n+\t\t\trun_git(\"config --add remote.\". $remotename\n+\t\t\t\t.\".namespaceCache \\\"\". $name .\":\". $id .\"\\\"\");\n+\t\t\t$cached_mw_namespace_id{$name} = 1;\n+\t\t}\n+\t\treturn $id;\n+\t} else {\n+\t\tdie \"No such namespace $name on MediaWiki.\";\n+\t}\n+}\n-- \n1.7.11.1.147.g47a574d\n"},{"id":"194630","messageId":"7vk3yisnhe.fsf@alter.siamese.dyndns.org","threadId":"30905","inReplyTo":"1341406416-1251-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH v4] git-remote-mediawiki: import \"File:\" attachments","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-05T06:58:37Z","receivedAt":"2012-07-05T06:58:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> This is meant to replace commit 6a9e55b0fc5df40 in branch\n> mm/mediawiki-file-attachments in pu.\n\nBad timing for our mails to cross; it is already on 'next'.\n\nWould the following be a good \"incremental update\" on top of the\nnamed commit?\n\n-- >8 --\nFrom: Matthieu Moy \nDate: Wed, 4 Jul 2012 14:53:36 +0200\nSubject: [PATCH] git-remote-mediawiki: improve support for non-English Wikis\n\nMediafiles can live in namespaces with names different from Image\nand File. While at it, rework the code to make it simpler and easier\nto read.\n\nSigned-off-by: Matthieu Moy \nSigned-off-by: Junio C Hamano \n---\n contrib/mw-to-git/git-remote-mediawiki | 140 ++++++++++++++-------------------\n 1 file changed, 61 insertions(+), 79 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 76b78bc..063a978 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -33,7 +33,6 @@\n use strict;\n use MediaWiki::API;\n use DateTime::Format::ISO8601;\n-use FileHandle;\n \n # By default, use UTF-8 to communicate with Git and the user\n binmode STDERR, \":utf8\";\n@@ -90,9 +89,6 @@ my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shall\n chomp($shallow_import);\n $shallow_import = ($shallow_import eq \"true\");\n \n-# Cache for MediaWiki namespace ids.\n-my %namespace_id;\n-\n # Dumb push: don't update notes and mediawiki ref to reflect the last push.\n #\n # Configurable with mediawiki.dumbPush, or per-remote with\n@@ -267,7 +263,13 @@ sub mw_connect_maybe {\n ## Functions for listing pages on the remote wiki\n sub get_mw_tracked_pages {\n \tmy $pages = shift;\n-\tmy @some_pages = @tracked_pages;\n+\tget_mw_page_list(\\@tracked_pages, $pages);\n+}\n+\n+sub get_mw_page_list {\n+\tmy $page_list = shift;\n+\tmy $pages = shift;\n+\tmy @some_pages = @$page_list;\n \twhile (@some_pages) {\n \t\tmy $last = 50;\n \t\tif ($#some_pages < $last) {\n@@ -443,17 +445,17 @@ sub get_linked_mediafiles {\n \t\tmy $result = $mediawiki->api($query);\n \n \t\twhile (my ($id, $page) = each(%{$result->{query}->{pages}})) {\n-\t\t\tmy @titles;\n+\t\t\tmy @media_titles;\n \t\t\tif (defined($page->{links})) {\n \t\t\t\tmy @link_titles = map $_->{title}, @{$page->{links}};\n-\t\t\t\tpush(@titles, @link_titles);\n+\t\t\t\tpush(@media_titles, @link_titles);\n \t\t\t}\n \t\t\tif (defined($page->{images})) {\n \t\t\t\tmy @image_titles = map $_->{title}, @{$page->{images}};\n-\t\t\t\tpush(@titles, @image_titles);\n+\t\t\t\tpush(@media_titles, @image_titles);\n \t\t\t}\n-\t\t\tif (@titles) {\n-\t\t\t\tget_mw_first_pages(\\@titles, \\%{$pages});\n+\t\t\tif (@media_titles) {\n+\t\t\t\tget_mw_page_list(\\@media_titles, $pages);\n \t\t\t}\n \t\t}\n \n@@ -463,16 +465,16 @@ sub get_linked_mediafiles {\n \n sub get_mw_mediafile_for_page_revision {\n \t# Name of the file on Wiki, with the prefix.\n-\tmy $mw_filename = shift;\n+\tmy $filename = shift;\n \tmy $timestamp = shift;\n \tmy %mediafile;\n \n-\t# Search if on MediaWiki exists a media file with given\n-\t# timestamp. In that case download the file.\n+\t# Search if on a media file with given timestamp exists on\n+\t# MediaWiki. In that case download the file.\n \tmy $query = {\n \t\taction => 'query',\n \t\tprop => 'imageinfo',\n-\t\ttitles => $mw_filename,\n+\t\ttitles => \"File:\" . $filename,\n \t\tiistart => $timestamp,\n \t\tiiend => $timestamp,\n \t\tiiprop => 'timestamp|archivename|url',\n@@ -480,62 +482,33 @@ sub get_mw_mediafile_for_page_revision {\n \t};\n \tmy $result = $mediawiki->api($query);\n \n-\tmy ($fileid, $file) = each ( %{$result->{query}->{pages}} );\n+\tmy ($fileid, $file) = each( %{$result->{query}->{pages}} );\n \t# If not defined it means there is no revision of the file for\n \t# given timestamp.\n \tif (defined($file->{imageinfo})) {\n-\t\t# Get real name of media file.\n-\t\tmy $filename;\n-\t\tif (index($mw_filename, 'File:') == 0) {\n-\t\t\t$filename = substr $mw_filename, 5;\n-\t\t} else {\n-\t\t\t$filename = substr $mw_filename, 6;\n-\t\t}\n \t\t$mediafile{title} = $filename;\n \n \t\tmy $fileinfo = pop(@{$file->{imageinfo}});\n \t\t$mediafile{timestamp} = $fileinfo->{timestamp};\n-\t\t# If this is an old version of the file, the file has to be\n-\t\t# obtained from the archive. Otherwise it can be downloaded\n-\t\t# by MediaWiki API download() function.\n-\t\tif (defined($fileinfo->{archivename})) {\n-\t\t\t$mediafile{content} = download_mw_mediafile_from_archive($fileinfo->{url});\n-\t\t} else {\n-\t\t\t$mediafile{content} = download_mw_mediafile($mw_filename);\n-\t\t}\n+\t\t# Mediawiki::API's download function doesn't support https URLs\n+\t\t# and can't download old versions of files.\n+\t\tprint STDERR \"\\tDownloading file $mediafile{title}, version $mediafile{timestamp}\\n\";\n+\t\t$mediafile{content} = download_mw_mediafile($fileinfo->{url});\n \t}\n \treturn %mediafile;\n }\n \n-sub download_mw_mediafile_from_archive {\n+sub download_mw_mediafile {\n \tmy $url = shift;\n-\tmy $file;\n \n-\tmy $ua = LWP::UserAgent->new;\n-\tmy $response = $ua->get($url);\n-\tif ($response->code) {\n-\t\t$file = $response->decoded_content;\n+\tmy $response = $mediawiki->{ua}->get($url);\n+\tif ($response->code == 200) {\n+\t\treturn $response->decoded_content;\n \t} else {\n-\t\tprint STDERR \"Error downloading a file from archive.\\n\";\n-\t}\n-\n-\treturn $file;\n-}\n-\n-sub download_mw_mediafile {\n-\tmy $filename = shift;\n-\n-\t$mediawiki->{config}->{files_url} = $url;\n-\n-\tmy $file_content = $mediawiki->download( { title => $filename } );\n-\tif (!defined($file_content)) {\n-\t\tprint STDERR \"\\tFile \\'$filename\\' could not be downloaded.\\n\";\n-\t\texit 1;\n-\t} elsif ($file_content eq \"\") {\n-\t\tprint STDERR \"\\tFile \\'$filename\\' does not exist on the wiki.\\n\";\n+\t\tprint STDERR \"Error downloading mediafile from :\\n\";\n+\t\tprint STDERR \"URL: $url\\n\";\n+\t\tprint STDERR \"Server response: \" . $response->code . \" \" . $response->message . \"\\n\";\n \t\texit 1;\n-\t} else {\n-\t\treturn $file_content;\n \t}\n }\n \n@@ -878,24 +851,16 @@ sub mw_import_ref {\n \t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n \n \t\t# Differentiates classic pages and media files.\n-\t\tmy @prefix = split(\":\", $page_title);\n-\n+\t\tmy ($namespace, $filename) = $page_title =~ /^([^:]*):(.*)$/;\n \t\tmy %mediafile;\n-\t\tif ($prefix[0] eq \"File\" || $prefix[0] eq \"Image\") {\n-\t\t\t# The name of the file is the same as the media page.\n-\t\t\tmy $filename = $page_title;\n+\t\tif ($namespace && get_mw_namespace_id($namespace) == get_mw_namespace_id(\"File\")) {\n \t\t\t%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});\n \t\t}\n \t\t# If this is a revision of the media page for new version\n \t\t# of a file do one common commit for both file and media page.\n \t\t# Else do commit only for that page.\n \t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n-\t\tif (%mediafile) {\n-\t\t\tprint STDERR \"\\tDownloading file $mediafile{title}, version $mediafile{timestamp}\\n\";\n-\t\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n, \\%mediafile);\n-\t\t} else {\n-\t\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n-\t\t}\n+\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n, \\%mediafile);\n \t}\n \n \tif ($fetch_from == 1 && $n == 0) {\n@@ -1201,28 +1166,35 @@ sub get_allowed_file_extensions {\n \treturn %hashFile;\n }\n \n+# In memory cache for MediaWiki namespace ids.\n+my %namespace_id;\n+\n+# Namespaces whose id is cached in the configuration file\n+# (to avoid duplicates)\n+my %cached_mw_namespace_id;\n+\n # Return MediaWiki id for a canonical namespace name.\n # Ex.: \"File\", \"Project\".\n-# Looks for the namespace id in the local configuration\n-# variables, if it is not found asks MW API.\n sub get_mw_namespace_id {\n \tmw_connect_maybe();\n \tmy $name = shift;\n \n \tif (!exists $namespace_id{$name}) {\n \t\t# Look at configuration file, if the record for that namespace is\n-\t\t# already stored. Namespaces are stored in form:\n+\t\t# already cached. Namespaces are stored in form:\n \t\t# \"Name_of_namespace:Id_namespace\", ex.: \"File:6\".\n \t\tmy @temp = split(/[ \\n]/, run_git(\"config --get-all remote.\"\n-\t\t\t\t\t\t. $remotename .\".namespaces\"));\n+\t\t\t\t\t\t. $remotename .\".namespaceCache\"));\n \t\tchomp(@temp);\n \t\tforeach my $ns (@temp) {\n-\t\t\tmy ($n, $s) = split(/:/, $ns);\n-\t\t\t$namespace_id{$n} = $s;\n+\t\t\tmy ($n, $id) = split(/:/, $ns);\n+\t\t\t$namespace_id{$n} = $id;\n+\t\t\t$cached_mw_namespace_id{$n} = 1;\n \t\t}\n \t}\n \n \tif (!exists $namespace_id{$name}) {\n+\t\tprint STDERR \"Namespace $name not found in cache, querying the wiki ...\\n\";\n \t\t# NS not found => get namespace id from MW and store it in\n \t # configuration file.\n \t my $query = {\n@@ -1233,16 +1205,26 @@ sub get_mw_namespace_id {\n \t my $result = $mediawiki->api($query);\n \n \t while (my ($id, $ns) = each(%{$result->{query}->{namespaces}})) {\n-\t if (defined($ns->{canonical}) && ($ns->{canonical} eq $name)) {\n-\t run_git(\"config --add remote.\". $remotename\n-\t\t\t\t\t.\".namespaces \". $name .\":\". $ns->{id});\n-\t\t\t\t$namespace_id{$name} = $ns->{id};\n-\t }\n+\t if (defined($ns->{id}) && defined($ns->{canonical})) {\n+\t\t\t\t$namespace_id{$ns->{canonical}} = $ns->{id};\n+\t\t\t\tif ($ns->{'*'}) {\n+\t\t\t\t\t# alias (e.g. french Fichier: as alias for canonical File:)\n+\t\t\t\t\t$namespace_id{$ns->{'*'}} = $ns->{id};\n+\t\t\t\t}\n+\t\t\t}\n \t }\n \t}\n \n-\tif (exists $namespace_id{$name}) {\n-\t\treturn $namespace_id{$name};\n+\tmy $id = $namespace_id{$name};\n+\n+\tif (defined $id) {\n+\t\t# Store explicitely requested namespaces on disk\n+\t\tif (!exists $cached_mw_namespace_id{$name}) {\n+\t\t\trun_git(\"config --add remote.\". $remotename\n+\t\t\t\t.\".namespaceCache \\\"\". $name .\":\". $id .\"\\\"\");\n+\t\t\t$cached_mw_namespace_id{$name} = 1;\n+\t\t}\n+\t\treturn $id;\n \t} else {\n \t\tdie \"No such namespace $name on MediaWiki.\";\n \t}\n-- \n1.7.11.1.243.g7462176\n"},{"id":"194647","messageId":"vpqtxxmejr9.fsf@bauges.imag.fr","threadId":"30905","inReplyTo":"7vk3yisnhe.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v4] git-remote-mediawiki: import \"File:\" attachments","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-07-05T07:42:50Z","receivedAt":"2012-07-05T07:42:50Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> Matthieu Moy writes:\n>\n>> This is meant to replace commit 6a9e55b0fc5df40 in branch\n>> mm/mediawiki-file-attachments in pu.\n>\n> Bad timing for our mails to cross; it is already on 'next'.\n>\n> Would the following be a good \"incremental update\" on top of the\n> named commit?\n\nYes, that is perfect.\n\nThanks,\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"}]}