{"thread":{"id":"28226","subject":"[PATCH 1/2] fast-import: initialize variable require_explicit_termination","startedAt":"2011-08-26T17:11:28Z","lastAt":"2011-09-03T10:35:18Z","messageCount":43,"participants":["Matthieu Moy","Junio C Hamano","Sverre Rabbelier","Jonathan Nieder"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"174329","messageId":"1314378689-8997-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":null,"subject":"[PATCH 1/2] fast-import: initialize variable require_explicit_termination","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-08-26T17:11:28Z","receivedAt":"2011-08-26T17:11:28Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"The uninitialized variable seems harmless in practice, but let's still be clean.\n\nSigned-off-by: Matthieu Moy \n---\nFor some reason, remote helpers seem to be forced to use the \"done\"\ncommand now. Investing why, I found this, but that wasn't what I was\nlooking for.\n\n fast-import.c | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex 7cc2262..ed8f3cd 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -355,7 +355,7 @@ static unsigned int cmd_save = 100;\n static uintmax_t next_mark;\n static struct strbuf new_data = STRBUF_INIT;\n static int seen_data_command;\n-static int require_explicit_termination;\n+static int require_explicit_termination = 0;\n \n /* Signal handling */\n static volatile sig_atomic_t checkpoint_requested;\n-- \n1.7.6.585.g5929f.dirty\n"},{"id":"174330","messageId":"1314378689-8997-2-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"1314378689-8997-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-08-26T17:11:29Z","receivedAt":"2011-08-26T17:11:29Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: Jeremie Nikaes \n\nImplement a gate between git and mediawiki, allowing git users to push\nand pull objects from mediawiki just as one would do with a classic git\nrepository thanks to remote-helpers.\n\nThe following packages need to be installed (available on common\nrepositories):\n\n libmediawiki-api-perl\n libdatetime-format-iso8601-perl\n\nUse remote helpers in order to be as transparent as possible to the git\nuser.\n\nDownload Mediawiki revisions through the Mediawiki API and then\nfast-import into git.\n\nMediawiki revision number and git commits are linked thanks to notes\nbound to commits.\n\nThe import part is done on a refs/mediawiki/ branch before\ncoming to refs/remote/origin/master (Huge thanks to Jonathan Nieder\nfor his help)\n\nWe use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but\nwe also read the output of Git commands in UTF-8 with the small helper\nrun_git, and write to the console (STDERR) in UTF-8. This allows a\nseamless use of non-ascii characters in page titles, but hasn't been\ntested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames\ncould raise problems if different file systems handle UTF-8 filenames\ndifferently. A uri_escape of mediawiki filenames could be imaginable, and\nis still to be discussed further.\n\nPartial cloning is supported using one of:\n\ngit clone -c remote.origin.pages='A_Page Another_Page' mediawiki::http://wikiurl\n\ngit clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl\n\ngit clone -c remote.origin.shallow='True' mediawiki::http://wikiurl\n\nThanks to notes metadata, it is possible to compare remote and local last\nmediawiki revision to warn non-fast forward pushes and \"everything\nup-to-date\" case.\n\nWhen allowed, push looks for each commit between remotes/origin/master\nand HEAD, catches every blob related to these commit and push them in\nchronological order. To do so, it uses git rev-list --children HEAD and\ntravels the tree from remotes/origin/master to HEAD through children. In\nother words :\n\n\t* Shortest path from remotes/origin/master to HEAD\n\t* For each commit encountered, push blobs related to this commit\n\nSigned-off-by: Jérémie Nikaes \nSigned-off-by: Arnaud Lacurie \nSigned-off-by: Claire Fousse \nSigned-off-by: David Amouyal \nSigned-off-by: Matthieu Moy \nSigned-off-by: Sylvain Boulmé \nSigned-off-by: Matthieu Moy \n---\nThis is a resurection of an old patch serie.\n\nThe previous round was here:\n\n http://thread.gmane.org/gmane.comp.version-control.git/175523\n\nMaintaining two separate patches for import/export was getting\npainful, so I've merged them into a single patch.\n\nGit-Mediawiki is originally a student project, and since the students\ndidn't have time to finish the work this summer, I've taken time to\ndo some cleanup and testing.\n\nIn short, the changes since v3 are:\n\n* Adapt to newer Git, which seem to require a \"done\" command at the\n end of the fast-import stream. I don't understand why this is\n needed, since fast-import is called without the --done flag by\n remote-helpers, but if I don't do this, \"git fetch\" doesn't\n terminate and keeps waiting ...\n\n* Allow importing just a category\n\n* Support shallow clone and fetch (for fetch, this means fetch just\n the last revision each time).\n\n* Basic support for authentication (with password cleartext in\n .git/config :-( ).\n\n* Support pushing to a new wiki (i.e. not the one you've cloned to).\n\n* Accented characters in filenames now displayed correctly on the\n terminal.\n\n* Support page deletion (by replacing the page content with\n [[Category:Deleted]]).\n\n* Support creation of empty files\n\n* Support forbidden characters {}[]| in filenames\n\n* Support partial clone with more than 50 pages\n\nI'd like to get this merged in contrib/, so that the code be in a\nsafe place, where other people can take care of it too. I'll probably\noffer a student project \"improve Git-Mediawiki\" to my students next\nJune.\n\n contrib/mw-to-git/git-remote-mediawiki | 722 ++++++++++++++++++++++++++++\n contrib/mw-to-git/git-remote-mediawiki.txt | 7 +\n 2 files changed, 729 insertions(+), 0 deletions(-)\n create mode 100755 contrib/mw-to-git/git-remote-mediawiki\n create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nnew file mode 100755\nindex 0000000..62c6794\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -0,0 +1,722 @@\n+#! /usr/bin/perl\n+\n+# Copyright (C) 2011\n+# Jérémie Nikaes \n+# Arnaud Lacurie \n+# Claire Fousse \n+# David Amouyal \n+# Matthieu Moy \n+# License: GPL v2 or later\n+\n+# Gateway between Git and MediaWiki.\n+# https://github.com/Bibzball/Git-Mediawiki/wiki\n+#\n+# Known limitations:\n+#\n+# - Only wiki pages are managed, no support for [[File:...]]\n+# attachments.\n+#\n+# - Poor performance in the best case: it takes forever to check\n+# whether we're up-to-date (on fetch or push) or to fetch a few\n+# revisions from a large wiki, because we use exclusively a\n+# page-based synchronization. We could switch to a wiki-wide\n+# synchronization when the synchronization involves few revisions\n+# but the wiki is large.\n+#\n+# - Git renames could be turned into MediaWiki renames (see TODO\n+# below)\n+#\n+# - login/password support requires the user to write the password\n+# cleartext in a file (see TODO below).\n+#\n+# - No way to import \"one page, and all pages included in it\"\n+#\n+# - Multiple remote MediaWikis have not been very well tested.\n+\n+use strict;\n+use MediaWiki::API;\n+use DateTime::Format::ISO8601;\n+use encoding 'utf8';\n+\n+# use encoding 'utf8' doesn't change STDERROR\n+# but we're going to output UTF-8 filenames to STDERR\n+binmode STDERR, \":utf8\";\n+\n+use URI::Escape;\n+use warnings;\n+\n+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced\n+use constant SLASH_REPLACEMENT => \"%2F\";\n+\n+# It's not always possible to delete pages (may require some\n+# priviledges). Deleted pages are replaced with this content.\n+use constant DELETED_CONTENT => \"[[Category:Deleted]]\\n\";\n+\n+# It's not possible to create empty pages. New empty files in Git are\n+# sent with this content instead.\n+use constant EMPTY_CONTENT => \"\\n\";\n+\n+# used to reflect file creation or deletion in diff.\n+use constant NULL_SHA1 => \"0000000000000000000000000000000000000000\";\n+\n+my $remotename = $ARGV[0];\n+my $url = $ARGV[1];\n+\n+# Accept both space-separated and multiple keys in config file.\n+# Spaces should be written as _ anyway because we'll use chomp.\n+my @tracked_pages = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".pages\"));\n+chomp(@tracked_pages);\n+\n+# Just like @tracked_pages, but for MediaWiki categories.\n+my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".categories\"));\n+chomp(@tracked_categories);\n+\n+my $wiki_login = run_git(\"config --get remote.\". $remotename .\".mwLogin\");\n+# TODO: ideally, this should be able to read from keyboard, but we're\n+# inside a remote helper, so our stdin is connect to git, not to a\n+# terminal.\n+my $wiki_passwd = run_git(\"config --get remote.\". $remotename .\".mwPassword\");\n+chomp ($wiki_login);\n+chomp ($wiki_passwd);\n+\n+# Import only last revisions (both for clone and fetch)\n+my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shallow\");\n+chomp($shallow_import);\n+$shallow_import = ($shallow_import eq \"true\");\n+\n+my $wiki_name = $url;\n+$wiki_name =~ s/[^\\/]*:\\/\\///;\n+\n+my $import_started;\n+\n+# Commands parser\n+my $entry;\n+my @cmd;\n+while () {\n+\tchomp;\n+\t@cmd = split(/ /);\n+\tif (defined($cmd[0])) {\n+\t\t# Line not blank\n+\t\tif ($cmd[0] eq \"capabilities\") {\n+\t\t\tdie(\"Too many arguments for capabilities\") unless (!defined($cmd[1]));\n+\t\t\tmw_capabilities();\n+\t\t} elsif ($cmd[0] eq \"list\") {\n+\t\t\tdie(\"Too many arguments for list\") unless (!defined($cmd[2]));\n+\t\t\tmw_list($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"import\") {\n+\t\t\tdie(\"Invalid arguments for import\") unless ($cmd[1] ne \"\" && !defined($cmd[2]));\n+\t\t\tmw_import($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"option\") {\n+\t\t\tdie(\"Too many arguments for option\") unless ($cmd[1] ne \"\" && $cmd[2] ne \"\" && !defined($cmd[3]));\n+\t\t\tmw_option($cmd[1],$cmd[2]);\n+\t\t} elsif ($cmd[0] eq \"push\") {\n+\t\t\t# Check the pattern :\n+\t\t\tmy @pushargs = split(/:/,$cmd[1]);\n+\t\t\tdie(\"Invalid arguments for push\") unless ($pushargs[1] ne \"\" && !defined($pushargs[2]));\n+\t\t\tmw_push($pushargs[0],$pushargs[1]);\n+\t\t} else {\n+\t\t\tprint STDERR \"Unknown command. Aborting...\\n\";\n+\t\t\tlast;\n+\t\t}\n+\t} else {\n+\t\t# blank line: we should terminate\n+\t\tlast;\n+\t}\n+\n+\tBEGIN { $| = 1 } # flush STDOUT, to make sure the previous\n+\t\t\t # command is fully processed.\n+}\n+# End of input\n+if ($import_started) {\n+\t# Terminate the fast-import stream properly.\n+\t# Git requires one \"done\" command, and only\n+\t# one This is OK since we only have one\n+\t# branch, so import will be called only once\n+\t# (plus once for HEAD, for which we won't\n+\t# reach this point).\n+\tprint STDOUT \"done\\n\";\n+}\n+BEGIN { $| = 1 };\n+if (!eof(STDIN)) {\n+\t# Wait for Git to terminate. If we don't, git fetch\n+\t# (transport-helper.c's sendline function) will try to write\n+\t# to our stdin, which will be closed, and git fetch will be\n+\t# killed. That's probably a bug in transport-helper.c, but in\n+\t# the meantime ...\n+\tsleep .1;\n+};\n+\n+########################## Functions ##############################\n+\n+# MediaWiki API instance, created lazily.\n+my $mediawiki;\n+\n+sub mw_connect_maybe {\n+\tif ($mediawiki) {\n+\t return;\n+\t}\n+\t$mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\tif ($wiki_login) {\n+\t\tif (!$mediawiki->login({\n+\t\t\tlgname => $wiki_login,\n+\t\t\tlgpassword => $wiki_passwd,\n+\t\t})) {\n+\t\t\tprint STDERR \"Failed to log in mediawiki user \\\"$wiki_login\\\" on $url\\n\";\n+\t\t\tprint STDERR \"(error \" .\n+\t\t\t $mediawiki->{error}->{code} . ': ' .\n+\t\t\t $mediawiki->{error}->{details} . \")\\n\";\n+\t\t\texit 1;\n+\t\t} else {\n+\t\t\tprint STDERR \"Logged in with user \\\"$wiki_login\\\".\\n\";\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_first_pages {\n+\tmy $some_pages = shift;\n+\tmy @some_pages = @{$some_pages};\n+\n+\tmy $pages = shift;\n+\n+\t# pattern 'page1|page2|...' required by the API\n+\tmy $titles = join('|', @some_pages);\n+\n+\tmy $mw_pages = $mediawiki->api({\n+\t\taction => 'query',\n+\t\ttitles => $titles,\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not query the list of wiki pages.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\twhile (my ($id, $page) = each (%{$mw_pages->{query}->{pages}})) {\n+\t\tif ($id < 0) {\n+\t\t\tprint STDERR \"Warning: page $page->{title} not found on wiki\\n\";\n+\t\t} else {\n+\t\t\t$pages->{$page->{title}} = $page;\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_pages {\n+\tmw_connect_maybe();\n+\n+\tmy %pages; # hash on page titles to avoid duplicates\n+\tmy $user_defined;\n+\tif (@tracked_pages) {\n+\t\t$user_defined = 1;\n+\t\t# The user provided a list of pages titles, but we\n+\t\t# still need to query the API to get the page IDs.\n+\n+\t\tmy @some_pages = @tracked_pages;\n+\t\twhile (@some_pages) {\n+\t\t\tmy $last = 50;\n+\t\t\tif ($#some_pages < $last) {\n+\t\t\t\t$last = $#some_pages;\n+\t\t\t}\n+\t\t\tmy @slice = @some_pages[0..$last];\n+\t\t\tget_mw_first_pages(\\@slice, \\%pages);\n+\t\t\t@some_pages = @some_pages[51..$#some_pages];\n+\t\t}\n+\t}\n+\tif (@tracked_categories) {\n+\t\t$user_defined = 1;\n+\t\tforeach my $category (@tracked_categories) {\n+\t\t\tif (index($category, ':') < 0) {\n+\t\t\t\t# Mediawiki requires the Category\n+\t\t\t\t# prefix, but let's not force the user\n+\t\t\t\t# to specify it.\n+\t\t\t\t$category = \"Category:\" . $category;\n+\t\t\t}\n+\t\t\tmy $mw_pages = $mediawiki->list ( {\n+\t\t\t\taction => 'query',\n+\t\t\t\tlist => 'categorymembers',\n+\t\t\t\tcmtitle => $category,\n+\t\t\t\tcmlimit => 'max' } )\n+\t\t\t || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};\n+\t\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t\t$pages{$page->{title}} = $page;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (!$user_defined) {\n+\t\t# No user-provided list, get the list of pages from\n+\t\t# the API.\n+\t\tmy $mw_pages = $mediawiki->list({\n+\t\t\taction => 'query',\n+\t\t\tlist => 'allpages',\n+\t\t\taplimit => 500,\n+\t\t});\n+\t\tif (!defined($mw_pages)) {\n+\t\t\tprint STDERR \"fatal: could not get the list of wiki pages.\\n\";\n+\t\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\t\texit 1;\n+\t\t}\n+\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t$pages{$page->{title}} = $page;\n+\t\t}\n+\t}\n+\treturn values(%pages);\n+}\n+\n+sub run_git {\n+\topen(my $git, \"-|:encoding(UTF-8)\", \"git \" . $_[0]);\n+\tmy $res = do { local $/; <$git> };\n+\tclose($git);\n+\n+\treturn $res;\n+}\n+\n+\n+sub get_last_local_revision {\n+\t# Get note regarding last mediawiki revision\n+\tmy $note = run_git(\"notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n+\tmy @note_info = split(/ /, $note);\n+\n+\tmy $lastrevision_number;\n+\tif (!(defined($note_info[0]) && $note_info[0] eq \"mediawiki_revision:\")) {\n+\t\tprint STDERR \"No previous mediawiki revision found\";\n+\t\t$lastrevision_number = 0;\n+\t} else {\n+\t\t# Notes are formatted : mediawiki_revision: #number\n+\t\t$lastrevision_number = $note_info[1];\n+\t\tchomp($lastrevision_number);\n+\t\tprint STDERR \"Last local mediawiki revision found is $lastrevision_number\";\n+\t}\n+\treturn $lastrevision_number;\n+}\n+\n+sub get_last_remote_revision {\n+\tmw_connect_maybe();\n+\n+\tmy @pages = get_mw_pages();\n+\n+\tmy $max_rev_num = 0;\n+\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\tpageids => $id,\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});\n+\n+\t\t$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);\n+\t}\n+\n+\tprint STDERR \"Last remote revision found is $max_rev_num.\\n\";\n+\treturn $max_rev_num;\n+}\n+\n+# Clean content before sending it to MediaWiki\n+sub mediawiki_clean {\n+\tmy $string = shift;\n+\tmy $page_created = shift;\n+\t# Mediawiki does not allow blank space at the end of a page and ends with a single \\n.\n+\t# This function right trims a string and adds a \\n at the end to follow this rule\n+\t$string =~ s/\\s+$//;\n+\tif ($string eq \"\" && $page_created) {\n+\t\t# Creating empty pages is forbidden.\n+\t\t$string = EMPTY_CONTENT;\n+\t}\n+\treturn $string.\"\\n\";\n+}\n+\n+# Filter applied on MediaWiki data before adding them to Git\n+sub mediawiki_smudge {\n+\tmy $string = shift;\n+\tif ($string eq EMPTY_CONTENT) {\n+\t\t$string = \"\";\n+\t}\n+\t# This \\n is important. This is due to mediawiki's way to handle end of files.\n+\treturn $string.\"\\n\";\n+}\n+\n+sub mediawiki_clean_filename {\n+\tmy $filename = shift;\n+\t$filename =~ s/@{[SLASH_REPLACEMENT]}/\\//g;\n+\t# [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.\n+\t# Do a variant of URL-encoding, i.e. looks like URL-encoding,\n+\t# but with _ added to prevent MediaWiki from thinking this is\n+\t# an actual special character.\n+\t$filename =~ s/[\\[\\]\\{\\}\\|]/sprintf(\"_%%_%x\", ord($&))/ge;\n+\t# If we use the uri escape before\n+\t# we should unescape here, before anything\n+\n+\treturn $filename;\n+}\n+\n+sub mediawiki_smudge_filename {\n+\tmy $filename = shift;\n+\t$filename =~ s/\\//@{[SLASH_REPLACEMENT]}/g;\n+\t$filename =~ s/ /_/g;\n+\t# Decode forbidden characters encoded in mediawiki_clean_filename\n+\t$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf(\"%c\", hex($1))/ge;\n+\treturn $filename;\n+}\n+\n+sub literal_data {\n+\tmy ($content) = @_;\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+}\n+\n+sub mw_capabilities {\n+\t# Revisions are imported to the private namespace\n+\t# refs/mediawiki/$remotename/ by the helper and fetched into\n+\t# refs/remotes/$remotename later by fetch.\n+\tprint STDOUT \"refspec refs/heads/*:refs/mediawiki/$remotename/*\\n\";\n+\tprint STDOUT \"import\\n\";\n+\tprint STDOUT \"list\\n\";\n+\tprint STDOUT \"push\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_list {\n+\t# MediaWiki do not have branches, we consider one branch arbitrarily\n+\t# called master, and HEAD pointing to it.\n+\tprint STDOUT \"? refs/heads/master\\n\";\n+\tprint STDOUT \"\\@refs/heads/master HEAD\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_option {\n+\tprint STDERR \"remote-helper command 'option $_[0]' not yet implemented\\n\";\n+\tprint STDOUT \"unsupported\\n\";\n+}\n+\n+sub fetch_mw_revisions_for_page {\n+\tmy $page = shift;\n+\tmy $id = shift;\n+\tmy $fetch_from = shift;\n+\tmy @page_revs = ();\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tprop => 'revisions',\n+\t\trvprop => 'ids',\n+\t\trvdir => 'newer',\n+\t\trvstartid => $fetch_from,\n+\t\trvlimit => 500,\n+\t\tpageids => $id,\n+\t};\n+\n+\tmy $revnum = 0;\n+\t# Get 500 revisions at a time due to the mediawiki api limit\n+\twhile (1) {\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\t# Parse each of those 500 revisions\n+\t\tforeach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n+\t\t\tmy $page_rev_ids;\n+\t\t\t$page_rev_ids->{pageid} = $page->{pageid};\n+\t\t\t$page_rev_ids->{revid} = $revision->{revid};\n+\t\t\tpush (@page_revs, $page_rev_ids);\n+\t\t\t$revnum++;\n+\t\t}\n+\t\tlast unless $result->{'query-continue'};\n+\t\t$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n+\t}\n+\tif ($shallow_import && @page_revs) {\n+\t\tprint STDERR \" Found 1 revision (shallow import).\\n\";\n+\t\t@page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);\n+\t\treturn $page_revs[0];\n+\t}\n+\tprint STDERR \" Found \", $revnum, \" revision(s).\\n\";\n+\treturn @page_revs;\n+}\n+\n+sub fetch_mw_revisions {\n+\tmy $pages = shift; my @pages = @{$pages};\n+\tmy $fetch_from = shift;\n+\n+\tmy @revisions = ();\n+\tmy $n = 1;\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tprint STDERR \"page $n/\", scalar(@pages), \": \". $page->{title} .\"\\n\";\n+\t\t$n++;\n+\t\tmy @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);\n+\t\t@revisions = (@page_revs, @revisions);\n+\t}\n+\n+\treturn ($n, @revisions);\n+}\n+\n+sub import_file_revision {\n+\tmy $commit = shift;\n+\tmy %commit = %{$commit};\n+\tmy $full_import = shift;\n+\tmy $n = shift;\n+\n+\tmy $title = $commit{title};\n+\tmy $comment = $commit{comment};\n+\tmy $content = $commit{content};\n+\tmy $author = $commit{author};\n+\tmy $date = $commit{date};\n+\n+\tprint STDOUT \"commit refs/mediawiki/$remotename/master\\n\";\n+\tprint STDOUT \"mark :$n\\n\";\n+\tprint STDOUT \"committer $author <$author\\@$wiki_name> \", $date->epoch, \" +0000\\n\";\n+\tliteral_data($comment);\n+\n+\t# If it's not a clone, we need to know where to start from\n+\tif (!$full_import && $n == 1) {\n+\t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n+\t}\n+\tif ($content ne DELETED_CONTENT) {\n+\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tliteral_data($content);\n+\t\tprint STDOUT \"\\n\\n\";\n+\t} else {\n+\t\tprint STDOUT \"D $title.mw\\n\";\n+\t}\n+\n+\t# mediawiki revision number in the git note\n+\tif ($full_import && $n == 1) {\n+\t\tprint STDOUT \"reset refs/notes/$remotename/mediawiki\\n\";\n+\t}\n+\tprint STDOUT \"commit refs/notes/$remotename/mediawiki\\n\";\n+\tprint STDOUT \"committer $author <$author\\@$wiki_name> \", $date->epoch, \" +0000\\n\";\n+\tliteral_data(\"note added by git-mediawiki\");\n+\tif (!$full_import && $n == 1) {\n+\t\tprint STDOUT \"from refs/notes/$remotename/mediawiki^0\\n\";\n+\t}\n+\tprint STDOUT \"N inline :$n\\n\";\n+\tliteral_data(\"mediawiki_revision: \" . $commit{mw_revision});\n+\tprint STDOUT \"\\n\\n\";\n+}\n+\n+sub mw_import {\n+\t$import_started = 1;\n+\tmy $ref = shift;\n+\t# the remote helper will call \"import HEAD\" and\n+\t# \"import refs/heads/master\"\n+\t# Since HEAD is a symbolic ref to master (by convention,\n+\t# followed by the output of the command \"list\" that we gave),\n+\t# we don't need to do anything in this case.\n+\tif ($ref eq \"HEAD\") {\n+\t\treturn;\n+\t}\n+\n+\tmw_connect_maybe();\n+\n+\tmy @pages = get_mw_pages();\n+\n+\tprint STDERR \"Searching revisions...\\n\";\n+\tmy $last_local = get_last_local_revision();\n+\tmy $fetch_from = $last_local + 1;\n+\tif ($fetch_from == 1) {\n+\t\tprint STDERR \", fetching from beginning.\\n\";\n+\t} else {\n+\t\tprint STDERR \", fetching from here.\\n\";\n+\t}\n+\tmy ($n, @revisions) = fetch_mw_revisions(\\@pages, $fetch_from);\n+\n+\t# Creation of the fast-import stream\n+\tprint STDERR \"Fetching & writing export data...\\n\";\n+\n+\t$n = 0;\n+\tmy $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined\n+\n+\tforeach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n+\t\t# fetch the content of the pages\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'content|timestamp|comment|user|ids',\n+\t\t\trevids => $pagerevid->{revid},\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});\n+\n+\t\t$n++;\n+\n+\t\tmy %commit;\n+\t\t$commit{author} = $rev->{user} || 'Anonymous';\n+\t\t$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';\n+\t\t$commit{title} = mediawiki_smudge_filename(\n+\t\t\t$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}\n+\t\t );\n+\t\t$commit{mw_revision} = $pagerevid->{revid};\n+\t\t$commit{content} = mediawiki_smudge($rev->{'*'});\n+\n+\t\tif (!defined($rev->{timestamp})) {\n+\t\t\t$last_timestamp++;\n+\t\t} else {\n+\t\t\t$last_timestamp = $rev->{timestamp};\n+\t\t}\n+\t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n+\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\n+\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t}\n+\n+\tif ($fetch_from == 1) {\n+\t\tif ($n != 0) {\n+\t\t\tprint STDOUT \"reset $ref\\n\";\n+\t\t\tprint STDOUT \"from :$n\\n\";\n+\t\t} else {\n+\t\t\tprint STDERR \"You appear to have cloned an empty mediawiki\\n\";\n+\t\t\t# Something has to be done remote-helper side. If nothing is done, an error is\n+\t\t\t# thrown saying that HEAD is refering to unknown object 0000000000000000000\n+\t\t}\n+\t}\n+}\n+\n+sub error_non_fast_forward {\n+\t# Native git-push would show this after the summary.\n+\t# We can't ask it to display it cleanly, so print it\n+\t# ourselves before.\n+\tprint STDERR \"To prevent you from losing history, non-fast-forward updates were rejected\\n\";\n+\tprint STDERR \"Merge the remote changes (e.g. 'git pull') before pushing again. See the\\n\";\n+\tprint STDERR \"'Note about fast-forwards' section of 'git push --help' for details.\\n\";\n+\n+\tprint STDOUT \"error $_[0] \\\"non-fast-forward\\\"\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_push_file {\n+\tmy $diff_info = shift;\n+\t# $diff_info contains a string in this format:\n+\t# 100644 100644 \n+\tmy @diff_info_split = split(/[ \\t]/, $diff_info);\n+\n+\t# Filename, including .mw extension\n+\tmy $complete_file_name = shift;\n+\t# Commit message\n+\tmy $summary = shift;\n+\n+\tmy $new_sha1 = $diff_info_split[3];\n+\tmy $old_sha1 = $diff_info_split[2];\n+\tmy $page_created = ($old_sha1 eq NULL_SHA1);\n+\tmy $page_deleted = ($new_sha1 eq NULL_SHA1);\n+\t$complete_file_name = mediawiki_clean_filename($complete_file_name);\n+\n+\tif (substr($complete_file_name,-3) eq \".mw\"){\n+\t\tmy $title = substr($complete_file_name,0,-3);\n+\n+\t\tmy $file_content;\n+\t\tif ($page_deleted) {\n+\t\t\t# Deleting a page usually requires\n+\t\t\t# special priviledges. A common\n+\t\t\t# convention is to replace the page\n+\t\t\t# with this content instead:\n+\t\t\t$file_content = DELETED_CONTENT;\n+\t\t} else {\n+\t\t\t$file_content = run_git(\"cat-file -p $new_sha1\");\n+\t\t}\n+\n+\t\tmw_connect_maybe();\n+\n+\t\tmy $result = $mediawiki->edit( {\n+\t\t\taction => 'edit',\n+\t\t\tsummary => $summary,\n+\t\t\ttitle => $title,\n+\t\t\ttext => mediawiki_clean($file_content, $page_created),\n+\t\t\t\t }, {\n+\t\t\t\t\t skip_encoding => 1 # Helps with names with accentuated characters\n+\t\t\t\t }) || die 'Fatal: Error ' .\n+\t\t\t\t $mediawiki->{error}->{code} .\n+\t\t\t\t ' from mediwiki: ' . $mediawiki->{error}->{details};\n+\t\tprint STDERR \"Pushed file : $new_sha1 - $title\\n\";\n+\t} else {\n+\t\tprint STDERR \"$complete_file_name not a mediawiki file (Not pushable on this version).\\n\"\n+\t}\n+}\n+\n+sub mw_push {\n+\tmy $last_local_revid = get_last_local_revision();\n+\tprint STDERR \".\\n\"; # Finish sentence started by get_last_local_revision()\n+\tmy $last_remote_revid = get_last_remote_revision();\n+\n+\t# Get sha1 of commit pointed by local HEAD\n+\tmy $HEAD_sha1 = run_git(\"rev-parse $_[0] 2>/dev/null\"); chomp($HEAD_sha1);\n+\t# Get sha1 of commit pointed by remotes/$remotename/master\n+\tmy $remoteorigin_sha1 = run_git(\"rev-parse refs/remotes/$remotename/master 2>/dev/null\");\n+\tchomp($remoteorigin_sha1);\n+\n+\tif ($last_local_revid > 0 &&\n+\t $last_local_revid < $last_remote_revid){\n+\t\treturn error_non_fast_forward($_[0]);\n+\t}\n+\n+\tif ($HEAD_sha1 eq $remoteorigin_sha1) {\n+\t\tprint STDOUT \"\\n\";\n+\t\treturn;\n+\t}\n+\n+\t# Get every commit in between HEAD and refs/remotes/origin/master,\n+\t# including HEAD and refs/remotes/origin/master\n+\tmy @commit_pairs = ();\n+\tif ($last_local_revid > 0) {\n+\t\tmy $parsed_sha1 = $remoteorigin_sha1;\n+\t\t# Find a path from last MediaWiki commit to pushed commit\n+\t\twhile ($parsed_sha1 ne $HEAD_sha1) {\n+\t\t\tmy @commit_info = grep(/^$parsed_sha1/, split(/\\n/, run_git(\"rev-list --children $_[0]\")));\n+\t\t\tif (!@commit_info) {\n+\t\t\t\treturn error_non_fast_forward($_[0]);\n+\t\t\t}\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $commit_info[0]);\n+\t\t\t# $commit_info_split[1] is the sha1 of the commit to export\n+\t\t\t# $commit_info_split[0] is the sha1 of its direct child\n+\t\t\tpush (@commit_pairs, \\@commit_info_split);\n+\t\t\t$parsed_sha1 = $commit_info_split[1];\n+\t\t}\n+\t} else {\n+\t\t# No remote mediawiki revision. Export the whole\n+\t\t# history (linearized with --first-parent)\n+\t\tprint STDERR \"Warning: no common ancestor, pushing complete history\\n\";\n+\t\tmy $history = run_git(\"rev-list --first-parent --children $_[0]\");\n+\t\tmy @history = split('\\n', $history);\n+\t\t@history = @history[1..$#history];\n+\t\tforeach my $line (reverse @history) {\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $line);\n+\t\t\tpush (@commit_pairs, \\@commit_info_split);\n+\t\t}\n+\t}\n+\n+\tforeach my $commit_info_split (@commit_pairs) {\n+\t\tmy $sha1_child = @{$commit_info_split}[0];\n+\t\tmy $sha1_commit = @{$commit_info_split}[1];\n+\t\tmy $diff_infos = run_git(\"diff-tree -r --raw -z $sha1_child $sha1_commit\");\n+\t\t# TODO: we could detect rename, and encode them with a #redirect on the wiki.\n+\t\t# TODO: for now, it's just a delete+add\n+\t\tmy @diff_info_list = split(/\\0/, $diff_infos);\n+\t\t# Keep the first line of the commit message as mediawiki comment for the revision\n+\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $sha1_commit\")))[0];\n+\t\tchomp($commit_msg);\n+\t\t# Push every blob\n+\t\twhile (@diff_info_list) {\n+\t\t\t# git diff-tree -z gives an output like\n+\t\t\t# \\0\\0\n+\t\t\t# \\0\\0\n+\t\t\t# and we've split on \\0.\n+\t\t\tmy $info = shift(@diff_info_list);\n+\t\t\tmy $file = shift(@diff_info_list);\n+\t\t\tmw_push_file($info, $file, $commit_msg);\n+\t\t}\n+\t}\n+\n+\tprint STDOUT \"ok $_[1]\\n\";\n+\tprint STDOUT \"\\n\";\n+\n+\tprint STDERR \"Just pushed some revisions to MediaWiki.\\n\";\n+\tprint STDERR \"The pushed revisions now have to be re-imported, and your current branch\\n\";\n+\tprint STDERR \"needs to be updated with these re-imported commits. You can do this with\\n\";\n+\tprint STDERR \"\\n\";\n+\tprint STDERR \" git pull --rebase\\n\";\n+\tprint STDERR \"\\n\";\n+}\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt\nnew file mode 100644\nindex 0000000..4d211f5\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki.txt\n@@ -0,0 +1,7 @@\n+Git-Mediawiki is a project which aims the creation of a gate\n+between git and mediawiki, allowing git users to push and pull\n+objects from mediawiki just as one would do with a classic git\n+repository thanks to remote-helpers.\n+\n+For more information, visit the wiki at\n+https://github.com/Bibzball/Git-Mediawiki/wiki\n-- \n1.7.6.585.g5929f.dirty\n"},{"id":"174333","messageId":"7v8vqgdpsv.fsf@alter.siamese.dyndns.org","threadId":"28226","inReplyTo":"1314378689-8997-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-08-26T17:51:28Z","receivedAt":"2011-08-26T17:51:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> The uninitialized variable seems harmless in practice, but let's still be clean.\n\nIt is not \"in practice\", but by definition, file scope \"static int\"\nvariables are initialized to 0 by the C language (a typical implementation\nachieves this by placing the variable in BSS section).\n\nPlease do not write unnecessary \" = 0\" there.\n\n>\n> Signed-off-by: Matthieu Moy \n> ---\n> For some reason, remote helpers seem to be forced to use the \"done\"\n> command now. Investing why, I found this, but that wasn't what I was\n> looking for.\n>\n> fast-import.c | 2 +-\n> 1 files changed, 1 insertions(+), 1 deletions(-)\n>\n> diff --git a/fast-import.c b/fast-import.c\n> index 7cc2262..ed8f3cd 100644\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -355,7 +355,7 @@ static unsigned int cmd_save = 100;\n> static uintmax_t next_mark;\n> static struct strbuf new_data = STRBUF_INIT;\n> static int seen_data_command;\n> -static int require_explicit_termination;\n> +static int require_explicit_termination = 0;\n> \n> /* Signal handling */\n> static volatile sig_atomic_t checkpoint_requested;\n"},{"id":"174334","messageId":"7v4o14dppz.fsf@alter.siamese.dyndns.org","threadId":"28226","inReplyTo":"1314378689-8997-2-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-08-26T17:53:12Z","receivedAt":"2011-08-26T17:53:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> In short, the changes since v3 are:\n>\n> * Adapt to newer Git, which seem to require a \"done\" command at the\n> end of the fast-import stream. I don't understand why this is\n> needed, since fast-import is called without the --done flag by\n> remote-helpers, but if I don't do this, \"git fetch\" doesn't\n> terminate and keeps waiting ...\n\nHmmmm, is this a regression in fast-import? Can this be bisected if so?\n"},{"id":"174335","messageId":"1314381329-8989-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"1314378689-8997-2-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH v5] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-08-26T17:55:29Z","receivedAt":"2011-08-26T17:55:29Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: Jeremie Nikaes \n\nImplement a gate between git and mediawiki, allowing git users to push\nand pull objects from mediawiki just as one would do with a classic git\nrepository thanks to remote-helpers.\n\nThe following packages need to be installed (available on common\nrepositories):\n\n libmediawiki-api-perl\n libdatetime-format-iso8601-perl\n\nUse remote helpers in order to be as transparent as possible to the git\nuser.\n\nDownload Mediawiki revisions through the Mediawiki API and then\nfast-import into git.\n\nMediawiki revision number and git commits are linked thanks to notes\nbound to commits.\n\nThe import part is done on a refs/mediawiki/ branch before\ncoming to refs/remote/origin/master (Huge thanks to Jonathan Nieder\nfor his help)\n\nWe use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but\nwe also read the output of Git commands in UTF-8 with the small helper\nrun_git, and write to the console (STDERR) in UTF-8. This allows a\nseamless use of non-ascii characters in page titles, but hasn't been\ntested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames\ncould raise problems if different file systems handle UTF-8 filenames\ndifferently. A uri_escape of mediawiki filenames could be imaginable, and\nis still to be discussed further.\n\nPartial cloning is supported using one of:\n\ngit clone -c remote.origin.pages='A_Page Another_Page' mediawiki::http://wikiurl\n\ngit clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl\n\ngit clone -c remote.origin.shallow='True' mediawiki::http://wikiurl\n\nThanks to notes metadata, it is possible to compare remote and local last\nmediawiki revision to warn non-fast forward pushes and \"everything\nup-to-date\" case.\n\nWhen allowed, push looks for each commit between remotes/origin/master\nand HEAD, catches every blob related to these commit and push them in\nchronological order. To do so, it uses git rev-list --children HEAD and\ntravels the tree from remotes/origin/master to HEAD through children. In\nother words :\n\n\t* Shortest path from remotes/origin/master to HEAD\n\t* For each commit encountered, push blobs related to this commit\n\nSigned-off-by: Jérémie Nikaes \nSigned-off-by: Arnaud Lacurie \nSigned-off-by: Claire Fousse \nSigned-off-by: David Amouyal \nSigned-off-by: Matthieu Moy \nSigned-off-by: Sylvain Boulmé \nSigned-off-by: Matthieu Moy \n---\nOops, I had an uncommited fix when I sent the previous patch. Change\nsince v4 is just this:\n\n- if ($fetch_from == 1) {\n- if ($n != 0) {\n- print STDOUT \"reset $ref\\n\";\n- print STDOUT \"from :$n\\n\";\n- } else {\n- print STDERR \"You appear to have cloned an empty mediawiki\\n\";\n- # Something has to be done remote-helper side. If nothing is done, an error is\n- # thrown saying that HEAD is refering to unknown object 0000000000000000000\n- }\n+ if ($fetch_from == 1 && $n == 0) {\n+ print STDERR \"You appear to have cloned an empty MediaWiki.\\n\";\n+ # Something has to be done remote-helper side. If nothing is done, an error is\n+ # thrown saying that HEAD is refering to unknown object 0000000000000000000\n+ # and the clone fails.\n\nSorry for the noise.\n\n contrib/mw-to-git/git-remote-mediawiki | 718 ++++++++++++++++++++++++++++\n contrib/mw-to-git/git-remote-mediawiki.txt | 7 +\n 2 files changed, 725 insertions(+), 0 deletions(-)\n create mode 100755 contrib/mw-to-git/git-remote-mediawiki\n create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nnew file mode 100755\nindex 0000000..e24a640\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -0,0 +1,718 @@\n+#! /usr/bin/perl\n+\n+# Copyright (C) 2011\n+# Jérémie Nikaes \n+# Arnaud Lacurie \n+# Claire Fousse \n+# David Amouyal \n+# Matthieu Moy \n+# License: GPL v2 or later\n+\n+# Gateway between Git and MediaWiki.\n+# https://github.com/Bibzball/Git-Mediawiki/wiki\n+#\n+# Known limitations:\n+#\n+# - Only wiki pages are managed, no support for [[File:...]]\n+# attachments.\n+#\n+# - Poor performance in the best case: it takes forever to check\n+# whether we're up-to-date (on fetch or push) or to fetch a few\n+# revisions from a large wiki, because we use exclusively a\n+# page-based synchronization. We could switch to a wiki-wide\n+# synchronization when the synchronization involves few revisions\n+# but the wiki is large.\n+#\n+# - Git renames could be turned into MediaWiki renames (see TODO\n+# below)\n+#\n+# - login/password support requires the user to write the password\n+# cleartext in a file (see TODO below).\n+#\n+# - No way to import \"one page, and all pages included in it\"\n+#\n+# - Multiple remote MediaWikis have not been very well tested.\n+\n+use strict;\n+use MediaWiki::API;\n+use DateTime::Format::ISO8601;\n+use encoding 'utf8';\n+\n+# use encoding 'utf8' doesn't change STDERROR\n+# but we're going to output UTF-8 filenames to STDERR\n+binmode STDERR, \":utf8\";\n+\n+use URI::Escape;\n+use warnings;\n+\n+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced\n+use constant SLASH_REPLACEMENT => \"%2F\";\n+\n+# It's not always possible to delete pages (may require some\n+# priviledges). Deleted pages are replaced with this content.\n+use constant DELETED_CONTENT => \"[[Category:Deleted]]\\n\";\n+\n+# It's not possible to create empty pages. New empty files in Git are\n+# sent with this content instead.\n+use constant EMPTY_CONTENT => \"\\n\";\n+\n+# used to reflect file creation or deletion in diff.\n+use constant NULL_SHA1 => \"0000000000000000000000000000000000000000\";\n+\n+my $remotename = $ARGV[0];\n+my $url = $ARGV[1];\n+\n+# Accept both space-separated and multiple keys in config file.\n+# Spaces should be written as _ anyway because we'll use chomp.\n+my @tracked_pages = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".pages\"));\n+chomp(@tracked_pages);\n+\n+# Just like @tracked_pages, but for MediaWiki categories.\n+my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".categories\"));\n+chomp(@tracked_categories);\n+\n+my $wiki_login = run_git(\"config --get remote.\". $remotename .\".mwLogin\");\n+# TODO: ideally, this should be able to read from keyboard, but we're\n+# inside a remote helper, so our stdin is connect to git, not to a\n+# terminal.\n+my $wiki_passwd = run_git(\"config --get remote.\". $remotename .\".mwPassword\");\n+chomp ($wiki_login);\n+chomp ($wiki_passwd);\n+\n+# Import only last revisions (both for clone and fetch)\n+my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shallow\");\n+chomp($shallow_import);\n+$shallow_import = ($shallow_import eq \"true\");\n+\n+my $wiki_name = $url;\n+$wiki_name =~ s/[^\\/]*:\\/\\///;\n+\n+my $import_started;\n+\n+# Commands parser\n+my $entry;\n+my @cmd;\n+while () {\n+\tchomp;\n+\t@cmd = split(/ /);\n+\tif (defined($cmd[0])) {\n+\t\t# Line not blank\n+\t\tif ($cmd[0] eq \"capabilities\") {\n+\t\t\tdie(\"Too many arguments for capabilities\") unless (!defined($cmd[1]));\n+\t\t\tmw_capabilities();\n+\t\t} elsif ($cmd[0] eq \"list\") {\n+\t\t\tdie(\"Too many arguments for list\") unless (!defined($cmd[2]));\n+\t\t\tmw_list($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"import\") {\n+\t\t\tdie(\"Invalid arguments for import\") unless ($cmd[1] ne \"\" && !defined($cmd[2]));\n+\t\t\tmw_import($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"option\") {\n+\t\t\tdie(\"Too many arguments for option\") unless ($cmd[1] ne \"\" && $cmd[2] ne \"\" && !defined($cmd[3]));\n+\t\t\tmw_option($cmd[1],$cmd[2]);\n+\t\t} elsif ($cmd[0] eq \"push\") {\n+\t\t\t# Check the pattern :\n+\t\t\tmy @pushargs = split(/:/,$cmd[1]);\n+\t\t\tdie(\"Invalid arguments for push\") unless ($pushargs[1] ne \"\" && !defined($pushargs[2]));\n+\t\t\tmw_push($pushargs[0],$pushargs[1]);\n+\t\t} else {\n+\t\t\tprint STDERR \"Unknown command. Aborting...\\n\";\n+\t\t\tlast;\n+\t\t}\n+\t} else {\n+\t\t# blank line: we should terminate\n+\t\tlast;\n+\t}\n+\n+\tBEGIN { $| = 1 } # flush STDOUT, to make sure the previous\n+\t\t\t # command is fully processed.\n+}\n+# End of input\n+if ($import_started) {\n+\t# Terminate the fast-import stream properly.\n+\t# Git requires one \"done\" command, and only\n+\t# one This is OK since we only have one\n+\t# branch, so import will be called only once\n+\t# (plus once for HEAD, for which we won't\n+\t# reach this point).\n+\tprint STDOUT \"done\\n\";\n+}\n+BEGIN { $| = 1 };\n+if (!eof(STDIN)) {\n+\t# Wait for Git to terminate. If we don't, git fetch\n+\t# (transport-helper.c's sendline function) will try to write\n+\t# to our stdin, which will be closed, and git fetch will be\n+\t# killed. That's probably a bug in transport-helper.c, but in\n+\t# the meantime ...\n+\tsleep .1;\n+};\n+\n+########################## Functions ##############################\n+\n+# MediaWiki API instance, created lazily.\n+my $mediawiki;\n+\n+sub mw_connect_maybe {\n+\tif ($mediawiki) {\n+\t return;\n+\t}\n+\t$mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\tif ($wiki_login) {\n+\t\tif (!$mediawiki->login({\n+\t\t\tlgname => $wiki_login,\n+\t\t\tlgpassword => $wiki_passwd,\n+\t\t})) {\n+\t\t\tprint STDERR \"Failed to log in mediawiki user \\\"$wiki_login\\\" on $url\\n\";\n+\t\t\tprint STDERR \"(error \" .\n+\t\t\t $mediawiki->{error}->{code} . ': ' .\n+\t\t\t $mediawiki->{error}->{details} . \")\\n\";\n+\t\t\texit 1;\n+\t\t} else {\n+\t\t\tprint STDERR \"Logged in with user \\\"$wiki_login\\\".\\n\";\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_first_pages {\n+\tmy $some_pages = shift;\n+\tmy @some_pages = @{$some_pages};\n+\n+\tmy $pages = shift;\n+\n+\t# pattern 'page1|page2|...' required by the API\n+\tmy $titles = join('|', @some_pages);\n+\n+\tmy $mw_pages = $mediawiki->api({\n+\t\taction => 'query',\n+\t\ttitles => $titles,\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not query the list of wiki pages.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\twhile (my ($id, $page) = each (%{$mw_pages->{query}->{pages}})) {\n+\t\tif ($id < 0) {\n+\t\t\tprint STDERR \"Warning: page $page->{title} not found on wiki\\n\";\n+\t\t} else {\n+\t\t\t$pages->{$page->{title}} = $page;\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_pages {\n+\tmw_connect_maybe();\n+\n+\tmy %pages; # hash on page titles to avoid duplicates\n+\tmy $user_defined;\n+\tif (@tracked_pages) {\n+\t\t$user_defined = 1;\n+\t\t# The user provided a list of pages titles, but we\n+\t\t# still need to query the API to get the page IDs.\n+\n+\t\tmy @some_pages = @tracked_pages;\n+\t\twhile (@some_pages) {\n+\t\t\tmy $last = 50;\n+\t\t\tif ($#some_pages < $last) {\n+\t\t\t\t$last = $#some_pages;\n+\t\t\t}\n+\t\t\tmy @slice = @some_pages[0..$last];\n+\t\t\tget_mw_first_pages(\\@slice, \\%pages);\n+\t\t\t@some_pages = @some_pages[51..$#some_pages];\n+\t\t}\n+\t}\n+\tif (@tracked_categories) {\n+\t\t$user_defined = 1;\n+\t\tforeach my $category (@tracked_categories) {\n+\t\t\tif (index($category, ':') < 0) {\n+\t\t\t\t# Mediawiki requires the Category\n+\t\t\t\t# prefix, but let's not force the user\n+\t\t\t\t# to specify it.\n+\t\t\t\t$category = \"Category:\" . $category;\n+\t\t\t}\n+\t\t\tmy $mw_pages = $mediawiki->list ( {\n+\t\t\t\taction => 'query',\n+\t\t\t\tlist => 'categorymembers',\n+\t\t\t\tcmtitle => $category,\n+\t\t\t\tcmlimit => 'max' } )\n+\t\t\t || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};\n+\t\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t\t$pages{$page->{title}} = $page;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (!$user_defined) {\n+\t\t# No user-provided list, get the list of pages from\n+\t\t# the API.\n+\t\tmy $mw_pages = $mediawiki->list({\n+\t\t\taction => 'query',\n+\t\t\tlist => 'allpages',\n+\t\t\taplimit => 500,\n+\t\t});\n+\t\tif (!defined($mw_pages)) {\n+\t\t\tprint STDERR \"fatal: could not get the list of wiki pages.\\n\";\n+\t\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\t\texit 1;\n+\t\t}\n+\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t$pages{$page->{title}} = $page;\n+\t\t}\n+\t}\n+\treturn values(%pages);\n+}\n+\n+sub run_git {\n+\topen(my $git, \"-|:encoding(UTF-8)\", \"git \" . $_[0]);\n+\tmy $res = do { local $/; <$git> };\n+\tclose($git);\n+\n+\treturn $res;\n+}\n+\n+\n+sub get_last_local_revision {\n+\t# Get note regarding last mediawiki revision\n+\tmy $note = run_git(\"notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n+\tmy @note_info = split(/ /, $note);\n+\n+\tmy $lastrevision_number;\n+\tif (!(defined($note_info[0]) && $note_info[0] eq \"mediawiki_revision:\")) {\n+\t\tprint STDERR \"No previous mediawiki revision found\";\n+\t\t$lastrevision_number = 0;\n+\t} else {\n+\t\t# Notes are formatted : mediawiki_revision: #number\n+\t\t$lastrevision_number = $note_info[1];\n+\t\tchomp($lastrevision_number);\n+\t\tprint STDERR \"Last local mediawiki revision found is $lastrevision_number\";\n+\t}\n+\treturn $lastrevision_number;\n+}\n+\n+sub get_last_remote_revision {\n+\tmw_connect_maybe();\n+\n+\tmy @pages = get_mw_pages();\n+\n+\tmy $max_rev_num = 0;\n+\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\tpageids => $id,\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});\n+\n+\t\t$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);\n+\t}\n+\n+\tprint STDERR \"Last remote revision found is $max_rev_num.\\n\";\n+\treturn $max_rev_num;\n+}\n+\n+# Clean content before sending it to MediaWiki\n+sub mediawiki_clean {\n+\tmy $string = shift;\n+\tmy $page_created = shift;\n+\t# Mediawiki does not allow blank space at the end of a page and ends with a single \\n.\n+\t# This function right trims a string and adds a \\n at the end to follow this rule\n+\t$string =~ s/\\s+$//;\n+\tif ($string eq \"\" && $page_created) {\n+\t\t# Creating empty pages is forbidden.\n+\t\t$string = EMPTY_CONTENT;\n+\t}\n+\treturn $string.\"\\n\";\n+}\n+\n+# Filter applied on MediaWiki data before adding them to Git\n+sub mediawiki_smudge {\n+\tmy $string = shift;\n+\tif ($string eq EMPTY_CONTENT) {\n+\t\t$string = \"\";\n+\t}\n+\t# This \\n is important. This is due to mediawiki's way to handle end of files.\n+\treturn $string.\"\\n\";\n+}\n+\n+sub mediawiki_clean_filename {\n+\tmy $filename = shift;\n+\t$filename =~ s/@{[SLASH_REPLACEMENT]}/\\//g;\n+\t# [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.\n+\t# Do a variant of URL-encoding, i.e. looks like URL-encoding,\n+\t# but with _ added to prevent MediaWiki from thinking this is\n+\t# an actual special character.\n+\t$filename =~ s/[\\[\\]\\{\\}\\|]/sprintf(\"_%%_%x\", ord($&))/ge;\n+\t# If we use the uri escape before\n+\t# we should unescape here, before anything\n+\n+\treturn $filename;\n+}\n+\n+sub mediawiki_smudge_filename {\n+\tmy $filename = shift;\n+\t$filename =~ s/\\//@{[SLASH_REPLACEMENT]}/g;\n+\t$filename =~ s/ /_/g;\n+\t# Decode forbidden characters encoded in mediawiki_clean_filename\n+\t$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf(\"%c\", hex($1))/ge;\n+\treturn $filename;\n+}\n+\n+sub literal_data {\n+\tmy ($content) = @_;\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+}\n+\n+sub mw_capabilities {\n+\t# Revisions are imported to the private namespace\n+\t# refs/mediawiki/$remotename/ by the helper and fetched into\n+\t# refs/remotes/$remotename later by fetch.\n+\tprint STDOUT \"refspec refs/heads/*:refs/mediawiki/$remotename/*\\n\";\n+\tprint STDOUT \"import\\n\";\n+\tprint STDOUT \"list\\n\";\n+\tprint STDOUT \"push\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_list {\n+\t# MediaWiki do not have branches, we consider one branch arbitrarily\n+\t# called master, and HEAD pointing to it.\n+\tprint STDOUT \"? refs/heads/master\\n\";\n+\tprint STDOUT \"\\@refs/heads/master HEAD\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_option {\n+\tprint STDERR \"remote-helper command 'option $_[0]' not yet implemented\\n\";\n+\tprint STDOUT \"unsupported\\n\";\n+}\n+\n+sub fetch_mw_revisions_for_page {\n+\tmy $page = shift;\n+\tmy $id = shift;\n+\tmy $fetch_from = shift;\n+\tmy @page_revs = ();\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tprop => 'revisions',\n+\t\trvprop => 'ids',\n+\t\trvdir => 'newer',\n+\t\trvstartid => $fetch_from,\n+\t\trvlimit => 500,\n+\t\tpageids => $id,\n+\t};\n+\n+\tmy $revnum = 0;\n+\t# Get 500 revisions at a time due to the mediawiki api limit\n+\twhile (1) {\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\t# Parse each of those 500 revisions\n+\t\tforeach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n+\t\t\tmy $page_rev_ids;\n+\t\t\t$page_rev_ids->{pageid} = $page->{pageid};\n+\t\t\t$page_rev_ids->{revid} = $revision->{revid};\n+\t\t\tpush (@page_revs, $page_rev_ids);\n+\t\t\t$revnum++;\n+\t\t}\n+\t\tlast unless $result->{'query-continue'};\n+\t\t$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n+\t}\n+\tif ($shallow_import && @page_revs) {\n+\t\tprint STDERR \" Found 1 revision (shallow import).\\n\";\n+\t\t@page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);\n+\t\treturn $page_revs[0];\n+\t}\n+\tprint STDERR \" Found \", $revnum, \" revision(s).\\n\";\n+\treturn @page_revs;\n+}\n+\n+sub fetch_mw_revisions {\n+\tmy $pages = shift; my @pages = @{$pages};\n+\tmy $fetch_from = shift;\n+\n+\tmy @revisions = ();\n+\tmy $n = 1;\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tprint STDERR \"page $n/\", scalar(@pages), \": \". $page->{title} .\"\\n\";\n+\t\t$n++;\n+\t\tmy @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);\n+\t\t@revisions = (@page_revs, @revisions);\n+\t}\n+\n+\treturn ($n, @revisions);\n+}\n+\n+sub import_file_revision {\n+\tmy $commit = shift;\n+\tmy %commit = %{$commit};\n+\tmy $full_import = shift;\n+\tmy $n = shift;\n+\n+\tmy $title = $commit{title};\n+\tmy $comment = $commit{comment};\n+\tmy $content = $commit{content};\n+\tmy $author = $commit{author};\n+\tmy $date = $commit{date};\n+\n+\tprint STDOUT \"commit refs/mediawiki/$remotename/master\\n\";\n+\tprint STDOUT \"mark :$n\\n\";\n+\tprint STDOUT \"committer $author <$author\\@$wiki_name> \", $date->epoch, \" +0000\\n\";\n+\tliteral_data($comment);\n+\n+\t# If it's not a clone, we need to know where to start from\n+\tif (!$full_import && $n == 1) {\n+\t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n+\t}\n+\tif ($content ne DELETED_CONTENT) {\n+\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tliteral_data($content);\n+\t\tprint STDOUT \"\\n\\n\";\n+\t} else {\n+\t\tprint STDOUT \"D $title.mw\\n\";\n+\t}\n+\n+\t# mediawiki revision number in the git note\n+\tif ($full_import && $n == 1) {\n+\t\tprint STDOUT \"reset refs/notes/$remotename/mediawiki\\n\";\n+\t}\n+\tprint STDOUT \"commit refs/notes/$remotename/mediawiki\\n\";\n+\tprint STDOUT \"committer $author <$author\\@$wiki_name> \", $date->epoch, \" +0000\\n\";\n+\tliteral_data(\"note added by git-mediawiki\");\n+\tif (!$full_import && $n == 1) {\n+\t\tprint STDOUT \"from refs/notes/$remotename/mediawiki^0\\n\";\n+\t}\n+\tprint STDOUT \"N inline :$n\\n\";\n+\tliteral_data(\"mediawiki_revision: \" . $commit{mw_revision});\n+\tprint STDOUT \"\\n\\n\";\n+}\n+\n+sub mw_import {\n+\t$import_started = 1;\n+\tmy $ref = shift;\n+\t# the remote helper will call \"import HEAD\" and\n+\t# \"import refs/heads/master\"\n+\t# Since HEAD is a symbolic ref to master (by convention,\n+\t# followed by the output of the command \"list\" that we gave),\n+\t# we don't need to do anything in this case.\n+\tif ($ref eq \"HEAD\") {\n+\t\treturn;\n+\t}\n+\n+\tmw_connect_maybe();\n+\n+\tmy @pages = get_mw_pages();\n+\n+\tprint STDERR \"Searching revisions...\\n\";\n+\tmy $last_local = get_last_local_revision();\n+\tmy $fetch_from = $last_local + 1;\n+\tif ($fetch_from == 1) {\n+\t\tprint STDERR \", fetching from beginning.\\n\";\n+\t} else {\n+\t\tprint STDERR \", fetching from here.\\n\";\n+\t}\n+\tmy ($n, @revisions) = fetch_mw_revisions(\\@pages, $fetch_from);\n+\n+\t# Creation of the fast-import stream\n+\tprint STDERR \"Fetching & writing export data...\\n\";\n+\n+\t$n = 0;\n+\tmy $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined\n+\n+\tforeach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n+\t\t# fetch the content of the pages\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'content|timestamp|comment|user|ids',\n+\t\t\trevids => $pagerevid->{revid},\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});\n+\n+\t\t$n++;\n+\n+\t\tmy %commit;\n+\t\t$commit{author} = $rev->{user} || 'Anonymous';\n+\t\t$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';\n+\t\t$commit{title} = mediawiki_smudge_filename(\n+\t\t\t$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}\n+\t\t );\n+\t\t$commit{mw_revision} = $pagerevid->{revid};\n+\t\t$commit{content} = mediawiki_smudge($rev->{'*'});\n+\n+\t\tif (!defined($rev->{timestamp})) {\n+\t\t\t$last_timestamp++;\n+\t\t} else {\n+\t\t\t$last_timestamp = $rev->{timestamp};\n+\t\t}\n+\t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n+\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\n+\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t}\n+\n+\tif ($fetch_from == 1 && $n == 0) {\n+\t\tprint STDERR \"You appear to have cloned an empty MediaWiki.\\n\";\n+\t\t# Something has to be done remote-helper side. If nothing is done, an error is\n+\t\t# thrown saying that HEAD is refering to unknown object 0000000000000000000\n+\t\t# and the clone fails.\n+\t}\n+}\n+\n+sub error_non_fast_forward {\n+\t# Native git-push would show this after the summary.\n+\t# We can't ask it to display it cleanly, so print it\n+\t# ourselves before.\n+\tprint STDERR \"To prevent you from losing history, non-fast-forward updates were rejected\\n\";\n+\tprint STDERR \"Merge the remote changes (e.g. 'git pull') before pushing again. See the\\n\";\n+\tprint STDERR \"'Note about fast-forwards' section of 'git push --help' for details.\\n\";\n+\n+\tprint STDOUT \"error $_[0] \\\"non-fast-forward\\\"\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_push_file {\n+\tmy $diff_info = shift;\n+\t# $diff_info contains a string in this format:\n+\t# 100644 100644 \n+\tmy @diff_info_split = split(/[ \\t]/, $diff_info);\n+\n+\t# Filename, including .mw extension\n+\tmy $complete_file_name = shift;\n+\t# Commit message\n+\tmy $summary = shift;\n+\n+\tmy $new_sha1 = $diff_info_split[3];\n+\tmy $old_sha1 = $diff_info_split[2];\n+\tmy $page_created = ($old_sha1 eq NULL_SHA1);\n+\tmy $page_deleted = ($new_sha1 eq NULL_SHA1);\n+\t$complete_file_name = mediawiki_clean_filename($complete_file_name);\n+\n+\tif (substr($complete_file_name,-3) eq \".mw\"){\n+\t\tmy $title = substr($complete_file_name,0,-3);\n+\n+\t\tmy $file_content;\n+\t\tif ($page_deleted) {\n+\t\t\t# Deleting a page usually requires\n+\t\t\t# special priviledges. A common\n+\t\t\t# convention is to replace the page\n+\t\t\t# with this content instead:\n+\t\t\t$file_content = DELETED_CONTENT;\n+\t\t} else {\n+\t\t\t$file_content = run_git(\"cat-file -p $new_sha1\");\n+\t\t}\n+\n+\t\tmw_connect_maybe();\n+\n+\t\tmy $result = $mediawiki->edit( {\n+\t\t\taction => 'edit',\n+\t\t\tsummary => $summary,\n+\t\t\ttitle => $title,\n+\t\t\ttext => mediawiki_clean($file_content, $page_created),\n+\t\t\t\t }, {\n+\t\t\t\t\t skip_encoding => 1 # Helps with names with accentuated characters\n+\t\t\t\t }) || die 'Fatal: Error ' .\n+\t\t\t\t $mediawiki->{error}->{code} .\n+\t\t\t\t ' from mediwiki: ' . $mediawiki->{error}->{details};\n+\t\tprint STDERR \"Pushed file : $new_sha1 - $title\\n\";\n+\t} else {\n+\t\tprint STDERR \"$complete_file_name not a mediawiki file (Not pushable on this version).\\n\"\n+\t}\n+}\n+\n+sub mw_push {\n+\tmy $last_local_revid = get_last_local_revision();\n+\tprint STDERR \".\\n\"; # Finish sentence started by get_last_local_revision()\n+\tmy $last_remote_revid = get_last_remote_revision();\n+\n+\t# Get sha1 of commit pointed by local HEAD\n+\tmy $HEAD_sha1 = run_git(\"rev-parse $_[0] 2>/dev/null\"); chomp($HEAD_sha1);\n+\t# Get sha1 of commit pointed by remotes/$remotename/master\n+\tmy $remoteorigin_sha1 = run_git(\"rev-parse refs/remotes/$remotename/master 2>/dev/null\");\n+\tchomp($remoteorigin_sha1);\n+\n+\tif ($last_local_revid > 0 &&\n+\t $last_local_revid < $last_remote_revid){\n+\t\treturn error_non_fast_forward($_[0]);\n+\t}\n+\n+\tif ($HEAD_sha1 eq $remoteorigin_sha1) {\n+\t\tprint STDOUT \"\\n\";\n+\t\treturn;\n+\t}\n+\n+\t# Get every commit in between HEAD and refs/remotes/origin/master,\n+\t# including HEAD and refs/remotes/origin/master\n+\tmy @commit_pairs = ();\n+\tif ($last_local_revid > 0) {\n+\t\tmy $parsed_sha1 = $remoteorigin_sha1;\n+\t\t# Find a path from last MediaWiki commit to pushed commit\n+\t\twhile ($parsed_sha1 ne $HEAD_sha1) {\n+\t\t\tmy @commit_info = grep(/^$parsed_sha1/, split(/\\n/, run_git(\"rev-list --children $_[0]\")));\n+\t\t\tif (!@commit_info) {\n+\t\t\t\treturn error_non_fast_forward($_[0]);\n+\t\t\t}\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $commit_info[0]);\n+\t\t\t# $commit_info_split[1] is the sha1 of the commit to export\n+\t\t\t# $commit_info_split[0] is the sha1 of its direct child\n+\t\t\tpush (@commit_pairs, \\@commit_info_split);\n+\t\t\t$parsed_sha1 = $commit_info_split[1];\n+\t\t}\n+\t} else {\n+\t\t# No remote mediawiki revision. Export the whole\n+\t\t# history (linearized with --first-parent)\n+\t\tprint STDERR \"Warning: no common ancestor, pushing complete history\\n\";\n+\t\tmy $history = run_git(\"rev-list --first-parent --children $_[0]\");\n+\t\tmy @history = split('\\n', $history);\n+\t\t@history = @history[1..$#history];\n+\t\tforeach my $line (reverse @history) {\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $line);\n+\t\t\tpush (@commit_pairs, \\@commit_info_split);\n+\t\t}\n+\t}\n+\n+\tforeach my $commit_info_split (@commit_pairs) {\n+\t\tmy $sha1_child = @{$commit_info_split}[0];\n+\t\tmy $sha1_commit = @{$commit_info_split}[1];\n+\t\tmy $diff_infos = run_git(\"diff-tree -r --raw -z $sha1_child $sha1_commit\");\n+\t\t# TODO: we could detect rename, and encode them with a #redirect on the wiki.\n+\t\t# TODO: for now, it's just a delete+add\n+\t\tmy @diff_info_list = split(/\\0/, $diff_infos);\n+\t\t# Keep the first line of the commit message as mediawiki comment for the revision\n+\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $sha1_commit\")))[0];\n+\t\tchomp($commit_msg);\n+\t\t# Push every blob\n+\t\twhile (@diff_info_list) {\n+\t\t\t# git diff-tree -z gives an output like\n+\t\t\t# \\0\\0\n+\t\t\t# \\0\\0\n+\t\t\t# and we've split on \\0.\n+\t\t\tmy $info = shift(@diff_info_list);\n+\t\t\tmy $file = shift(@diff_info_list);\n+\t\t\tmw_push_file($info, $file, $commit_msg);\n+\t\t}\n+\t}\n+\n+\tprint STDOUT \"ok $_[1]\\n\";\n+\tprint STDOUT \"\\n\";\n+\n+\tprint STDERR \"Just pushed some revisions to MediaWiki.\\n\";\n+\tprint STDERR \"The pushed revisions now have to be re-imported, and your current branch\\n\";\n+\tprint STDERR \"needs to be updated with these re-imported commits. You can do this with\\n\";\n+\tprint STDERR \"\\n\";\n+\tprint STDERR \" git pull --rebase\\n\";\n+\tprint STDERR \"\\n\";\n+}\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt\nnew file mode 100644\nindex 0000000..4d211f5\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki.txt\n@@ -0,0 +1,7 @@\n+Git-Mediawiki is a project which aims the creation of a gate\n+between git and mediawiki, allowing git users to push and pull\n+objects from mediawiki just as one would do with a classic git\n+repository thanks to remote-helpers.\n+\n+For more information, visit the wiki at\n+https://github.com/Bibzball/Git-Mediawiki/wiki\n-- \n1.7.6.585.g5929f.dirty\n"},{"id":"174336","messageId":"vpqr548hx40.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"7v8vqgdpsv.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-08-26T17:59:59Z","receivedAt":"2011-08-26T17:59:59Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> Matthieu Moy writes:\n>\n>> The uninitialized variable seems harmless in practice, but let's still be clean.\n>\n> It is not \"in practice\", but by definition, file scope \"static int\"\n> variables are initialized to 0 by the C language\n\nOK, I didn't know this was specified in C, but you're right.\n\n> Please do not write unnecessary \" = 0\" there.\n\nI prefer being explicit, but that's a matter of taste, and I don't\nreally care, so let's drop this patch.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174338","messageId":"7vvctkc89o.fsf@alter.siamese.dyndns.org","threadId":"28226","inReplyTo":"vpqr548hx40.fsf@bauges.imag.fr","subject":"Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-08-26T18:55:31Z","receivedAt":"2011-08-26T18:55:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n>> Please do not write unnecessary \" = 0\" there.\n>\n> I prefer being explicit, but that's a matter of taste, and I don't\n> really care, so let's drop this patch.\n\nI prefer being explicit, too, but \"static int foo;\" is explicit enough if\nyou know C.\n"},{"id":"174442","messageId":"CAGdFq_iYRkfnTbYAgmX1g4uOxWb_ZYxr+TNinVfWRV-zXrnzAg@mail.gmail.com","threadId":"28226","inReplyTo":"7v4o14dppz.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-29T05:42:48Z","receivedAt":"2011-08-29T05:42:48Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\n2011/8/26 Junio C Hamano :\n> Matthieu Moy writes:\n>> In short, the changes since v3 are:\n>>\n>> * Adapt to newer Git, which seem to require a \"done\" command at the\n>>   end of the fast-import stream. I don't understand why this is\n>>   needed, since fast-import is called without the --done flag by\n>>   remote-helpers, but if I don't do this, \"git fetch\" doesn't\n>>   terminate and keeps waiting ...\n>\n> Hmmmm, is this a regression in fast-import? Can this be bisected if so?\n\nWe agreed that making this change is the best way to go forward, since\nthere's so few (read: none) remote-helpers yet, the fix is trivial,\nand it solves a design mistake while the api is not yet set in stone.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174444","messageId":"7v4o1092hn.fsf@alter.siamese.dyndns.org","threadId":"28226","inReplyTo":"CAGdFq_iYRkfnTbYAgmX1g4uOxWb_ZYxr+TNinVfWRV-zXrnzAg@mail.gmail.com","subject":"Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-08-29T06:05:24Z","receivedAt":"2011-08-29T06:05:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sverre Rabbelier writes:\n\n> 2011/8/26 Junio C Hamano :\n>> Matthieu Moy writes:\n>>> In short, the changes since v3 are:\n>>>\n>>> * Adapt to newer Git, which seem to require a \"done\" command at the\n>>>   end of the fast-import stream. I don't understand why this is\n>>>   needed, since fast-import is called without the --done flag by\n>>>   remote-helpers, but if I don't do this, \"git fetch\" doesn't\n>>>   terminate and keeps waiting ...\n>>\n>> Hmmmm, is this a regression in fast-import? Can this be bisected if so?\n>\n> We agreed that making this change is the best way to go forward, since\n> there's so few (read: none) remote-helpers yet, the fix is trivial,\n> and it solves a design mistake while the api is not yet set in stone.\n\nDoes this exchange suggest that at least we would need an update to\ndocumentation around \"done\", as Matthieu's \"why is 'done' needed even\nthough I am not calling with --done?\" sounds like a very fair question.\n"},{"id":"174448","messageId":"CAGdFq_jwLGZ+tLKramRrLJmwyY_uDtj7JXUwYBO9pSqOmZ20xQ@mail.gmail.com","threadId":"28226","inReplyTo":"7v4o1092hn.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-29T06:41:30Z","receivedAt":"2011-08-29T06:41:30Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Mon, Aug 29, 2011 at 08:05, Junio C Hamano wrote:\n> Does this exchange suggest that at least we would need an update to\n> documentation around \"done\", as Matthieu's \"why is 'done' needed even\n> though I am not calling with --done?\" sounds like a very fair question.\n\nNo I think the documentation for fast-import is correct. If you pass\n--use-done-feature or print 'feature done' in the stream the use of\n'done' is required, otherwise it isn't. We did recently changed git to\npass '--use-done-feature' to the fast-import process though :).\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174534","messageId":"20110830035608.GB6647@elie.gateway.2wire.net","threadId":"28226","inReplyTo":"CAGdFq_jwLGZ+tLKramRrLJmwyY_uDtj7JXUwYBO9pSqOmZ20xQ@mail.gmail.com","subject":"Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-08-30T03:56:08Z","receivedAt":"2011-08-30T03:56:08Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Sverre Rabbelier wrote:\n> On Mon, Aug 29, 2011 at 08:05, Junio C Hamano wrote:\n\n>> Does this exchange suggest that at least we would need an update to\n>> documentation around \"done\", as Matthieu's \"why is 'done' needed even\n>> though I am not calling with --done?\" sounds like a very fair question.\n>\n> No I think the documentation for fast-import is correct.\n\nHowever, the documentation for git-remote-helpers is hopelessly outdated.\n\nHere's an old attempt to make the documentation a little easier to read,\nand hopefully also to add to. The text from \"Capabilities for Pushing\"\non needs help to avoid seeming redundant in light of the list of\ncapabilities before, but I'm sending it now to get comments and\nhopefully help from other interested people.\n\nIt does _not_ document recent changes; patches for that would\npresumably come on top. Last time I sent this to list[1], it brought\nto life some inconsistencies in how the various \"list\" commands are\nused. Another response was to flesh out what a capability for\nfetching a single tree (\"git archive --remote\") would look like[2],\nfor which I was very grateful (and I'd happy to see it come to life).\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/169341\n[2] http://colabti.org/irclogger/irclogger_log/git-devel?date=2011-03-19#l434\n\n-- >8 --\nSubject: Documentation/remote-helpers: explain capabilities first\n\nThe current remote helper documentation is from the perspective of\ngit, so to speak: it presents a full menu of commands for a person\ninvoking a remote helper to choose from. In practice, that's less\nuseful than it could be, since the daunted novice remote-helper author\nprobably just wanted a list of commands needs to implement to get\nstarted. So preface the command list with an overview of each\ncapability, its purpose, and what commands it requires.\n\nAs a side effect, this makes it a little clearer that git doesn't\nchoose arbitrary commands to run, even if the remote helper advertises\nall capabilities --- instead, there are well defined command sequences\nfor various tasks.\n\nSigned-off-by: Jonathan Nieder \n---\n Documentation/git-remote-helpers.txt | 159 +++++++++++++++++++++++++++-------\n 1 files changed, 129 insertions(+), 30 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 58f6ad49..f3fd63a4 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -23,22 +23,141 @@ output. Because a remote helper runs as an independent process from\n git, there is no need to re-link git to add a new helper, nor any\n need to link the helper with the implementation of git.\n \n-Every helper must support the \"capabilities\" command, which git will\n-use to determine what other commands the helper will accept. Other\n-commands generally concern facilities like discovering and updating\n-remote refs, transporting objects between the object database and\n-the remote repository, and updating the local object store.\n-\n-Helpers supporting the 'fetch' capability can discover refs from the\n-remote repository and transfer objects reachable from those refs to\n-the local object store. Helpers supporting the 'push' capability can\n-transfer local objects to the remote repository and update remote refs.\n+Every helper must support the \"capabilities\" command, which git\n+uses to determine what other commands the helper will accept. Those\n+other commands can be used to discover and update remote refs,\n+transport objects between the object database and the remote repository,\n+and update the local object store.\n \n Git comes with a \"curl\" family of remote helpers, that handle various\n transport protocols, such as 'git-remote-http', 'git-remote-https',\n 'git-remote-ftp' and 'git-remote-ftps'. They implement the capabilities\n 'fetch', 'option', and 'push'.\n \n+INPUT FORMAT\n+------------\n+\n+Git sends the remote helper a list of commands on standard input, one\n+per line. The first command is always the 'capabilities' command, in\n+response to which the remote helper must print a list of the\n+capabilities it supports (see below) followed by a blank line. The\n+response to the capabilities command determines what commands Git uses\n+in the remainder of the command stream.\n+\n+The command stream is terminated by a blank line. In some cases\n+(indicated in the documentation of the relevant commands), this blank\n+line is followed by a payload in some other protocol (e.g., the pack\n+protocol), while in others it indicates the end of input.\n+\n+Capabilities\n+~~~~~~~~~~~~\n+\n+Each remote helper is expected to support only a subset of commands.\n+The operations a helper supports are declared to git in the response\n+to the `capabilities` command (see COMMANDS, below).\n+\n+'option'::\n+\tFor specifying settings like `verbosity` (how much output to\n+\twrite to stderr) and `depth` (how much history is wanted in the\n+\tcase of a shallow clone) that affect how other commands are\n+\tcarried out.\n+\n+'connect'::\n+\tFor fetching and pushing using git's native packfile protocol\n+\tthat requires a bidirectional, full-duplex connection.\n+\n+'push'::\n+\tFor listing remote refs and pushing specified objects from the\n+\tlocal object store to remote refs.\n+\n+'fetch'::\n+\tFor listing remote refs and fetching the associated history to\n+\tthe local object store.\n+\n+'import'::\n+\tFor listing remote refs and fetching the associated history as\n+\ta fast-import stream.\n+\n+'refspec' ::\n+\tThis modifies the 'import' capability, allowing the produced\n+\tfast-import stream to modify refs in a private namespace\n+\tinstead of writing to refs/heads or refs/remotes directly.\n+\tIt is recommended that all importers providing the 'import'\n+\tcapability use this.\n++\n+A helper advertising the capability\n+`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`\n+is saying that, when it is asked to `import refs/heads/topic`, the\n+stream it outputs will update the `refs/svn/origin/branches/topic`\n+ref.\n++\n+This capability can be advertised multiple times. The first\n+applicable refspec takes precedence. The left-hand of refspecs\n+advertised with this capability must cover all refs reported by\n+the list command. If no 'refspec' capability is advertised,\n+there is an implied `refspec {asterisk}:{asterisk}`.\n+\n+Capabilities for Pushing\n+~~~~~~~~~~~~~~~~~~~~~~~~\n+'connect'::\n+\tCan attempt to connect to 'git receive-pack' (for pushing),\n+\t'git upload-pack', etc for communication using the\n+\tpackfile protocol.\n++\n+Supported commands: 'connect'.\n+\n+'push'::\n+\tCan discover remote refs and push local commits and the\n+\thistory leading up to them to new or existing remote refs.\n++\n+Supported commands: 'list for-push', 'push'.\n+\n+If a helper advertises both 'connect' and 'push', git will use\n+'connect' if possible and fall back to 'push' if the helper requests\n+so when connecting (see the 'connect' command under COMMANDS).\n+\n+Capabilities for Fetching\n+~~~~~~~~~~~~~~~~~~~~~~~~~\n+'connect'::\n+\tCan try to connect to 'git upload-pack' (for fetching),\n+\t'git receive-pack', etc for communication using the\n+\tpackfile protocol.\n++\n+Supported commands: 'connect'.\n+\n+'fetch'::\n+\tCan discover remote refs and transfer objects reachable from\n+\tthem to the local object store.\n++\n+Supported commands: 'list', 'fetch'.\n+\n+'import'::\n+\tCan discover remote refs and output objects reachable from\n+\tthem as a stream in fast-import format.\n++\n+Supported commands: 'list', 'import'.\n+\n+If a helper advertises 'connect', git will use it if possible and\n+fall back to another capability if the helper requests so when\n+connecting (see the 'connect' command under COMMANDS).\n+When choosing between 'fetch' and 'import', git prefers 'fetch'.\n+Other frontends may have some other order of preference.\n+\n+'refspec' ::\n+\tThis modifies the 'import' capability.\n++\n+A helper advertising\n+`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`\n+in its capabilities is saying that, when it handles\n+`import refs/heads/topic`, the stream it outputs will update the\n+`refs/svn/origin/branches/topic` ref.\n++\n+This capability can be advertised multiple times. The first\n+applicable refspec takes precedence. The left-hand of refspecs\n+advertised with this capability must cover all refs reported by\n+the list command. If no 'refspec' capability is advertised,\n+there is an implied `refspec {asterisk}:{asterisk}`.\n+\n INVOCATION\n ----------\n \n@@ -167,26 +286,6 @@ completing a valid response for the current command.\n Additional commands may be supported, as may be determined from\n capabilities reported by the helper.\n \n-CAPABILITIES\n-------------\n-\n-'fetch'::\n-'option'::\n-'push'::\n-'import'::\n-'connect'::\n-\tThis helper supports the corresponding command with the same name.\n-\n-'refspec' 'spec'::\n-\tWhen using the import command, expect the source ref to have\n-\tbeen written to the destination ref. The earliest applicable\n-\trefspec takes precedence. For example\n-\t\"refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}\" means\n-\tthat, after an \"import refs/heads/name\", the script has written to\n-\trefs/svn/origin/branches/name. If this capability is used at\n-\tall, it must cover all refs reported by the list command; if\n-\tit is not used, it is effectively \"{asterisk}:{asterisk}\"\n-\n REF LIST ATTRIBUTES\n -------------------\n \n-- \n1.7.6\n"},{"id":"174561","messageId":"7vfwki3jqp.fsf@alter.siamese.dyndns.org","threadId":"28226","inReplyTo":"20110830035608.GB6647@elie.gateway.2wire.net","subject":"Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-08-30T17:13:50Z","receivedAt":"2011-08-30T17:13:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Sverre Rabbelier wrote:\n>> On Mon, Aug 29, 2011 at 08:05, Junio C Hamano wrote:\n>\n>>> Does this exchange suggest that at least we would need an update to\n>>> documentation around \"done\", as Matthieu's \"why is 'done' needed even\n>>> though I am not calling with --done?\" sounds like a very fair question.\n>>\n>> No I think the documentation for fast-import is correct.\n>\n> However, the documentation for git-remote-helpers is hopelessly outdated.\n\nGood point. If somebody writes a working remote-helper using \"import\" to\ntrigger fast-import as the back-end, and if it breaks because we changed\nthe way fast-import is invoked and the helper needs to be updated, we at\nleast need to give them a useful documentation that clearly states how the\nhelper is expected to behave.\n\n> It does _not_ document recent changes; patches for that would\n> presumably come on top.\n\nWill queue this version for now, if only as a reminder that we would need\nmore updates.\n\nThanks.\n"},{"id":"174600","messageId":"vpqobz53ig3.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"20110830035608.GB6647@elie.gateway.2wire.net","subject":"Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-08-31T11:54:04Z","receivedAt":"2011-08-31T11:54:04Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jonathan Nieder writes:\n\n> Here's an old attempt to make the documentation a little easier to read,\n> and hopefully also to add to.\n\nThanks, that helps. I wish we had all this earlier ;-).\n\n> +Git sends the remote helper a list of commands on standard input, one\n> +per line. The first command is always the 'capabilities' command, \n\nDo we want to set this in stone? Wouldn't a Git implementation calling\n\"option\" before \"capabilities\" be correct?\n\n> +Capabilities\n> +~~~~~~~~~~~~\n\n(perhaps name the section \"Overview of Capabilities\"?)\n\n[...]\n> +'refspec' ::\n> +\tThis modifies the 'import' capability, allowing the produced\n> +\tfast-import stream to modify refs in a private namespace\n> +\tinstead of writing to refs/heads or refs/remotes directly.\n> +\tIt is recommended that all importers providing the 'import'\n> +\tcapability use this.\n> ++\n> +A helper advertising the capability\n> +`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`\n> +is saying that, when it is asked to `import refs/heads/topic`, the\n> +stream it outputs will update the `refs/svn/origin/branches/topic`\n> +ref.\n> ++\n> +This capability can be advertised multiple times. The first\n> +applicable refspec takes precedence. The left-hand of refspecs\n> +advertised with this capability must cover all refs reported by\n> +the list command. If no 'refspec' capability is advertised,\n> +there is an implied `refspec {asterisk}:{asterisk}`.\n\nSince this \"Capabilities\" section is meant to be an overview, I'd\nshorten this to\n\n+'refspec' ::\n+\tThis modifies the 'import' capability, allowing the produced\n+\tfast-import stream to modify refs in a private namespace\n+\tinstead of writing to refs/heads or refs/remotes directly.\n\nand drop the detailed explanation here.\n\n> +Capabilities for Fetching\n> +~~~~~~~~~~~~~~~~~~~~~~~~~\n[...]\n> +'refspec' ::\n> +\tThis modifies the 'import' capability.\n\nSince this would be the \"detailed explanation\" part, this is the one\nreaders will read more carefully, so I'd put the recommandation right\nhere:\n\n+\tIt is recommended that all importers providing the 'import'\n+\tcapability use this.\n\nand of course, keep this:\n\n> ++\n> +A helper advertising\n> +`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`\n> +in its capabilities is saying that, when it handles\n> +`import refs/heads/topic`, the stream it outputs will update the\n> +`refs/svn/origin/branches/topic` ref.\n> ++\n> +This capability can be advertised multiple times. The first\n> +applicable refspec takes precedence. The left-hand of refspecs\n> +advertised with this capability must cover all refs reported by\n> +the list command. If no 'refspec' capability is advertised,\n> +there is an implied `refspec {asterisk}:{asterisk}`.\n> +\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174601","messageId":"vpq1uw13hx3.fsf_-_@bauges.imag.fr","threadId":"28226","inReplyTo":"CAGdFq_jwLGZ+tLKramRrLJmwyY_uDtj7JXUwYBO9pSqOmZ20xQ@mail.gmail.com","subject":"done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-08-31T12:05:28Z","receivedAt":"2011-08-31T12:05:28Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Sverre Rabbelier writes:\n\n> Heya,\n>\n> On Mon, Aug 29, 2011 at 08:05, Junio C Hamano wrote:\n>> Does this exchange suggest that at least we would need an update to\n>> documentation around \"done\", as Matthieu's \"why is 'done' needed even\n>> though I am not calling with --done?\" sounds like a very fair question.\n>\n> No I think the documentation for fast-import is correct. If you pass\n> --use-done-feature or print 'feature done' in the stream the use of\n> 'done' is required, otherwise it isn't. We did recently changed git to\n> pass '--use-done-feature' to the fast-import process though :).\n\nThat doesn't help much someone writting a remote helper.\n\nThe documentation for remote-helpers neither talks about \"done\" nor\nabout \"--use-done-feature\" or whatever way Git uses this feature when\nusing remote-helpers.\n\nThe current state is particularly confusing: git seems to expect one and\nonly one \"done\" feature, even when multiple \"import\" commands are\nissued. That's very strange, and I'm not sure whether it's the expected\nbehavior (I can try a documentation patch, but I need to understand\nbetter what's expected and what's not).\n\nIt would be natural to write remote-helpers like\n\nwhile ($cmd = ) {\n if ($cmd eq \"import\") {\n \n print \"done\\n\";\n } ...\n}\n\nbut in the current state, it doesn't work since we'll get a first\n\"import HEAD\", issue a \"done\", then get a \"import refs/heads/master\" and\nwrite to a dead pipe.\n\nThen, it would be very tempting to write it like\n\nwhile ($cmd = ) {\n if ($cmd eq \"import\") {\n \n } ...\n}\nprint \"done\\n\";\n\nbut this doesn't work either, because when calling \"git push\", no\n\"import\" command is involved, no fast-import is started, and the \"done\"\nbreaks everything.\n\nThat's why I had to make it like\n\nmy $import_started;\nwhile ($cmd = ) {\n if ($cmd eq \"import\") {\n $import_started = 1;\n \n } ...\n}\nif ($import_started) {\n print \"done\\n\";\n}\n\nand I really had the feeling I was working around a mis-feature of Git\nhere.\n\nSo, is this the expected behavior? Wouldn't it be more sensible to allow\nthe remote-helper to issue a \"done\" after each \"import\" command? Right\nnow, my understanding is that after an \"import\" command is issued, it's\nno longer possible to output anything other than fast-import stream on\nstdout, and I guess it'd be more future-proof to allow closing the\nfast-import with a \"done\", and allow any dialog between git and the\nremote helper afterwards.\n\nThanks,\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174602","messageId":"CAGdFq_jyVK3_THYXzCOLDpNww0Npn2qzZ1qv-BMuLbg1vgVjZw@mail.gmail.com","threadId":"28226","inReplyTo":"vpq1uw13hx3.fsf_-_@bauges.imag.fr","subject":"Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-31T12:17:59Z","receivedAt":"2011-08-31T12:17:59Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Aug 31, 2011 at 14:05, Matthieu Moy\n wrote:\n> So, is this the expected behavior? Wouldn't it be more sensible to allow\n> the remote-helper to issue a \"done\" after each \"import\" command? Right\n> now, my understanding is that after an \"import\" command is issued, it's\n> no longer possible to output anything other than fast-import stream on\n> stdout, and I guess it'd be more future-proof to allow closing the\n> fast-import with a \"done\", and allow any dialog between git and the\n> remote helper afterwards.\n\nWow, no that's not the intended behavior. We meant to make it exactly\nas you describe, after each import command you end with a done. This\nis (should) also be what the testgit implementation does currently,\nand it's what my remote-hg helper does as well. I'm not sure why it's\nnot working for you, but if it's not then that is definitely a bug,\nand not intended behavior.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174603","messageId":"vpqd3flzrow.fsf_-_@bauges.imag.fr","threadId":"28226","inReplyTo":"1314378689-8997-2-git-send-email-Matthieu.Moy@imag.fr","subject":"Clean termination of remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-08-31T12:33:19Z","receivedAt":"2011-08-31T12:33:19Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Matthieu Moy writes:\n\n> +BEGIN { $| = 1 };\n> +if (!eof(STDIN)) {\n> +\t# Wait for Git to terminate. If we don't, git fetch\n> +\t# (transport-helper.c's sendline function) will try to write\n> +\t# to our stdin, which will be closed, and git fetch will be\n> +\t# killed. That's probably a bug in transport-helper.c, but in\n> +\t# the meantime ...\n> +\tsleep .1;\n> +};\n\nI was expecting this part to be more controversial, so I'm just\nrepeating it to draw more attention ;-).\n\nI just found a \"cleaner\" way to terminate, but I still don't find it\nreally satisfactory:\n\n# Inform Git that we're done, otherwise Git won't close it's stdin,\n# and the next loop will be infinite.\nclose(STDOUT);\n# Flush stdin before we terminate. If we don't, git fetch\n# (transport-helper.c's sendline function) will try to write to our\n# stdin, which may be closed, and git fetch will be killed. That's\n# probably a bug in transport-helper.c, but in the meantime ...\nwhile () {};\n\nThis seems reliable (just did 100+ imports without crash).\n\nIf I comment-out the \"close(STDOUT)\", then the while loop is indeed\ninfinite. If I comment-out the \"while () {};\", then the import\nunreliably fails (for example, I just did 10 clones of a 1-page wiki,\nand got one failure). No error message, just a non-zero exit status, and\nin the case of \"clone\", the newly created repository is deleted before\nthe command terminates.\n\nWith debug activated in transport-helper.c, the last messages are just:\n\nDebug: Disconnecting.\nDebug: Remote helper: -> \n\nand gdb says:\n\nProgram received signal SIGPIPE, Broken pipe.\n0xb7fe2424 in __kernel_vsyscall ()\n(gdb) bt\n#0 0xb7fe2424 in __kernel_vsyscall ()\n#1 0xb7e66ff3 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:82\n#2 0x0811dc08 in xwrite (fd=8, buf=0x81bd840, len=1) at wrapper.c:137\n#3 0x0811dc67 in write_in_full (fd=8, buf=0x81bd840, count=1) at wrapper.c:169\n#4 0x08115035 in sendline (helper=0x81a2d00, buffer=0xbfffe674) at transport-helper.c:41\n#5 0x081158f3 in disconnect_helper (transport=0x81a2cc0) at transport-helper.c:231\n#6 release_helper (transport=0x81a2cc0) at transport-helper.c:324\n#7 0x08111f1d in transport_disconnect (transport=0x81a2cc0) at transport.c:1144\n#8 0x08061711 in cmd_clone (argc=3, argv=0xbfffeb58, prefix=0x0) at builtin/clone.c:739\n#9 0x0804ba27 in run_builtin (argc=, argv=) at git.c:308\n#10 handle_internal_command (argc=, argv=) at git.c:466\n#11 0x0804bc33 in run_argv (argc=3, argv=0xbfffeb58) at git.c:512\n#12 main (argc=3, argv=0xbfffeb58) at git.c:585\n\nAny idea how to fix this?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174604","messageId":"vpqk49tviza.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"CAGdFq_jyVK3_THYXzCOLDpNww0Npn2qzZ1qv-BMuLbg1vgVjZw@mail.gmail.com","subject":"Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-08-31T12:55:05Z","receivedAt":"2011-08-31T12:55:05Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Sverre Rabbelier writes:\n\n> Heya,\n>\n> On Wed, Aug 31, 2011 at 14:05, Matthieu Moy\n> wrote:\n>> So, is this the expected behavior? Wouldn't it be more sensible to allow\n>> the remote-helper to issue a \"done\" after each \"import\" command? Right\n>> now, my understanding is that after an \"import\" command is issued, it's\n>> no longer possible to output anything other than fast-import stream on\n>> stdout, and I guess it'd be more future-proof to allow closing the\n>> fast-import with a \"done\", and allow any dialog between git and the\n>> remote helper afterwards.\n>\n> Wow, no that's not the intended behavior. We meant to make it exactly\n> as you describe, after each import command you end with a done. This\n> is (should) also be what the testgit implementation does currently,\n\nErr, no, it isn't. From git-remote-testgit.py:\n\ndef do_import(repo, args):\n[...]\n while True:\n line = sys.stdin.readline()\n[...]\n # strip of leading 'import '\n ref = line[7:].strip()\n refs.append(ref)\n\n repo = update_local_repo(repo)\n repo.exporter.export_repo(repo.gitdir, refs)\n\n print \"done\"\n\nWhat it does is that it reads multiple \"import\" commands, and process\nthem all at once, with a single \"print done\" at the end. Actually,\ntestgit would die(\"Expected import line.\") if Git sent another command\nafter \"import\".\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174605","messageId":"CAGdFq_hFPBeogpX0Qp8Knfssw8QzV+GwCAyH+zZhM2iEJfFYMw@mail.gmail.com","threadId":"28226","inReplyTo":"vpqk49tviza.fsf@bauges.imag.fr","subject":"Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-31T12:58:38Z","receivedAt":"2011-08-31T12:58:38Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Aug 31, 2011 at 14:55, Matthieu Moy\n wrote:\n> Err, no, it isn't. From git-remote-testgit.py:\n>\n> def do_import(repo, args):\n> [...]\n>    while True:\n>        line = sys.stdin.readline()\n> [...]\n>        # strip of leading 'import '\n>        ref = line[7:].strip()\n>        refs.append(ref)\n>\n>    repo = update_local_repo(repo)\n>    repo.exporter.export_repo(repo.gitdir, refs)\n>\n>    print \"done\"\n>\n> What it does is that it reads multiple \"import\" commands, and process\n> them all at once, with a single \"print done\" at the end. Actually,\n> testgit would die(\"Expected import line.\") if Git sent another command\n> after \"import\".\n\nAh, see, this is why in my original version the syntax was:\n\nimport\nrefs/heads/master\nrefs/heads/next\n\\n\n\nInstead of the current:\n\nimport refs/heads/master\nimport refs/heads/next\n\\n\n\nYou can have multiple imports in the current system, you just need to\nremember to add the trailing newline.\n\nimport refs/heads/master\n\\n\nimport refs/heads/next\n\\n\n\nIn the above case you'll have to have two done commands.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174606","messageId":"vpq4o0xpvwy.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"CAGdFq_hFPBeogpX0Qp8Knfssw8QzV+GwCAyH+zZhM2iEJfFYMw@mail.gmail.com","subject":"Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-08-31T13:12:13Z","receivedAt":"2011-08-31T13:12:13Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Sverre Rabbelier writes:\n\n> You can have multiple imports in the current system, you just need to\n> remember to add the trailing newline.\n>\n> import refs/heads/master\n> \\n\n> import refs/heads/next\n> \\n\n\nThen I'm lost. Isn't \\n supposed to mean that the list of commands is\nover, and that the remote-helper should terminate?\n\nAlso, who is \"you\" in your sentence? It can't be the remote helper\n(which reads this sequence), so it has to be Git's transport-helper. Are\nyou saying that the transport-helper should be modified to add \\n after\nsending an import command?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174607","messageId":"CAGdFq_gooPGQMa5D19-wag=X+mDntHF1MwXiYF0zovB=L-Sybw@mail.gmail.com","threadId":"28226","inReplyTo":"vpq4o0xpvwy.fsf@bauges.imag.fr","subject":"Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-31T13:16:06Z","receivedAt":"2011-08-31T13:16:06Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Aug 31, 2011 at 15:12, Matthieu Moy\n wrote:\n> Sverre Rabbelier writes:\n>\n>> You can have multiple imports in the current system, you just need to\n>> remember to add the trailing newline.\n>>\n>> import refs/heads/master\n>> \\n\n>> import refs/heads/next\n>> \\n\n>\n> Then I'm lost. Isn't \\n supposed to mean that the list of commands is\n> over, and that the remote-helper should terminate?\n\nNo, a list of 'import' statements has to be followed by a \\n, similar\nto the push command. You can have multiple 'blocks' of import/push\ncommands, but each block has to be followed by a newline. Again, you\nshould read:\n\nimport A\nimport B\n\\n\nimport C\nimport D\n\\n\n\nas:\nimport\nA\nB\n\\n\nimport\nC\nD\n\\n\n\n> Also, who is \"you\" in your sentence? It can't be the remote helper\n> (which reads this sequence), so it has to be Git's transport-helper.\n\nCorrect.\n\n> Are you saying that the transport-helper should be modified to add \\n\n> after sending an import command?\n\nAs said above, it already does that. Each group of import statements\nis terminated with a \\n.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174608","messageId":"CAGdFq_gSpFm8D1qHs5smUgsqyZXRjw73QFCCkBjTi0n4pwzmHA@mail.gmail.com","threadId":"28226","inReplyTo":"vpqd3flzrow.fsf_-_@bauges.imag.fr","subject":"Re: Clean termination of remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-31T13:25:53Z","receivedAt":"2011-08-31T13:25:53Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Aug 31, 2011 at 14:33, Matthieu Moy\n wrote:\n> I was expecting this part to be more controversial, so I'm just\n> repeating it to draw more attention ;-).\n\nEek! :)\n\n> # Inform Git that we're done, otherwise Git won't close it's stdin,\n> # and the next loop will be infinite.\n> close(STDOUT);\n> # Flush stdin before we terminate. If we don't, git fetch\n> # (transport-helper.c's sendline function) will try to write to our\n> # stdin, which may be closed, and git fetch will be killed. That's\n> # probably a bug in transport-helper.c, but in the meantime ...\n> while () {};\n\nIs this caused by you not reading the terminating '\\n' that git sends\nwhen all commands are done? We previously didn't do this for imports\n(we couldn't, hence the new import/done behavior), so perhaps you just\nneed to fix that? It's entirely possible you've found a bug though.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174612","messageId":"vpqty8x3a5b.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"CAGdFq_gSpFm8D1qHs5smUgsqyZXRjw73QFCCkBjTi0n4pwzmHA@mail.gmail.com","subject":"Re: Clean termination of remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-08-31T14:53:20Z","receivedAt":"2011-08-31T14:53:20Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Sverre Rabbelier writes:\n\n> Heya,\n>\n> On Wed, Aug 31, 2011 at 14:33, Matthieu Moy\n> wrote:\n>> I was expecting this part to be more controversial, so I'm just\n>> repeating it to draw more attention ;-).\n>\n> Eek! :)\n>\n>> # Inform Git that we're done, otherwise Git won't close it's stdin,\n>> # and the next loop will be infinite.\n>> close(STDOUT);\n>> # Flush stdin before we terminate. If we don't, git fetch\n>> # (transport-helper.c's sendline function) will try to write to our\n>> # stdin, which may be closed, and git fetch will be killed. That's\n>> # probably a bug in transport-helper.c, but in the meantime ...\n>> while () {};\n>\n> Is this caused by you not reading the terminating '\\n' that git sends\n> when all commands are done?\n\nIndeed. The stream sent by git looks like\n\nimport HEAD\nimport refs/heads/master\n\\n <-- this one closes the sequence of import\n\\n <-- this one closes the sequence of commands.\n\nand I was interpreting it as\n\nimport HEAD\nimport refs/heads/master\n\\n <-- this one closes the sequence of commands\n\\n <-- what's this??\n\nSo it seems the only bug I've found is insufficient documentation. A\npatch follows.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174613","messageId":"CAGdFq_giHLBgGOLDhj3zuxV0pnJ8twDfY7DD9ckfp4s4O7E8_g@mail.gmail.com","threadId":"28226","inReplyTo":"vpqty8x3a5b.fsf@bauges.imag.fr","subject":"Re: Clean termination of remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-31T15:00:44Z","receivedAt":"2011-08-31T15:00:44Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Aug 31, 2011 at 16:53, Matthieu Moy\n wrote:\n> So it seems the only bug I've found is insufficient documentation. A\n> patch follows.\n\nYay! Thanks :)\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174624","messageId":"1314809222-30528-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"CAGdFq_gooPGQMa5D19-wag=X+mDntHF1MwXiYF0zovB=L-Sybw@mail.gmail.com","subject":"[PATCH] git-remote-helpers.txt: explain how import works with multiple refs","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-08-31T16:47:02Z","receivedAt":"2011-08-31T16:47:02Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"This is important for two reasons:\n\n* when two \"import\" lines follow each other, only one \"done\" command\n should be issued in the fast-import stream, not one per \"import\".\n\n* The blank line terminating an import command should not be confused\n with the one terminating the sequence of commands.\n\nWhile we're there, illustrate the corresponding explanation for push\nbatches with an example.\n\nSigned-off-by: Matthieu Moy \n---\nSverre Rabbelier writes:\n\n> No, a list of 'import' statements has to be followed by a \\n, similar\n> to the push command. You can have multiple 'blocks' of import/push\n> commands, but each block has to be followed by a newline. Again, you\n> should read:\n>\n> import A\n> import B\n> \\n\n> import C\n> import D\n> \\n\n\nThanks for the explanation. This was documented for 'push', but your\nillustration made it easier for me to understand, so I've added the\nexample for push.\n\nThe documentation of \"batch import\" also fixes the \"it's not\ndocumented that remote-helpers should use 'done'\" as a side-effect.\n\nWhile investigating all this, I found in git-remote-testgit.py that\nthe remote-helpers had an \"export\" functionality, which seems totally\nundocumented. Any volunteer to document it?\n\n Documentation/git-remote-helpers.txt | 22 +++++++++++++++++++++-\n 1 files changed, 21 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 760b164..526fc6a 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -241,7 +241,22 @@ Supported if the helper has the \"fetch\" capability.\n 'push' +:::\n \tPushes the given local commit or branch to the\n \tremote branch described by . A batch sequence of\n-\tone or more push commands is terminated with a blank line.\n+\tone or more 'push' commands is terminated with a blank line\n+\t(if there is only one reference to push, a single 'push' command\n+\tis followed by a blank line). For example, the following would\n+\tbe two batches of 'push', the first asking the remote-helper\n+\tto push the local ref 'master' to the remote ref 'master' and\n+\tthe local 'HEAD' to the remote 'branch', and the second\n+\tasking to push ref 'foo' to ref 'bar' (forced update requested\n+\tby the '+').\n++\n+------------\n+push refs/heads/master:refs/heads/master\n+push HEAD:refs/heads/branch\n+\\n\n+push +refs/heads/foo:refs/heads/bar\n+\\n\n+------------\n +\n Zero or more protocol options may be entered after the last 'push'\n command, before the batch's terminating blank line.\n@@ -266,6 +281,11 @@ Supported if the helper has the \"push\" capability.\n Especially useful for interoperability with a foreign versioning\n system.\n +\n+Just like 'push', a batch sequence of one or more 'import' is\n+terminated with a blank line. For each batch of 'import', the remote\n+helper should produce a fast-import stream terminated by a 'done'\n+command.\n++\n Supported if the helper has the \"import\" capability.\n \n 'connect' ::\n-- \n1.7.7.rc0.78.ge4b5a.dirty\n"},{"id":"174629","messageId":"1314809708-8177-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"1314381329-8989-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-08-31T16:55:08Z","receivedAt":"2011-08-31T16:55:08Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: Jeremie Nikaes \n\nImplement a gate between git and mediawiki, allowing git users to push\nand pull objects from mediawiki just as one would do with a classic git\nrepository thanks to remote-helpers.\n\nThe following packages need to be installed (available on common\nrepositories):\n\n libmediawiki-api-perl\n libdatetime-format-iso8601-perl\n\nUse remote helpers in order to be as transparent as possible to the git\nuser.\n\nDownload Mediawiki revisions through the Mediawiki API and then\nfast-import into git.\n\nMediawiki revision number and git commits are linked thanks to notes\nbound to commits.\n\nThe import part is done on a refs/mediawiki/ branch before\ncoming to refs/remote/origin/master (Huge thanks to Jonathan Nieder\nfor his help)\n\nWe use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but\nwe also read the output of Git commands in UTF-8 with the small helper\nrun_git, and write to the console (STDERR) in UTF-8. This allows a\nseamless use of non-ascii characters in page titles, but hasn't been\ntested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames\ncould raise problems if different file systems handle UTF-8 filenames\ndifferently. A uri_escape of mediawiki filenames could be imaginable, and\nis still to be discussed further.\n\nPartial cloning is supported using one of:\n\ngit clone -c remote.origin.pages='A_Page Another_Page' mediawiki::http://wikiurl\n\ngit clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl\n\ngit clone -c remote.origin.shallow='True' mediawiki::http://wikiurl\n\nThanks to notes metadata, it is possible to compare remote and local last\nmediawiki revision to warn non-fast forward pushes and \"everything\nup-to-date\" case.\n\nWhen allowed, push looks for each commit between remotes/origin/master\nand HEAD, catches every blob related to these commit and push them in\nchronological order. To do so, it uses git rev-list --children HEAD and\ntravels the tree from remotes/origin/master to HEAD through children. In\nother words :\n\n\t* Shortest path from remotes/origin/master to HEAD\n\t* For each commit encountered, push blobs related to this commit\n\nSigned-off-by: Jérémie Nikaes \nSigned-off-by: Arnaud Lacurie \nSigned-off-by: Claire Fousse \nSigned-off-by: David Amouyal \nSigned-off-by: Matthieu Moy \nSigned-off-by: Sylvain Boulmé \nSigned-off-by: Matthieu Moy \n---\nSo, after understanding better how import works, here's an updated\npatch that gets rid of the hacky workaround to terminate and send the\n\"done\" command at the right time.\n\nActually, push had the same problem but it just went unnoticed (the\nremote has just one branch, so it's silly to try to push multiple\nbranches at the same time ...). This version handles push more\ncleanly, giving accurate error message in cases like\n\n git push origin :master\n git push origin foo bar master\n\nor perhaps more commonly\n\n git push --all\n\nin a repository with branches other than master.\n\nThere are still a few TODO, but I think all of them can wait.\n\n contrib/mw-to-git/git-remote-mediawiki | 758 ++++++++++++++++++++++++++++\n contrib/mw-to-git/git-remote-mediawiki.txt | 7 +\n 2 files changed, 765 insertions(+), 0 deletions(-)\n create mode 100755 contrib/mw-to-git/git-remote-mediawiki\n create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nnew file mode 100755\nindex 0000000..51f01cd\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -0,0 +1,758 @@\n+#! /usr/bin/perl\n+\n+# Copyright (C) 2011\n+# Jérémie Nikaes \n+# Arnaud Lacurie \n+# Claire Fousse \n+# David Amouyal \n+# Matthieu Moy \n+# License: GPL v2 or later\n+\n+# Gateway between Git and MediaWiki.\n+# https://github.com/Bibzball/Git-Mediawiki/wiki\n+#\n+# Known limitations:\n+#\n+# - Only wiki pages are managed, no support for [[File:...]]\n+# attachments.\n+#\n+# - Poor performance in the best case: it takes forever to check\n+# whether we're up-to-date (on fetch or push) or to fetch a few\n+# revisions from a large wiki, because we use exclusively a\n+# page-based synchronization. We could switch to a wiki-wide\n+# synchronization when the synchronization involves few revisions\n+# but the wiki is large.\n+#\n+# - Git renames could be turned into MediaWiki renames (see TODO\n+# below)\n+#\n+# - login/password support requires the user to write the password\n+# cleartext in a file (see TODO below).\n+#\n+# - No way to import \"one page, and all pages included in it\"\n+#\n+# - Multiple remote MediaWikis have not been very well tested.\n+\n+use strict;\n+use MediaWiki::API;\n+use DateTime::Format::ISO8601;\n+use encoding 'utf8';\n+\n+# use encoding 'utf8' doesn't change STDERROR\n+# but we're going to output UTF-8 filenames to STDERR\n+binmode STDERR, \":utf8\";\n+\n+use URI::Escape;\n+use warnings;\n+\n+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced\n+use constant SLASH_REPLACEMENT => \"%2F\";\n+\n+# It's not always possible to delete pages (may require some\n+# priviledges). Deleted pages are replaced with this content.\n+use constant DELETED_CONTENT => \"[[Category:Deleted]]\\n\";\n+\n+# It's not possible to create empty pages. New empty files in Git are\n+# sent with this content instead.\n+use constant EMPTY_CONTENT => \"\\n\";\n+\n+# used to reflect file creation or deletion in diff.\n+use constant NULL_SHA1 => \"0000000000000000000000000000000000000000\";\n+\n+my $remotename = $ARGV[0];\n+my $url = $ARGV[1];\n+\n+# Accept both space-separated and multiple keys in config file.\n+# Spaces should be written as _ anyway because we'll use chomp.\n+my @tracked_pages = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".pages\"));\n+chomp(@tracked_pages);\n+\n+# Just like @tracked_pages, but for MediaWiki categories.\n+my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".categories\"));\n+chomp(@tracked_categories);\n+\n+my $wiki_login = run_git(\"config --get remote.\". $remotename .\".mwLogin\");\n+# TODO: ideally, this should be able to read from keyboard, but we're\n+# inside a remote helper, so our stdin is connect to git, not to a\n+# terminal.\n+my $wiki_passwd = run_git(\"config --get remote.\". $remotename .\".mwPassword\");\n+chomp ($wiki_login);\n+chomp ($wiki_passwd);\n+\n+# Import only last revisions (both for clone and fetch)\n+my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shallow\");\n+chomp($shallow_import);\n+$shallow_import = ($shallow_import eq \"true\");\n+\n+my $wiki_name = $url;\n+$wiki_name =~ s/[^\\/]*:\\/\\///;\n+\n+# Commands parser\n+my $entry;\n+my @cmd;\n+while () {\n+\tchomp;\n+\t@cmd = split(/ /);\n+\tif (defined($cmd[0])) {\n+\t\t# Line not blank\n+\t\tif ($cmd[0] eq \"capabilities\") {\n+\t\t\tdie(\"Too many arguments for capabilities\") unless (!defined($cmd[1]));\n+\t\t\tmw_capabilities();\n+\t\t} elsif ($cmd[0] eq \"list\") {\n+\t\t\tdie(\"Too many arguments for list\") unless (!defined($cmd[2]));\n+\t\t\tmw_list($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"import\") {\n+\t\t\tdie(\"Invalid arguments for import\") unless ($cmd[1] ne \"\" && !defined($cmd[2]));\n+\t\t\tmw_import($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"option\") {\n+\t\t\tdie(\"Too many arguments for option\") unless ($cmd[1] ne \"\" && $cmd[2] ne \"\" && !defined($cmd[3]));\n+\t\t\tmw_option($cmd[1],$cmd[2]);\n+\t\t} elsif ($cmd[0] eq \"push\") {\n+\t\t\tmw_push($cmd[1]);\n+\t\t} else {\n+\t\t\tprint STDERR \"Unknown command. Aborting...\\n\";\n+\t\t\tlast;\n+\t\t}\n+\t} else {\n+\t\t# blank line: we should terminate\n+\t\tlast;\n+\t}\n+\n+\tBEGIN { $| = 1 } # flush STDOUT, to make sure the previous\n+\t\t\t # command is fully processed.\n+}\n+\n+########################## Functions ##############################\n+\n+# MediaWiki API instance, created lazily.\n+my $mediawiki;\n+\n+sub mw_connect_maybe {\n+\tif ($mediawiki) {\n+\t return;\n+\t}\n+\t$mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\tif ($wiki_login) {\n+\t\tif (!$mediawiki->login({\n+\t\t\tlgname => $wiki_login,\n+\t\t\tlgpassword => $wiki_passwd,\n+\t\t})) {\n+\t\t\tprint STDERR \"Failed to log in mediawiki user \\\"$wiki_login\\\" on $url\\n\";\n+\t\t\tprint STDERR \"(error \" .\n+\t\t\t $mediawiki->{error}->{code} . ': ' .\n+\t\t\t $mediawiki->{error}->{details} . \")\\n\";\n+\t\t\texit 1;\n+\t\t} else {\n+\t\t\tprint STDERR \"Logged in with user \\\"$wiki_login\\\".\\n\";\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_first_pages {\n+\tmy $some_pages = shift;\n+\tmy @some_pages = @{$some_pages};\n+\n+\tmy $pages = shift;\n+\n+\t# pattern 'page1|page2|...' required by the API\n+\tmy $titles = join('|', @some_pages);\n+\n+\tmy $mw_pages = $mediawiki->api({\n+\t\taction => 'query',\n+\t\ttitles => $titles,\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not query the list of wiki pages.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\twhile (my ($id, $page) = each (%{$mw_pages->{query}->{pages}})) {\n+\t\tif ($id < 0) {\n+\t\t\tprint STDERR \"Warning: page $page->{title} not found on wiki\\n\";\n+\t\t} else {\n+\t\t\t$pages->{$page->{title}} = $page;\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_pages {\n+\tmw_connect_maybe();\n+\n+\tmy %pages; # hash on page titles to avoid duplicates\n+\tmy $user_defined;\n+\tif (@tracked_pages) {\n+\t\t$user_defined = 1;\n+\t\t# The user provided a list of pages titles, but we\n+\t\t# still need to query the API to get the page IDs.\n+\n+\t\tmy @some_pages = @tracked_pages;\n+\t\twhile (@some_pages) {\n+\t\t\tmy $last = 50;\n+\t\t\tif ($#some_pages < $last) {\n+\t\t\t\t$last = $#some_pages;\n+\t\t\t}\n+\t\t\tmy @slice = @some_pages[0..$last];\n+\t\t\tget_mw_first_pages(\\@slice, \\%pages);\n+\t\t\t@some_pages = @some_pages[51..$#some_pages];\n+\t\t}\n+\t}\n+\tif (@tracked_categories) {\n+\t\t$user_defined = 1;\n+\t\tforeach my $category (@tracked_categories) {\n+\t\t\tif (index($category, ':') < 0) {\n+\t\t\t\t# Mediawiki requires the Category\n+\t\t\t\t# prefix, but let's not force the user\n+\t\t\t\t# to specify it.\n+\t\t\t\t$category = \"Category:\" . $category;\n+\t\t\t}\n+\t\t\tmy $mw_pages = $mediawiki->list ( {\n+\t\t\t\taction => 'query',\n+\t\t\t\tlist => 'categorymembers',\n+\t\t\t\tcmtitle => $category,\n+\t\t\t\tcmlimit => 'max' } )\n+\t\t\t || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};\n+\t\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t\t$pages{$page->{title}} = $page;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (!$user_defined) {\n+\t\t# No user-provided list, get the list of pages from\n+\t\t# the API.\n+\t\tmy $mw_pages = $mediawiki->list({\n+\t\t\taction => 'query',\n+\t\t\tlist => 'allpages',\n+\t\t\taplimit => 500,\n+\t\t});\n+\t\tif (!defined($mw_pages)) {\n+\t\t\tprint STDERR \"fatal: could not get the list of wiki pages.\\n\";\n+\t\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\t\texit 1;\n+\t\t}\n+\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t$pages{$page->{title}} = $page;\n+\t\t}\n+\t}\n+\treturn values(%pages);\n+}\n+\n+sub run_git {\n+\topen(my $git, \"-|:encoding(UTF-8)\", \"git \" . $_[0]);\n+\tmy $res = do { local $/; <$git> };\n+\tclose($git);\n+\n+\treturn $res;\n+}\n+\n+\n+sub get_last_local_revision {\n+\t# Get note regarding last mediawiki revision\n+\tmy $note = run_git(\"notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n+\tmy @note_info = split(/ /, $note);\n+\n+\tmy $lastrevision_number;\n+\tif (!(defined($note_info[0]) && $note_info[0] eq \"mediawiki_revision:\")) {\n+\t\tprint STDERR \"No previous mediawiki revision found\";\n+\t\t$lastrevision_number = 0;\n+\t} else {\n+\t\t# Notes are formatted : mediawiki_revision: #number\n+\t\t$lastrevision_number = $note_info[1];\n+\t\tchomp($lastrevision_number);\n+\t\tprint STDERR \"Last local mediawiki revision found is $lastrevision_number\";\n+\t}\n+\treturn $lastrevision_number;\n+}\n+\n+sub get_last_remote_revision {\n+\tmw_connect_maybe();\n+\n+\tmy @pages = get_mw_pages();\n+\n+\tmy $max_rev_num = 0;\n+\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\tpageids => $id,\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});\n+\n+\t\t$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);\n+\t}\n+\n+\tprint STDERR \"Last remote revision found is $max_rev_num.\\n\";\n+\treturn $max_rev_num;\n+}\n+\n+# Clean content before sending it to MediaWiki\n+sub mediawiki_clean {\n+\tmy $string = shift;\n+\tmy $page_created = shift;\n+\t# Mediawiki does not allow blank space at the end of a page and ends with a single \\n.\n+\t# This function right trims a string and adds a \\n at the end to follow this rule\n+\t$string =~ s/\\s+$//;\n+\tif ($string eq \"\" && $page_created) {\n+\t\t# Creating empty pages is forbidden.\n+\t\t$string = EMPTY_CONTENT;\n+\t}\n+\treturn $string.\"\\n\";\n+}\n+\n+# Filter applied on MediaWiki data before adding them to Git\n+sub mediawiki_smudge {\n+\tmy $string = shift;\n+\tif ($string eq EMPTY_CONTENT) {\n+\t\t$string = \"\";\n+\t}\n+\t# This \\n is important. This is due to mediawiki's way to handle end of files.\n+\treturn $string.\"\\n\";\n+}\n+\n+sub mediawiki_clean_filename {\n+\tmy $filename = shift;\n+\t$filename =~ s/@{[SLASH_REPLACEMENT]}/\\//g;\n+\t# [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.\n+\t# Do a variant of URL-encoding, i.e. looks like URL-encoding,\n+\t# but with _ added to prevent MediaWiki from thinking this is\n+\t# an actual special character.\n+\t$filename =~ s/[\\[\\]\\{\\}\\|]/sprintf(\"_%%_%x\", ord($&))/ge;\n+\t# If we use the uri escape before\n+\t# we should unescape here, before anything\n+\n+\treturn $filename;\n+}\n+\n+sub mediawiki_smudge_filename {\n+\tmy $filename = shift;\n+\t$filename =~ s/\\//@{[SLASH_REPLACEMENT]}/g;\n+\t$filename =~ s/ /_/g;\n+\t# Decode forbidden characters encoded in mediawiki_clean_filename\n+\t$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf(\"%c\", hex($1))/ge;\n+\treturn $filename;\n+}\n+\n+sub literal_data {\n+\tmy ($content) = @_;\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+}\n+\n+sub mw_capabilities {\n+\t# Revisions are imported to the private namespace\n+\t# refs/mediawiki/$remotename/ by the helper and fetched into\n+\t# refs/remotes/$remotename later by fetch.\n+\tprint STDOUT \"refspec refs/heads/*:refs/mediawiki/$remotename/*\\n\";\n+\tprint STDOUT \"import\\n\";\n+\tprint STDOUT \"list\\n\";\n+\tprint STDOUT \"push\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_list {\n+\t# MediaWiki do not have branches, we consider one branch arbitrarily\n+\t# called master, and HEAD pointing to it.\n+\tprint STDOUT \"? refs/heads/master\\n\";\n+\tprint STDOUT \"\\@refs/heads/master HEAD\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_option {\n+\tprint STDERR \"remote-helper command 'option $_[0]' not yet implemented\\n\";\n+\tprint STDOUT \"unsupported\\n\";\n+}\n+\n+sub fetch_mw_revisions_for_page {\n+\tmy $page = shift;\n+\tmy $id = shift;\n+\tmy $fetch_from = shift;\n+\tmy @page_revs = ();\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tprop => 'revisions',\n+\t\trvprop => 'ids',\n+\t\trvdir => 'newer',\n+\t\trvstartid => $fetch_from,\n+\t\trvlimit => 500,\n+\t\tpageids => $id,\n+\t};\n+\n+\tmy $revnum = 0;\n+\t# Get 500 revisions at a time due to the mediawiki api limit\n+\twhile (1) {\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\t# Parse each of those 500 revisions\n+\t\tforeach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n+\t\t\tmy $page_rev_ids;\n+\t\t\t$page_rev_ids->{pageid} = $page->{pageid};\n+\t\t\t$page_rev_ids->{revid} = $revision->{revid};\n+\t\t\tpush (@page_revs, $page_rev_ids);\n+\t\t\t$revnum++;\n+\t\t}\n+\t\tlast unless $result->{'query-continue'};\n+\t\t$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n+\t}\n+\tif ($shallow_import && @page_revs) {\n+\t\tprint STDERR \" Found 1 revision (shallow import).\\n\";\n+\t\t@page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);\n+\t\treturn $page_revs[0];\n+\t}\n+\tprint STDERR \" Found \", $revnum, \" revision(s).\\n\";\n+\treturn @page_revs;\n+}\n+\n+sub fetch_mw_revisions {\n+\tmy $pages = shift; my @pages = @{$pages};\n+\tmy $fetch_from = shift;\n+\n+\tmy @revisions = ();\n+\tmy $n = 1;\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tprint STDERR \"page $n/\", scalar(@pages), \": \". $page->{title} .\"\\n\";\n+\t\t$n++;\n+\t\tmy @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);\n+\t\t@revisions = (@page_revs, @revisions);\n+\t}\n+\n+\treturn ($n, @revisions);\n+}\n+\n+sub import_file_revision {\n+\tmy $commit = shift;\n+\tmy %commit = %{$commit};\n+\tmy $full_import = shift;\n+\tmy $n = shift;\n+\n+\tmy $title = $commit{title};\n+\tmy $comment = $commit{comment};\n+\tmy $content = $commit{content};\n+\tmy $author = $commit{author};\n+\tmy $date = $commit{date};\n+\n+\tprint STDOUT \"commit refs/mediawiki/$remotename/master\\n\";\n+\tprint STDOUT \"mark :$n\\n\";\n+\tprint STDOUT \"committer $author <$author\\@$wiki_name> \", $date->epoch, \" +0000\\n\";\n+\tliteral_data($comment);\n+\n+\t# If it's not a clone, we need to know where to start from\n+\tif (!$full_import && $n == 1) {\n+\t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n+\t}\n+\tif ($content ne DELETED_CONTENT) {\n+\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tliteral_data($content);\n+\t\tprint STDOUT \"\\n\\n\";\n+\t} else {\n+\t\tprint STDOUT \"D $title.mw\\n\";\n+\t}\n+\n+\t# mediawiki revision number in the git note\n+\tif ($full_import && $n == 1) {\n+\t\tprint STDOUT \"reset refs/notes/$remotename/mediawiki\\n\";\n+\t}\n+\tprint STDOUT \"commit refs/notes/$remotename/mediawiki\\n\";\n+\tprint STDOUT \"committer $author <$author\\@$wiki_name> \", $date->epoch, \" +0000\\n\";\n+\tliteral_data(\"note added by git-mediawiki\");\n+\tif (!$full_import && $n == 1) {\n+\t\tprint STDOUT \"from refs/notes/$remotename/mediawiki^0\\n\";\n+\t}\n+\tprint STDOUT \"N inline :$n\\n\";\n+\tliteral_data(\"mediawiki_revision: \" . $commit{mw_revision});\n+\tprint STDOUT \"\\n\\n\";\n+}\n+\n+# parse a sequence of\n+# \n+# \n+# \\n\n+# (like batch sequence of import and sequence of push statements)\n+sub get_more_refs {\n+\tmy $cmd = shift;\n+\tmy @refs;\n+\twhile (1) {\n+\t\tmy $line = ;\n+\t\tif ($line =~ m/^$cmd (.*)$/) {\n+\t\t\tpush(@refs, $1);\n+\t\t} elsif ($line eq \"\\n\") {\n+\t\t\treturn @refs;\n+\t\t} else {\n+\t\t\tdie(\"Invalid command in a '$cmd' batch: \". $_);\n+\t\t}\n+\t}\n+}\n+\n+sub mw_import {\n+\t# multiple import commands can follow each other.\n+\tmy @refs = (shift, get_more_refs(\"import\"));\n+\tforeach my $ref (@refs) {\n+\t\tmw_import_ref($ref);\n+\t}\n+\tprint STDOUT \"done\\n\";\n+}\n+\n+sub mw_import_ref {\n+\tmy $ref = shift;\n+\t# The remote helper will call \"import HEAD\" and\n+\t# \"import refs/heads/master\".\n+\t# Since HEAD is a symbolic ref to master (by convention,\n+\t# followed by the output of the command \"list\" that we gave),\n+\t# we don't need to do anything in this case.\n+\tif ($ref eq \"HEAD\") {\n+\t\treturn;\n+\t}\n+\n+\tmw_connect_maybe();\n+\n+\tmy @pages = get_mw_pages();\n+\n+\tprint STDERR \"Searching revisions...\\n\";\n+\tmy $last_local = get_last_local_revision();\n+\tmy $fetch_from = $last_local + 1;\n+\tif ($fetch_from == 1) {\n+\t\tprint STDERR \", fetching from beginning.\\n\";\n+\t} else {\n+\t\tprint STDERR \", fetching from here.\\n\";\n+\t}\n+\tmy ($n, @revisions) = fetch_mw_revisions(\\@pages, $fetch_from);\n+\n+\t# Creation of the fast-import stream\n+\tprint STDERR \"Fetching & writing export data...\\n\";\n+\n+\t$n = 0;\n+\tmy $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined\n+\n+\tforeach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n+\t\t# fetch the content of the pages\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'content|timestamp|comment|user|ids',\n+\t\t\trevids => $pagerevid->{revid},\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});\n+\n+\t\t$n++;\n+\n+\t\tmy %commit;\n+\t\t$commit{author} = $rev->{user} || 'Anonymous';\n+\t\t$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';\n+\t\t$commit{title} = mediawiki_smudge_filename(\n+\t\t\t$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}\n+\t\t );\n+\t\t$commit{mw_revision} = $pagerevid->{revid};\n+\t\t$commit{content} = mediawiki_smudge($rev->{'*'});\n+\n+\t\tif (!defined($rev->{timestamp})) {\n+\t\t\t$last_timestamp++;\n+\t\t} else {\n+\t\t\t$last_timestamp = $rev->{timestamp};\n+\t\t}\n+\t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n+\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\n+\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t}\n+\n+\tif ($fetch_from == 1 && $n == 0) {\n+\t\tprint STDERR \"You appear to have cloned an empty MediaWiki.\\n\";\n+\t\t# Something has to be done remote-helper side. If nothing is done, an error is\n+\t\t# thrown saying that HEAD is refering to unknown object 0000000000000000000\n+\t\t# and the clone fails.\n+\t}\n+}\n+\n+sub error_non_fast_forward {\n+\t# Native git-push would show this after the summary.\n+\t# We can't ask it to display it cleanly, so print it\n+\t# ourselves before.\n+\tprint STDERR \"To prevent you from losing history, non-fast-forward updates were rejected\\n\";\n+\tprint STDERR \"Merge the remote changes (e.g. 'git pull') before pushing again. See the\\n\";\n+\tprint STDERR \"'Note about fast-forwards' section of 'git push --help' for details.\\n\";\n+\n+\tprint STDOUT \"error $_[0] \\\"non-fast-forward\\\"\\n\";\n+\treturn 0;\n+}\n+\n+sub mw_push_file {\n+\tmy $diff_info = shift;\n+\t# $diff_info contains a string in this format:\n+\t# 100644 100644 \n+\tmy @diff_info_split = split(/[ \\t]/, $diff_info);\n+\n+\t# Filename, including .mw extension\n+\tmy $complete_file_name = shift;\n+\t# Commit message\n+\tmy $summary = shift;\n+\n+\tmy $new_sha1 = $diff_info_split[3];\n+\tmy $old_sha1 = $diff_info_split[2];\n+\tmy $page_created = ($old_sha1 eq NULL_SHA1);\n+\tmy $page_deleted = ($new_sha1 eq NULL_SHA1);\n+\t$complete_file_name = mediawiki_clean_filename($complete_file_name);\n+\n+\tif (substr($complete_file_name,-3) eq \".mw\"){\n+\t\tmy $title = substr($complete_file_name,0,-3);\n+\n+\t\tmy $file_content;\n+\t\tif ($page_deleted) {\n+\t\t\t# Deleting a page usually requires\n+\t\t\t# special priviledges. A common\n+\t\t\t# convention is to replace the page\n+\t\t\t# with this content instead:\n+\t\t\t$file_content = DELETED_CONTENT;\n+\t\t} else {\n+\t\t\t$file_content = run_git(\"cat-file blob $new_sha1\");\n+\t\t}\n+\n+\t\tmw_connect_maybe();\n+\n+\t\tmy $result = $mediawiki->edit( {\n+\t\t\taction => 'edit',\n+\t\t\tsummary => $summary,\n+\t\t\ttitle => $title,\n+\t\t\ttext => mediawiki_clean($file_content, $page_created),\n+\t\t\t\t }, {\n+\t\t\t\t\t skip_encoding => 1 # Helps with names with accentuated characters\n+\t\t\t\t }) || die 'Fatal: Error ' .\n+\t\t\t\t $mediawiki->{error}->{code} .\n+\t\t\t\t ' from mediwiki: ' . $mediawiki->{error}->{details};\n+\t\tprint STDERR \"Pushed file : $new_sha1 - $title\\n\";\n+\t} else {\n+\t\tprint STDERR \"$complete_file_name not a mediawiki file (Not pushable on this version of git-remote-mediawiki).\\n\"\n+\t}\n+}\n+\n+sub mw_push {\n+\t# multiple push statements can follow each other\n+\tmy @refsspecs = (shift, get_more_refs(\"push\"));\n+\tmy %status;\n+\tmy $pushed;\n+\tfor my $refspec (@refsspecs) {\n+\t\tunless ($refspec =~ m/^(\\+?)([^:]*):([^:]*)$/) {\n+\t\t\tdie(\"Invalid refspec for push. Expected : or +:\");\n+\t\t}\n+\t\tmy ($force, $local, $remote) = ($1 eq \"+\", $2, $3);\n+\t\tif ($force) {\n+\t\t\tprint STDERR \"Warning: forced push not allowed on a MediaWiki.\\n\";\n+\t\t}\n+\t\tif ($local eq \"\") {\n+\t\t\tprint STDERR \"Cannot delete remote branch on a MediaWiki\\n\";\n+\t\t\tprint STDOUT \"error $remote cannot delete\\n\";\n+\t\t\tnext;\n+\t\t}\n+\t\tif ($remote ne \"refs/heads/master\") {\n+\t\t\tprint STDERR \"Only push to the branch 'master' is supported on a MediaWiki\\n\";\n+\t\t\tprint STDOUT \"error $remote only master allowed\\n\";\n+\t\t\tnext;\n+\t\t}\n+\t\tif (mw_push_revision($local, $remote)) {\n+\t\t\t$pushed = 1;\n+\t\t}\n+\t}\n+\n+\t# Notify Git that the push is done\n+\tprint STDOUT \"\\n\";\n+\n+\tif ($pushed) {\n+\t\tprint STDERR \"Just pushed some revisions to MediaWiki.\\n\";\n+\t\tprint STDERR \"The pushed revisions now have to be re-imported, and your current branch\\n\";\n+\t\tprint STDERR \"needs to be updated with these re-imported commits. You can do this with\\n\";\n+\t\tprint STDERR \"\\n\";\n+\t\tprint STDERR \" git pull --rebase\\n\";\n+\t\tprint STDERR \"\\n\";\n+\t}\n+}\n+\n+sub mw_push_revision {\n+\tmy $local = shift;\n+\tmy $remote = shift; # actually, this has to be \"refs/heads/master\" at this point.\n+\tmy $last_local_revid = get_last_local_revision();\n+\tprint STDERR \".\\n\"; # Finish sentence started by get_last_local_revision()\n+\tmy $last_remote_revid = get_last_remote_revision();\n+\n+\t# Get sha1 of commit pointed by local HEAD\n+\tmy $HEAD_sha1 = run_git(\"rev-parse $local 2>/dev/null\"); chomp($HEAD_sha1);\n+\t# Get sha1 of commit pointed by remotes/$remotename/master\n+\tmy $remoteorigin_sha1 = run_git(\"rev-parse refs/remotes/$remotename/master 2>/dev/null\");\n+\tchomp($remoteorigin_sha1);\n+\n+\tif ($last_local_revid > 0 &&\n+\t $last_local_revid < $last_remote_revid){\n+\t\treturn error_non_fast_forward($remote);\n+\t}\n+\n+\tif ($HEAD_sha1 eq $remoteorigin_sha1) {\n+\t\t# nothing to push\n+\t\treturn 0;\n+\t}\n+\n+\t# Get every commit in between HEAD and refs/remotes/origin/master,\n+\t# including HEAD and refs/remotes/origin/master\n+\tmy @commit_pairs = ();\n+\tif ($last_local_revid > 0) {\n+\t\tmy $parsed_sha1 = $remoteorigin_sha1;\n+\t\t# Find a path from last MediaWiki commit to pushed commit\n+\t\twhile ($parsed_sha1 ne $HEAD_sha1) {\n+\t\t\tmy @commit_info = grep(/^$parsed_sha1/, split(/\\n/, run_git(\"rev-list --children $local\")));\n+\t\t\tif (!@commit_info) {\n+\t\t\t\treturn error_non_fast_forward($remote);\n+\t\t\t}\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $commit_info[0]);\n+\t\t\t# $commit_info_split[1] is the sha1 of the commit to export\n+\t\t\t# $commit_info_split[0] is the sha1 of its direct child\n+\t\t\tpush (@commit_pairs, \\@commit_info_split);\n+\t\t\t$parsed_sha1 = $commit_info_split[1];\n+\t\t}\n+\t} else {\n+\t\t# No remote mediawiki revision. Export the whole\n+\t\t# history (linearized with --first-parent)\n+\t\tprint STDERR \"Warning: no common ancestor, pushing complete history\\n\";\n+\t\tmy $history = run_git(\"rev-list --first-parent --children $local\");\n+\t\tmy @history = split('\\n', $history);\n+\t\t@history = @history[1..$#history];\n+\t\tforeach my $line (reverse @history) {\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $line);\n+\t\t\tpush (@commit_pairs, \\@commit_info_split);\n+\t\t}\n+\t}\n+\n+\tforeach my $commit_info_split (@commit_pairs) {\n+\t\tmy $sha1_child = @{$commit_info_split}[0];\n+\t\tmy $sha1_commit = @{$commit_info_split}[1];\n+\t\tmy $diff_infos = run_git(\"diff-tree -r --raw -z $sha1_child $sha1_commit\");\n+\t\t# TODO: we could detect rename, and encode them with a #redirect on the wiki.\n+\t\t# TODO: for now, it's just a delete+add\n+\t\tmy @diff_info_list = split(/\\0/, $diff_infos);\n+\t\t# Keep the first line of the commit message as mediawiki comment for the revision\n+\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $sha1_commit\")))[0];\n+\t\tchomp($commit_msg);\n+\t\t# Push every blob\n+\t\twhile (@diff_info_list) {\n+\t\t\t# git diff-tree -z gives an output like\n+\t\t\t# \\0\\0\n+\t\t\t# \\0\\0\n+\t\t\t# and we've split on \\0.\n+\t\t\tmy $info = shift(@diff_info_list);\n+\t\t\tmy $file = shift(@diff_info_list);\n+\t\t\tmw_push_file($info, $file, $commit_msg);\n+\t\t}\n+\t}\n+\n+\tprint STDOUT \"ok $remote\\n\";\n+\treturn 1;\n+}\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt\nnew file mode 100644\nindex 0000000..4d211f5\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki.txt\n@@ -0,0 +1,7 @@\n+Git-Mediawiki is a project which aims the creation of a gate\n+between git and mediawiki, allowing git users to push and pull\n+objects from mediawiki just as one would do with a classic git\n+repository thanks to remote-helpers.\n+\n+For more information, visit the wiki at\n+https://github.com/Bibzball/Git-Mediawiki/wiki\n-- \n1.7.7.rc0.78.ge4b5a.dirty\n"},{"id":"174634","messageId":"CAGdFq_gu=SyjUnUS1bcjPrcPPtKVt+UjDBvBmZqosk+OuDFDHw@mail.gmail.com","threadId":"28226","inReplyTo":"1314809708-8177-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-31T17:03:54Z","receivedAt":"2011-08-31T17:03:54Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\n2011/8/31 Matthieu Moy :\n> So, after understanding better how import works, here's an updated\n> patch that gets rid of the hacky workaround to terminate and send the\n> \"done\" command at the right time.\n\nSo what do you think of the way the protocol works now? Do you agree\nthat (modulo lacking docs) it is better than previously?\n\n> Actually, push had the same problem but it just went unnoticed (the\n> remote has just one branch, so it's silly to try to push multiple\n> branches at the same time ...). This version handles push more\n> cleanly, giving accurate error message in cases like\n>\n>  git push origin :master\n>  git push origin foo bar master\n>\n> or perhaps more commonly\n>\n>  git push --all\n>\n> in a repository with branches other than master.\n\nMy perl skills are minimal, but I'm curious how/where you implemented\nthis? Is this something that we can port to remote-testgit to document\nthe CPB on handling such things?\n\n\n\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174636","messageId":"vpq7h5tbia6.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"CAGdFq_gu=SyjUnUS1bcjPrcPPtKVt+UjDBvBmZqosk+OuDFDHw@mail.gmail.com","subject":"Re: [PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-08-31T17:30:25Z","receivedAt":"2011-08-31T17:30:25Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Sverre Rabbelier writes:\n\n> Heya,\n>\n> 2011/8/31 Matthieu Moy :\n>> So, after understanding better how import works, here's an updated\n>> patch that gets rid of the hacky workaround to terminate and send the\n>> \"done\" command at the right time.\n>\n> So what do you think of the way the protocol works now? Do you agree\n> that (modulo lacking docs) it is better than previously?\n\nI'm not sure I understood exactly how it was before, but the current\nprotocol seems indeed at least reasonable:\n\n* It's possible to specify a batch of imports, so the remote-helper has\n freedom to optimize the import of multiple refs.\n\n* A batch of import is clearly delimited, both on stdin and stdout, so\n it is possible to alternate import batches and other commands.\n\nI still have a few complaints, because even with a better doc, I still\nfound the debugging a bit hard. To make it easy for remote-helpers\nauthors, I think the transport-helper could have an explicit \"done\"\ncommand, so that the command stream look like\n\nimport foo\nimport bar\n\\n\ndone\n\ninstead of\n\nimport foo\nimport bar\n\\n\n\\n\n\nand to let the remote-helper's code be like\n\nwhile($cmd = ) {\n if ($cmd eq \"command1\") {\n do something;\n } elsif ($cmd eq \"command2\") {\n something else;\n } elsif ($cmd eq \"done\") {\n exit properly;\n }\n}\n\nI'm not sure whether changing this now is worth the trouble though.\n\nI'd have appreciated if the transport-helper had given me an explicit\nerror message when writting to a broken pipe too. I finally got it with\ngdb, but lost some time trying to understand (especially painfull since\nthere was a race condition between the remote-helper termination and git\nwritting to it, so the bug wasn't reproducible).\n\n>> Actually, push had the same problem but it just went unnoticed (the\n>> remote has just one branch, so it's silly to try to push multiple\n>> branches at the same time ...). This version handles push more\n>> cleanly, giving accurate error message in cases like\n>>\n>>  git push origin :master\n>>  git push origin foo bar master\n>>\n>> or perhaps more commonly\n>>\n>>  git push --all\n>>\n>> in a repository with branches other than master.\n>\n> My perl skills are minimal, but I'm curious how/where you implemented\n> this?\n\nHere:\n\n+\tfor my $refspec (@refsspecs) {\n+\t\tunless ($refspec =~ m/^(\\+?)([^:]*):([^:]*)$/) {\n+\t\t\tdie(\"Invalid refspec for push. Expected : or +:\");\n+\t\t}\n+\t\tmy ($force, $local, $remote) = ($1 eq \"+\", $2, $3);\n\nAt this point, $force is a boolean saying whether there were a +, and\n$local and $remote are as you can guess.\n\n+\t\tif ($force) {\n+\t\t\tprint STDERR \"Warning: forced push not allowed on a MediaWiki.\\n\";\n+\t\t}\n+\t\tif ($local eq \"\") {\n+\t\t\tprint STDERR \"Cannot delete remote branch on a MediaWiki\\n\";\n+\t\t\tprint STDOUT \"error $remote cannot delete\\n\";\n\nprint STDERR goes to the console (i.e. to the user), and print STDOUT\ngoes to the Git's transport-helper.\n\n+\t\t\tnext;\n+\t\t}\n+\t\tif ($remote ne \"refs/heads/master\") {\n+\t\t\tprint STDERR \"Only push to the branch 'master' is supported on a MediaWiki\\n\";\n+\t\t\tprint STDOUT \"error $remote only master allowed\\n\";\n+\t\t\tnext;\n+\t\t}\n\n> Is this something that we can port to remote-testgit to document the\n> CPB on handling such things?\n\nCPB = ?\n\nActually, my case is very particular, since the only thing to do with\nbranches is to make sure the user doesn't use them. In remote-testgit,\nthere are actually branches.\n\nAnd testgit use the undocumented \"export\" feature, which does not seem\nto support branch deletion:\n\ngit push origin :branch2\nfatal: remote-helpers do not support ref deletion\nmoy@bauges:/tmp/clone$ Traceback (most recent call last):\n File \"/home/moy/local/usr-squeeze/libexec/git-core/git-remote-testgit\", line 252, in \n sys.exit(main(sys.argv))\n File \"/home/moy/local/usr-squeeze/libexec/git-core/git-remote-testgit\", line 249, in main\n more = read_one_line(repo)\n File \"/home/moy/local/usr-squeeze/libexec/git-core/git-remote-testgit\", line 215, in read_one_line\n sys.stdout.flush()\nIOError: [Errno 32] Broken pipe\n\n(This comes from\n\ntransport-helper.c:750: die(\"remote-helpers do not support ref deletion\");\n\ncalled before starting the exporter)\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174638","messageId":"1314814498-13699-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"1314809222-30528-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH] (short) documentation for the testgit remote helper","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-08-31T18:14:58Z","receivedAt":"2011-08-31T18:14:58Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"While it's not a command meant to be used by actual users (hence, not\nmentionned in git(1)), this command is a very precious help for\nremote-helpers authors.\n\nThe best place for such technical doc is the source code, but users may\nnot find it without a link in a manpage.\n\nSigned-off-by: Matthieu Moy \n---\n\nWhile we're improving the docs, this is one more thing that would have\nsaved me some time ...\n\n Documentation/git-remote-helpers.txt | 2 ++\n Documentation/git-remote-testgit.txt | 30 ++++++++++++++++++++++++++++++\n git-remote-testgit.py | 14 ++++++++++++++\n 3 files changed, 46 insertions(+), 0 deletions(-)\n create mode 100644 Documentation/git-remote-testgit.txt\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 526fc6a..674797c 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -362,6 +362,8 @@ SEE ALSO\n --------\n linkgit:git-remote[1]\n \n+linkgit:git-remote-testgit[1]\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-remote-testgit.txt b/Documentation/git-remote-testgit.txt\nnew file mode 100644\nindex 0000000..2a67d45\n--- /dev/null\n+++ b/Documentation/git-remote-testgit.txt\n@@ -0,0 +1,30 @@\n+git-remote-testgit(1)\n+=====================\n+\n+NAME\n+----\n+git-remote-testgit - Example remote-helper\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+git clone testgit:: []\n+\n+DESCRIPTION\n+-----------\n+\n+This command is a simple remote-helper, that is used both as a\n+testcase for the remote-helper functionality, and as an example to\n+show remote-helper authors one possible implementation.\n+\n+The best way to learn more is to read the comments and source code in\n+'git-remote-testgit.py'.\n+\n+SEE ALSO\n+--------\n+linkgit:git-remote-helpers[1]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/git-remote-testgit.py b/git-remote-testgit.py\nindex e9c832b..91d4409 100644\n--- a/git-remote-testgit.py\n+++ b/git-remote-testgit.py\n@@ -1,5 +1,19 @@\n #!/usr/bin/env python\n \n+# This command is a simple remote-helper, that is used both as a\n+# testcase for the remote-helper functionality, and as an example to\n+# show remote-helper authors one possible implementation.\n+#\n+# This is a Git <-> Git importer/exporter, that simply uses git\n+# fast-import and git fast-export to consume and produce fast-import\n+# streams.\n+#\n+# To understand better the way things work, one can set the variable\n+# \"static int debug\" in transport-helper.c to 1, and/or the \"DEBUG\"\n+# variable in git_remote_helpers/util.py to True, and try various\n+# commands.\n+\n+\n # hashlib is only available in python >= 2.5\n try:\n import hashlib\n-- \n1.7.7.rc0.75.g56f27\n"},{"id":"174650","messageId":"7vobz5xg7u.fsf@alter.siamese.dyndns.org","threadId":"28226","inReplyTo":"vpq7h5tbia6.fsf@bauges.imag.fr","subject":"Re: [PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-09-01T00:24:05Z","receivedAt":"2011-09-01T00:24:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> Here:\n>\n> +\tfor my $refspec (@refsspecs) {\n> +\t\tunless ($refspec =~ m/^(\\+?)([^:]*):([^:]*)$/) {\n> +\t\t\tdie(\"Invalid refspec for push. Expected : or +:\");\n> +\t\t}\n> +\t\tmy ($force, $local, $remote) = ($1 eq \"+\", $2, $3);\n>\n> At this point, $force is a boolean saying whether there were a +, and\n> $local and $remote are as you can guess.\n\nIt may be slightly more Perl-ish to hoist the \"0-or-1\" outside the group\nand rely on $1 becoming undef, like this:\n\n my ($force, $local, $remote) = $refspec =~ /^(\\+)?([^:]*):([^:]*)$/\n\t\tor die(...);\n\nEven though it largely is a matter of taste, I think.\n"},{"id":"174660","messageId":"vpq39ggomtu.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"7vobz5xg7u.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-09-01T05:26:05Z","receivedAt":"2011-09-01T05:26:05Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> Matthieu Moy writes:\n>\n>> Here:\n>>\n>> +\tfor my $refspec (@refsspecs) {\n>> +\t\tunless ($refspec =~ m/^(\\+?)([^:]*):([^:]*)$/) {\n>> +\t\t\tdie(\"Invalid refspec for push. Expected : or +:\");\n>> +\t\t}\n>> +\t\tmy ($force, $local, $remote) = ($1 eq \"+\", $2, $3);\n>>\n>> At this point, $force is a boolean saying whether there were a +, and\n>> $local and $remote are as you can guess.\n>\n> It may be slightly more Perl-ish to hoist the \"0-or-1\" outside the group\n> and rely on $1 becoming undef, like this:\n>\n> my ($force, $local, $remote) = $refspec =~ /^(\\+)?([^:]*):([^:]*)$/\n> \t\tor die(...);\n\nThanks, I didn't know I could do this.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174664","messageId":"CAGdFq_jfCzy71AL3XSJ9q8LVNPUCYEY=yrwpJppP1asW1GoXmg@mail.gmail.com","threadId":"28226","inReplyTo":"1314809222-30528-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH] git-remote-helpers.txt: explain how import works with multiple refs","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-09-01T11:24:07Z","receivedAt":"2011-09-01T11:24:07Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\n[+Jonathan Nieder]\n\nOn Wed, Aug 31, 2011 at 18:47, Matthieu Moy wrote:\n> This is important for two reasons:\n>\n> * when two \"import\" lines follow each other, only one \"done\" command\n>  should be issued in the fast-import stream, not one per \"import\".\n>\n> * The blank line terminating an import command should not be confused\n>  with the one terminating the sequence of commands.\n>\n> While we're there, illustrate the corresponding explanation for push\n> batches with an example.\n\nThank you!\n\nAcked-by: Sverre Rabbelier wrote:\n> +# To understand better the way things work, one can set the variable\n> +# \"static int debug\" in transport-helper.c to 1, and/or the \"DEBUG\"\n> +# variable in git_remote_helpers/util.py to True, and try various\n> +# commands.\n\nBoth are controlled by an environmental variable, it would be better\nto mention these directly.\n\nI think that for remote-testgit it is GIT_DEBUG_TESTGIT=1.\n\nOther than that:\n\nAcked-by: Sverre Rabbelier \n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174671","messageId":"vpqd3fk1cq5.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"CAGdFq_grmJLoTt7JMUuoXrd02Gx8JdcEL-wa7YQ=-FkurRqfWA@mail.gmail.com","subject":"Re: [PATCH] (short) documentation for the testgit remote helper","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-09-01T15:52:50Z","receivedAt":"2011-09-01T15:52:50Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Sverre Rabbelier writes:\n\n> Heya,\n>\n> On Wed, Aug 31, 2011 at 20:14, Matthieu Moy wrote:\n>> +# To understand better the way things work, one can set the variable\n>> +# \"static int debug\" in transport-helper.c to 1, and/or the \"DEBUG\"\n>> +# variable in git_remote_helpers/util.py to True, and try various\n>> +# commands.\n>\n> Both are controlled by an environmental variable, it would be better\n> to mention these directly.\n>\n> I think that for remote-testgit it is GIT_DEBUG_TESTGIT=1.\n\nOops, I had missed it. Will resend.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"174679","messageId":"1314895778-17482-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"vpqd3fk1cq5.fsf@bauges.imag.fr","subject":"[PATCH 1/2 v2] Documentation/git-remote-helpers: explain how import works with multiple refs","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-09-01T16:49:37Z","receivedAt":"2011-09-01T16:49:37Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"This is important for two reasons:\n\n* when two \"import\" lines follow each other, only one \"done\" command\n should be issued in the fast-import stream, not one per \"import\".\n\n* The blank line terminating an import command should not be confused\n with the one terminating the sequence of commands.\n\nWhile we're there, illustrate the corresponding explanation for push\nbatches with an example.\n\nSigned-off-by: Matthieu Moy \n---\nNo real change since v1, but resending together with the other to make\nit a \"patch serie\" ;-)\n\n Documentation/git-remote-helpers.txt | 22 +++++++++++++++++++++-\n 1 files changed, 21 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 760b164..526fc6a 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -241,7 +241,22 @@ Supported if the helper has the \"fetch\" capability.\n 'push' +:::\n \tPushes the given local commit or branch to the\n \tremote branch described by . A batch sequence of\n-\tone or more push commands is terminated with a blank line.\n+\tone or more 'push' commands is terminated with a blank line\n+\t(if there is only one reference to push, a single 'push' command\n+\tis followed by a blank line). For example, the following would\n+\tbe two batches of 'push', the first asking the remote-helper\n+\tto push the local ref 'master' to the remote ref 'master' and\n+\tthe local 'HEAD' to the remote 'branch', and the second\n+\tasking to push ref 'foo' to ref 'bar' (forced update requested\n+\tby the '+').\n++\n+------------\n+push refs/heads/master:refs/heads/master\n+push HEAD:refs/heads/branch\n+\\n\n+push +refs/heads/foo:refs/heads/bar\n+\\n\n+------------\n +\n Zero or more protocol options may be entered after the last 'push'\n command, before the batch's terminating blank line.\n@@ -266,6 +281,11 @@ Supported if the helper has the \"push\" capability.\n Especially useful for interoperability with a foreign versioning\n system.\n +\n+Just like 'push', a batch sequence of one or more 'import' is\n+terminated with a blank line. For each batch of 'import', the remote\n+helper should produce a fast-import stream terminated by a 'done'\n+command.\n++\n Supported if the helper has the \"import\" capability.\n \n 'connect' ::\n-- \n1.7.7.rc0.75.g56f27\n"},{"id":"174680","messageId":"1314895778-17482-2-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"1314895778-17482-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 2/2 v2] (short) documentation for the testgit remote helper","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-09-01T16:49:38Z","receivedAt":"2011-09-01T16:49:38Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"While it's not a command meant to be used by actual users (hence, not\nmentionned in git(1)), this command is a very precious help for\nremote-helpers authors.\n\nThe best place for such technical doc is the source code, but users may\nnot find it without a link in a manpage.\n\nSigned-off-by: Matthieu Moy \n---\nchange since v1: document environment variables for debugging, not\nhacking the source code.\n\n Documentation/git-remote-helpers.txt | 2 ++\n Documentation/git-remote-testgit.txt | 30 ++++++++++++++++++++++++++++++\n git-remote-testgit.py | 13 +++++++++++++\n 3 files changed, 45 insertions(+), 0 deletions(-)\n create mode 100644 Documentation/git-remote-testgit.txt\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 526fc6a..674797c 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -362,6 +362,8 @@ SEE ALSO\n --------\n linkgit:git-remote[1]\n \n+linkgit:git-remote-testgit[1]\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-remote-testgit.txt b/Documentation/git-remote-testgit.txt\nnew file mode 100644\nindex 0000000..2a67d45\n--- /dev/null\n+++ b/Documentation/git-remote-testgit.txt\n@@ -0,0 +1,30 @@\n+git-remote-testgit(1)\n+=====================\n+\n+NAME\n+----\n+git-remote-testgit - Example remote-helper\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+git clone testgit:: []\n+\n+DESCRIPTION\n+-----------\n+\n+This command is a simple remote-helper, that is used both as a\n+testcase for the remote-helper functionality, and as an example to\n+show remote-helper authors one possible implementation.\n+\n+The best way to learn more is to read the comments and source code in\n+'git-remote-testgit.py'.\n+\n+SEE ALSO\n+--------\n+linkgit:git-remote-helpers[1]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/git-remote-testgit.py b/git-remote-testgit.py\nindex e9c832b..3dc4851 100644\n--- a/git-remote-testgit.py\n+++ b/git-remote-testgit.py\n@@ -1,5 +1,18 @@\n #!/usr/bin/env python\n \n+# This command is a simple remote-helper, that is used both as a\n+# testcase for the remote-helper functionality, and as an example to\n+# show remote-helper authors one possible implementation.\n+#\n+# This is a Git <-> Git importer/exporter, that simply uses git\n+# fast-import and git fast-export to consume and produce fast-import\n+# streams.\n+#\n+# To understand better the way things work, one can activate debug\n+# traces by setting (to any value) the environment variables\n+# GIT_TRANSPORT_HELPER_DEBUG and GIT_DEBUG_TESTGIT, to see messages\n+# from the transport-helper side, or from this example remote-helper.\n+\n # hashlib is only available in python >= 2.5\n try:\n import hashlib\n-- \n1.7.7.rc0.75.g56f27\n"},{"id":"174681","messageId":"1314896096-17544-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"vpq39ggomtu.fsf@bauges.imag.fr","subject":"[PATCH 0/2] Git-MediaWiki","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-09-01T16:54:54Z","receivedAt":"2011-09-01T16:54:54Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"First patch is just a resend of v6 with style improvements (Junio's\nremark, and whitespace fixes). The second patch is new, it can be\nsquashed into the first if needed, but I've left it appart since I\nthink \"what to do with metadata on push?\" is one of the crucial\nquestions to be answered when writting remote-helpers, so it may\n(hopefully) lead to fruitfull discussions.\n\nJeremie Nikaes (1):\n Add a remote helper to interact with mediawiki (fetch & push)\n\nMatthieu Moy (1):\n git-remote-mediawiki: allow push to set MediaWiki metadata\n\n contrib/mw-to-git/git-remote-mediawiki | 787 ++++++++++++++++++++++++++++\n contrib/mw-to-git/git-remote-mediawiki.txt | 7 +\n 2 files changed, 794 insertions(+), 0 deletions(-)\n create mode 100755 contrib/mw-to-git/git-remote-mediawiki\n create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt\n\n-- \n1.7.7.rc0.75.g56f27\n"},{"id":"174683","messageId":"1314896096-17544-2-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"1314896096-17544-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 1/2 v7] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-09-01T16:54:55Z","receivedAt":"2011-09-01T16:54:55Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"From: Jeremie Nikaes \n\nImplement a gate between git and mediawiki, allowing git users to push\nand pull objects from mediawiki just as one would do with a classic git\nrepository thanks to remote-helpers.\n\nThe following packages need to be installed (available on common\nrepositories):\n\n libmediawiki-api-perl\n libdatetime-format-iso8601-perl\n\nUse remote helpers in order to be as transparent as possible to the git\nuser.\n\nDownload Mediawiki revisions through the Mediawiki API and then\nfast-import into git.\n\nMediawiki revision number and git commits are linked thanks to notes\nbound to commits.\n\nThe import part is done on a refs/mediawiki/ branch before\ncoming to refs/remote/origin/master (Huge thanks to Jonathan Nieder\nfor his help)\n\nWe use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but\nwe also read the output of Git commands in UTF-8 with the small helper\nrun_git, and write to the console (STDERR) in UTF-8. This allows a\nseamless use of non-ascii characters in page titles, but hasn't been\ntested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames\ncould raise problems if different file systems handle UTF-8 filenames\ndifferently. A uri_escape of mediawiki filenames could be imaginable, and\nis still to be discussed further.\n\nPartial cloning is supported using one of:\n\ngit clone -c remote.origin.pages='A_Page Another_Page' mediawiki::http://wikiurl\n\ngit clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl\n\ngit clone -c remote.origin.shallow='True' mediawiki::http://wikiurl\n\nThanks to notes metadata, it is possible to compare remote and local last\nmediawiki revision to warn non-fast forward pushes and \"everything\nup-to-date\" case.\n\nWhen allowed, push looks for each commit between remotes/origin/master\nand HEAD, catches every blob related to these commit and push them in\nchronological order. To do so, it uses git rev-list --children HEAD and\ntravels the tree from remotes/origin/master to HEAD through children. In\nother words:\n\n* Shortest path from remotes/origin/master to HEAD\n* For each commit encountered, push blobs related to this commit\n\nSigned-off-by: Jérémie Nikaes \nSigned-off-by: Arnaud Lacurie \nSigned-off-by: Claire Fousse \nSigned-off-by: David Amouyal \nSigned-off-by: Matthieu Moy \nSigned-off-by: Sylvain Boulmé \n\n---\nChanges since v6:\n\n* Regexp match simplification, thanks to Junio\n\n* Trivial whitespace fixes, for consistancy.\n\n contrib/mw-to-git/git-remote-mediawiki | 756 ++++++++++++++++++++++++++++\n contrib/mw-to-git/git-remote-mediawiki.txt | 7 +\n 2 files changed, 763 insertions(+), 0 deletions(-)\n create mode 100755 contrib/mw-to-git/git-remote-mediawiki\n create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nnew file mode 100755\nindex 0000000..d4d1198\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -0,0 +1,756 @@\n+#! /usr/bin/perl\n+\n+# Copyright (C) 2011\n+# Jérémie Nikaes \n+# Arnaud Lacurie \n+# Claire Fousse \n+# David Amouyal \n+# Matthieu Moy \n+# License: GPL v2 or later\n+\n+# Gateway between Git and MediaWiki.\n+# https://github.com/Bibzball/Git-Mediawiki/wiki\n+#\n+# Known limitations:\n+#\n+# - Only wiki pages are managed, no support for [[File:...]]\n+# attachments.\n+#\n+# - Poor performance in the best case: it takes forever to check\n+# whether we're up-to-date (on fetch or push) or to fetch a few\n+# revisions from a large wiki, because we use exclusively a\n+# page-based synchronization. We could switch to a wiki-wide\n+# synchronization when the synchronization involves few revisions\n+# but the wiki is large.\n+#\n+# - Git renames could be turned into MediaWiki renames (see TODO\n+# below)\n+#\n+# - login/password support requires the user to write the password\n+# cleartext in a file (see TODO below).\n+#\n+# - No way to import \"one page, and all pages included in it\"\n+#\n+# - Multiple remote MediaWikis have not been very well tested.\n+\n+use strict;\n+use MediaWiki::API;\n+use DateTime::Format::ISO8601;\n+use encoding 'utf8';\n+\n+# use encoding 'utf8' doesn't change STDERROR\n+# but we're going to output UTF-8 filenames to STDERR\n+binmode STDERR, \":utf8\";\n+\n+use URI::Escape;\n+use warnings;\n+\n+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced\n+use constant SLASH_REPLACEMENT => \"%2F\";\n+\n+# It's not always possible to delete pages (may require some\n+# priviledges). Deleted pages are replaced with this content.\n+use constant DELETED_CONTENT => \"[[Category:Deleted]]\\n\";\n+\n+# It's not possible to create empty pages. New empty files in Git are\n+# sent with this content instead.\n+use constant EMPTY_CONTENT => \"\\n\";\n+\n+# used to reflect file creation or deletion in diff.\n+use constant NULL_SHA1 => \"0000000000000000000000000000000000000000\";\n+\n+my $remotename = $ARGV[0];\n+my $url = $ARGV[1];\n+\n+# Accept both space-separated and multiple keys in config file.\n+# Spaces should be written as _ anyway because we'll use chomp.\n+my @tracked_pages = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".pages\"));\n+chomp(@tracked_pages);\n+\n+# Just like @tracked_pages, but for MediaWiki categories.\n+my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.\". $remotename .\".categories\"));\n+chomp(@tracked_categories);\n+\n+my $wiki_login = run_git(\"config --get remote.\". $remotename .\".mwLogin\");\n+# TODO: ideally, this should be able to read from keyboard, but we're\n+# inside a remote helper, so our stdin is connect to git, not to a\n+# terminal.\n+my $wiki_passwd = run_git(\"config --get remote.\". $remotename .\".mwPassword\");\n+chomp($wiki_login);\n+chomp($wiki_passwd);\n+\n+# Import only last revisions (both for clone and fetch)\n+my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shallow\");\n+chomp($shallow_import);\n+$shallow_import = ($shallow_import eq \"true\");\n+\n+my $wiki_name = $url;\n+$wiki_name =~ s/[^\\/]*:\\/\\///;\n+\n+# Commands parser\n+my $entry;\n+my @cmd;\n+while () {\n+\tchomp;\n+\t@cmd = split(/ /);\n+\tif (defined($cmd[0])) {\n+\t\t# Line not blank\n+\t\tif ($cmd[0] eq \"capabilities\") {\n+\t\t\tdie(\"Too many arguments for capabilities\") unless (!defined($cmd[1]));\n+\t\t\tmw_capabilities();\n+\t\t} elsif ($cmd[0] eq \"list\") {\n+\t\t\tdie(\"Too many arguments for list\") unless (!defined($cmd[2]));\n+\t\t\tmw_list($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"import\") {\n+\t\t\tdie(\"Invalid arguments for import\") unless ($cmd[1] ne \"\" && !defined($cmd[2]));\n+\t\t\tmw_import($cmd[1]);\n+\t\t} elsif ($cmd[0] eq \"option\") {\n+\t\t\tdie(\"Too many arguments for option\") unless ($cmd[1] ne \"\" && $cmd[2] ne \"\" && !defined($cmd[3]));\n+\t\t\tmw_option($cmd[1],$cmd[2]);\n+\t\t} elsif ($cmd[0] eq \"push\") {\n+\t\t\tmw_push($cmd[1]);\n+\t\t} else {\n+\t\t\tprint STDERR \"Unknown command. Aborting...\\n\";\n+\t\t\tlast;\n+\t\t}\n+\t} else {\n+\t\t# blank line: we should terminate\n+\t\tlast;\n+\t}\n+\n+\tBEGIN { $| = 1 } # flush STDOUT, to make sure the previous\n+\t\t\t # command is fully processed.\n+}\n+\n+########################## Functions ##############################\n+\n+# MediaWiki API instance, created lazily.\n+my $mediawiki;\n+\n+sub mw_connect_maybe {\n+\tif ($mediawiki) {\n+\t return;\n+\t}\n+\t$mediawiki = MediaWiki::API->new;\n+\t$mediawiki->{config}->{api_url} = \"$url/api.php\";\n+\tif ($wiki_login) {\n+\t\tif (!$mediawiki->login({\n+\t\t\tlgname => $wiki_login,\n+\t\t\tlgpassword => $wiki_passwd,\n+\t\t})) {\n+\t\t\tprint STDERR \"Failed to log in mediawiki user \\\"$wiki_login\\\" on $url\\n\";\n+\t\t\tprint STDERR \"(error \" .\n+\t\t\t $mediawiki->{error}->{code} . ': ' .\n+\t\t\t $mediawiki->{error}->{details} . \")\\n\";\n+\t\t\texit 1;\n+\t\t} else {\n+\t\t\tprint STDERR \"Logged in with user \\\"$wiki_login\\\".\\n\";\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_first_pages {\n+\tmy $some_pages = shift;\n+\tmy @some_pages = @{$some_pages};\n+\n+\tmy $pages = shift;\n+\n+\t# pattern 'page1|page2|...' required by the API\n+\tmy $titles = join('|', @some_pages);\n+\n+\tmy $mw_pages = $mediawiki->api({\n+\t\taction => 'query',\n+\t\ttitles => $titles,\n+\t});\n+\tif (!defined($mw_pages)) {\n+\t\tprint STDERR \"fatal: could not query the list of wiki pages.\\n\";\n+\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\texit 1;\n+\t}\n+\twhile (my ($id, $page) = each(%{$mw_pages->{query}->{pages}})) {\n+\t\tif ($id < 0) {\n+\t\t\tprint STDERR \"Warning: page $page->{title} not found on wiki\\n\";\n+\t\t} else {\n+\t\t\t$pages->{$page->{title}} = $page;\n+\t\t}\n+\t}\n+}\n+\n+sub get_mw_pages {\n+\tmw_connect_maybe();\n+\n+\tmy %pages; # hash on page titles to avoid duplicates\n+\tmy $user_defined;\n+\tif (@tracked_pages) {\n+\t\t$user_defined = 1;\n+\t\t# The user provided a list of pages titles, but we\n+\t\t# still need to query the API to get the page IDs.\n+\n+\t\tmy @some_pages = @tracked_pages;\n+\t\twhile (@some_pages) {\n+\t\t\tmy $last = 50;\n+\t\t\tif ($#some_pages < $last) {\n+\t\t\t\t$last = $#some_pages;\n+\t\t\t}\n+\t\t\tmy @slice = @some_pages[0..$last];\n+\t\t\tget_mw_first_pages(\\@slice, \\%pages);\n+\t\t\t@some_pages = @some_pages[51..$#some_pages];\n+\t\t}\n+\t}\n+\tif (@tracked_categories) {\n+\t\t$user_defined = 1;\n+\t\tforeach my $category (@tracked_categories) {\n+\t\t\tif (index($category, ':') < 0) {\n+\t\t\t\t# Mediawiki requires the Category\n+\t\t\t\t# prefix, but let's not force the user\n+\t\t\t\t# to specify it.\n+\t\t\t\t$category = \"Category:\" . $category;\n+\t\t\t}\n+\t\t\tmy $mw_pages = $mediawiki->list( {\n+\t\t\t\taction => 'query',\n+\t\t\t\tlist => 'categorymembers',\n+\t\t\t\tcmtitle => $category,\n+\t\t\t\tcmlimit => 'max' } )\n+\t\t\t || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};\n+\t\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t\t$pages{$page->{title}} = $page;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (!$user_defined) {\n+\t\t# No user-provided list, get the list of pages from\n+\t\t# the API.\n+\t\tmy $mw_pages = $mediawiki->list({\n+\t\t\taction => 'query',\n+\t\t\tlist => 'allpages',\n+\t\t\taplimit => 500,\n+\t\t});\n+\t\tif (!defined($mw_pages)) {\n+\t\t\tprint STDERR \"fatal: could not get the list of wiki pages.\\n\";\n+\t\t\tprint STDERR \"fatal: '$url' does not appear to be a mediawiki\\n\";\n+\t\t\tprint STDERR \"fatal: make sure '$url/api.php' is a valid page.\\n\";\n+\t\t\texit 1;\n+\t\t}\n+\t\tforeach my $page (@{$mw_pages}) {\n+\t\t\t$pages{$page->{title}} = $page;\n+\t\t}\n+\t}\n+\treturn values(%pages);\n+}\n+\n+sub run_git {\n+\topen(my $git, \"-|:encoding(UTF-8)\", \"git \" . $_[0]);\n+\tmy $res = do { local $/; <$git> };\n+\tclose($git);\n+\n+\treturn $res;\n+}\n+\n+\n+sub get_last_local_revision {\n+\t# Get note regarding last mediawiki revision\n+\tmy $note = run_git(\"notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null\");\n+\tmy @note_info = split(/ /, $note);\n+\n+\tmy $lastrevision_number;\n+\tif (!(defined($note_info[0]) && $note_info[0] eq \"mediawiki_revision:\")) {\n+\t\tprint STDERR \"No previous mediawiki revision found\";\n+\t\t$lastrevision_number = 0;\n+\t} else {\n+\t\t# Notes are formatted : mediawiki_revision: #number\n+\t\t$lastrevision_number = $note_info[1];\n+\t\tchomp($lastrevision_number);\n+\t\tprint STDERR \"Last local mediawiki revision found is $lastrevision_number\";\n+\t}\n+\treturn $lastrevision_number;\n+}\n+\n+sub get_last_remote_revision {\n+\tmw_connect_maybe();\n+\n+\tmy @pages = get_mw_pages();\n+\n+\tmy $max_rev_num = 0;\n+\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'ids',\n+\t\t\tpageids => $id,\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});\n+\n+\t\t$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);\n+\t}\n+\n+\tprint STDERR \"Last remote revision found is $max_rev_num.\\n\";\n+\treturn $max_rev_num;\n+}\n+\n+# Clean content before sending it to MediaWiki\n+sub mediawiki_clean {\n+\tmy $string = shift;\n+\tmy $page_created = shift;\n+\t# Mediawiki does not allow blank space at the end of a page and ends with a single \\n.\n+\t# This function right trims a string and adds a \\n at the end to follow this rule\n+\t$string =~ s/\\s+$//;\n+\tif ($string eq \"\" && $page_created) {\n+\t\t# Creating empty pages is forbidden.\n+\t\t$string = EMPTY_CONTENT;\n+\t}\n+\treturn $string.\"\\n\";\n+}\n+\n+# Filter applied on MediaWiki data before adding them to Git\n+sub mediawiki_smudge {\n+\tmy $string = shift;\n+\tif ($string eq EMPTY_CONTENT) {\n+\t\t$string = \"\";\n+\t}\n+\t# This \\n is important. This is due to mediawiki's way to handle end of files.\n+\treturn $string.\"\\n\";\n+}\n+\n+sub mediawiki_clean_filename {\n+\tmy $filename = shift;\n+\t$filename =~ s/@{[SLASH_REPLACEMENT]}/\\//g;\n+\t# [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.\n+\t# Do a variant of URL-encoding, i.e. looks like URL-encoding,\n+\t# but with _ added to prevent MediaWiki from thinking this is\n+\t# an actual special character.\n+\t$filename =~ s/[\\[\\]\\{\\}\\|]/sprintf(\"_%%_%x\", ord($&))/ge;\n+\t# If we use the uri escape before\n+\t# we should unescape here, before anything\n+\n+\treturn $filename;\n+}\n+\n+sub mediawiki_smudge_filename {\n+\tmy $filename = shift;\n+\t$filename =~ s/\\//@{[SLASH_REPLACEMENT]}/g;\n+\t$filename =~ s/ /_/g;\n+\t# Decode forbidden characters encoded in mediawiki_clean_filename\n+\t$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf(\"%c\", hex($1))/ge;\n+\treturn $filename;\n+}\n+\n+sub literal_data {\n+\tmy ($content) = @_;\n+\tprint STDOUT \"data \", bytes::length($content), \"\\n\", $content;\n+}\n+\n+sub mw_capabilities {\n+\t# Revisions are imported to the private namespace\n+\t# refs/mediawiki/$remotename/ by the helper and fetched into\n+\t# refs/remotes/$remotename later by fetch.\n+\tprint STDOUT \"refspec refs/heads/*:refs/mediawiki/$remotename/*\\n\";\n+\tprint STDOUT \"import\\n\";\n+\tprint STDOUT \"list\\n\";\n+\tprint STDOUT \"push\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_list {\n+\t# MediaWiki do not have branches, we consider one branch arbitrarily\n+\t# called master, and HEAD pointing to it.\n+\tprint STDOUT \"? refs/heads/master\\n\";\n+\tprint STDOUT \"\\@refs/heads/master HEAD\\n\";\n+\tprint STDOUT \"\\n\";\n+}\n+\n+sub mw_option {\n+\tprint STDERR \"remote-helper command 'option $_[0]' not yet implemented\\n\";\n+\tprint STDOUT \"unsupported\\n\";\n+}\n+\n+sub fetch_mw_revisions_for_page {\n+\tmy $page = shift;\n+\tmy $id = shift;\n+\tmy $fetch_from = shift;\n+\tmy @page_revs = ();\n+\tmy $query = {\n+\t\taction => 'query',\n+\t\tprop => 'revisions',\n+\t\trvprop => 'ids',\n+\t\trvdir => 'newer',\n+\t\trvstartid => $fetch_from,\n+\t\trvlimit => 500,\n+\t\tpageids => $id,\n+\t};\n+\n+\tmy $revnum = 0;\n+\t# Get 500 revisions at a time due to the mediawiki api limit\n+\twhile (1) {\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\t# Parse each of those 500 revisions\n+\t\tforeach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {\n+\t\t\tmy $page_rev_ids;\n+\t\t\t$page_rev_ids->{pageid} = $page->{pageid};\n+\t\t\t$page_rev_ids->{revid} = $revision->{revid};\n+\t\t\tpush(@page_revs, $page_rev_ids);\n+\t\t\t$revnum++;\n+\t\t}\n+\t\tlast unless $result->{'query-continue'};\n+\t\t$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};\n+\t}\n+\tif ($shallow_import && @page_revs) {\n+\t\tprint STDERR \" Found 1 revision (shallow import).\\n\";\n+\t\t@page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);\n+\t\treturn $page_revs[0];\n+\t}\n+\tprint STDERR \" Found \", $revnum, \" revision(s).\\n\";\n+\treturn @page_revs;\n+}\n+\n+sub fetch_mw_revisions {\n+\tmy $pages = shift; my @pages = @{$pages};\n+\tmy $fetch_from = shift;\n+\n+\tmy @revisions = ();\n+\tmy $n = 1;\n+\tforeach my $page (@pages) {\n+\t\tmy $id = $page->{pageid};\n+\n+\t\tprint STDERR \"page $n/\", scalar(@pages), \": \". $page->{title} .\"\\n\";\n+\t\t$n++;\n+\t\tmy @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);\n+\t\t@revisions = (@page_revs, @revisions);\n+\t}\n+\n+\treturn ($n, @revisions);\n+}\n+\n+sub import_file_revision {\n+\tmy $commit = shift;\n+\tmy %commit = %{$commit};\n+\tmy $full_import = shift;\n+\tmy $n = shift;\n+\n+\tmy $title = $commit{title};\n+\tmy $comment = $commit{comment};\n+\tmy $content = $commit{content};\n+\tmy $author = $commit{author};\n+\tmy $date = $commit{date};\n+\n+\tprint STDOUT \"commit refs/mediawiki/$remotename/master\\n\";\n+\tprint STDOUT \"mark :$n\\n\";\n+\tprint STDOUT \"committer $author <$author\\@$wiki_name> \", $date->epoch, \" +0000\\n\";\n+\tliteral_data($comment);\n+\n+\t# If it's not a clone, we need to know where to start from\n+\tif (!$full_import && $n == 1) {\n+\t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n+\t}\n+\tif ($content ne DELETED_CONTENT) {\n+\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tliteral_data($content);\n+\t\tprint STDOUT \"\\n\\n\";\n+\t} else {\n+\t\tprint STDOUT \"D $title.mw\\n\";\n+\t}\n+\n+\t# mediawiki revision number in the git note\n+\tif ($full_import && $n == 1) {\n+\t\tprint STDOUT \"reset refs/notes/$remotename/mediawiki\\n\";\n+\t}\n+\tprint STDOUT \"commit refs/notes/$remotename/mediawiki\\n\";\n+\tprint STDOUT \"committer $author <$author\\@$wiki_name> \", $date->epoch, \" +0000\\n\";\n+\tliteral_data(\"Note added by git-mediawiki during import\");\n+\tif (!$full_import && $n == 1) {\n+\t\tprint STDOUT \"from refs/notes/$remotename/mediawiki^0\\n\";\n+\t}\n+\tprint STDOUT \"N inline :$n\\n\";\n+\tliteral_data(\"mediawiki_revision: \" . $commit{mw_revision});\n+\tprint STDOUT \"\\n\\n\";\n+}\n+\n+# parse a sequence of\n+# \n+# \n+# \\n\n+# (like batch sequence of import and sequence of push statements)\n+sub get_more_refs {\n+\tmy $cmd = shift;\n+\tmy @refs;\n+\twhile (1) {\n+\t\tmy $line = ;\n+\t\tif ($line =~ m/^$cmd (.*)$/) {\n+\t\t\tpush(@refs, $1);\n+\t\t} elsif ($line eq \"\\n\") {\n+\t\t\treturn @refs;\n+\t\t} else {\n+\t\t\tdie(\"Invalid command in a '$cmd' batch: \". $_);\n+\t\t}\n+\t}\n+}\n+\n+sub mw_import {\n+\t# multiple import commands can follow each other.\n+\tmy @refs = (shift, get_more_refs(\"import\"));\n+\tforeach my $ref (@refs) {\n+\t\tmw_import_ref($ref);\n+\t}\n+\tprint STDOUT \"done\\n\";\n+}\n+\n+sub mw_import_ref {\n+\tmy $ref = shift;\n+\t# The remote helper will call \"import HEAD\" and\n+\t# \"import refs/heads/master\".\n+\t# Since HEAD is a symbolic ref to master (by convention,\n+\t# followed by the output of the command \"list\" that we gave),\n+\t# we don't need to do anything in this case.\n+\tif ($ref eq \"HEAD\") {\n+\t\treturn;\n+\t}\n+\n+\tmw_connect_maybe();\n+\n+\tmy @pages = get_mw_pages();\n+\n+\tprint STDERR \"Searching revisions...\\n\";\n+\tmy $last_local = get_last_local_revision();\n+\tmy $fetch_from = $last_local + 1;\n+\tif ($fetch_from == 1) {\n+\t\tprint STDERR \", fetching from beginning.\\n\";\n+\t} else {\n+\t\tprint STDERR \", fetching from here.\\n\";\n+\t}\n+\tmy ($n, @revisions) = fetch_mw_revisions(\\@pages, $fetch_from);\n+\n+\t# Creation of the fast-import stream\n+\tprint STDERR \"Fetching & writing export data...\\n\";\n+\n+\t$n = 0;\n+\tmy $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined\n+\n+\tforeach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {\n+\t\t# fetch the content of the pages\n+\t\tmy $query = {\n+\t\t\taction => 'query',\n+\t\t\tprop => 'revisions',\n+\t\t\trvprop => 'content|timestamp|comment|user|ids',\n+\t\t\trevids => $pagerevid->{revid},\n+\t\t};\n+\n+\t\tmy $result = $mediawiki->api($query);\n+\n+\t\tmy $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});\n+\n+\t\t$n++;\n+\n+\t\tmy %commit;\n+\t\t$commit{author} = $rev->{user} || 'Anonymous';\n+\t\t$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';\n+\t\t$commit{title} = mediawiki_smudge_filename(\n+\t\t\t$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}\n+\t\t );\n+\t\t$commit{mw_revision} = $pagerevid->{revid};\n+\t\t$commit{content} = mediawiki_smudge($rev->{'*'});\n+\n+\t\tif (!defined($rev->{timestamp})) {\n+\t\t\t$last_timestamp++;\n+\t\t} else {\n+\t\t\t$last_timestamp = $rev->{timestamp};\n+\t\t}\n+\t\t$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);\n+\n+\t\tprint STDERR \"$n/\", scalar(@revisions), \": Revision #$pagerevid->{revid} of $commit{title}\\n\";\n+\n+\t\timport_file_revision(\\%commit, ($fetch_from == 1), $n);\n+\t}\n+\n+\tif ($fetch_from == 1 && $n == 0) {\n+\t\tprint STDERR \"You appear to have cloned an empty MediaWiki.\\n\";\n+\t\t# Something has to be done remote-helper side. If nothing is done, an error is\n+\t\t# thrown saying that HEAD is refering to unknown object 0000000000000000000\n+\t\t# and the clone fails.\n+\t}\n+}\n+\n+sub error_non_fast_forward {\n+\t# Native git-push would show this after the summary.\n+\t# We can't ask it to display it cleanly, so print it\n+\t# ourselves before.\n+\tprint STDERR \"To prevent you from losing history, non-fast-forward updates were rejected\\n\";\n+\tprint STDERR \"Merge the remote changes (e.g. 'git pull') before pushing again. See the\\n\";\n+\tprint STDERR \"'Note about fast-forwards' section of 'git push --help' for details.\\n\";\n+\n+\tprint STDOUT \"error $_[0] \\\"non-fast-forward\\\"\\n\";\n+\treturn 0;\n+}\n+\n+sub mw_push_file {\n+\tmy $diff_info = shift;\n+\t# $diff_info contains a string in this format:\n+\t# 100644 100644 \n+\tmy @diff_info_split = split(/[ \\t]/, $diff_info);\n+\n+\t# Filename, including .mw extension\n+\tmy $complete_file_name = shift;\n+\t# Commit message\n+\tmy $summary = shift;\n+\n+\tmy $new_sha1 = $diff_info_split[3];\n+\tmy $old_sha1 = $diff_info_split[2];\n+\tmy $page_created = ($old_sha1 eq NULL_SHA1);\n+\tmy $page_deleted = ($new_sha1 eq NULL_SHA1);\n+\t$complete_file_name = mediawiki_clean_filename($complete_file_name);\n+\n+\tif (substr($complete_file_name,-3) eq \".mw\") {\n+\t\tmy $title = substr($complete_file_name,0,-3);\n+\n+\t\tmy $file_content;\n+\t\tif ($page_deleted) {\n+\t\t\t# Deleting a page usually requires\n+\t\t\t# special priviledges. A common\n+\t\t\t# convention is to replace the page\n+\t\t\t# with this content instead:\n+\t\t\t$file_content = DELETED_CONTENT;\n+\t\t} else {\n+\t\t\t$file_content = run_git(\"cat-file blob $new_sha1\");\n+\t\t}\n+\n+\t\tmw_connect_maybe();\n+\n+\t\tmy $result = $mediawiki->edit( {\n+\t\t\taction => 'edit',\n+\t\t\tsummary => $summary,\n+\t\t\ttitle => $title,\n+\t\t\ttext => mediawiki_clean($file_content, $page_created),\n+\t\t\t\t }, {\n+\t\t\t\t\t skip_encoding => 1 # Helps with names with accentuated characters\n+\t\t\t\t }) || die 'Fatal: Error ' .\n+\t\t\t\t $mediawiki->{error}->{code} .\n+\t\t\t\t ' from mediwiki: ' . $mediawiki->{error}->{details};\n+\t\tprint STDERR \"Pushed file : $new_sha1 - $title\\n\";\n+\t} else {\n+\t\tprint STDERR \"$complete_file_name not a mediawiki file (Not pushable on this version of git-remote-mediawiki).\\n\"\n+\t}\n+}\n+\n+sub mw_push {\n+\t# multiple push statements can follow each other\n+\tmy @refsspecs = (shift, get_more_refs(\"push\"));\n+\tmy %status;\n+\tmy $pushed;\n+\tfor my $refspec (@refsspecs) {\n+\t\tmy ($force, $local, $remote) = $refspec =~ /^(\\+)?([^:]*):([^:]*)$/\n+\t\t or die(\"Invalid refspec for push. Expected : or +:\");\n+\t\tif ($force) {\n+\t\t\tprint STDERR \"Warning: forced push not allowed on a MediaWiki.\\n\";\n+\t\t}\n+\t\tif ($local eq \"\") {\n+\t\t\tprint STDERR \"Cannot delete remote branch on a MediaWiki\\n\";\n+\t\t\tprint STDOUT \"error $remote cannot delete\\n\";\n+\t\t\tnext;\n+\t\t}\n+\t\tif ($remote ne \"refs/heads/master\") {\n+\t\t\tprint STDERR \"Only push to the branch 'master' is supported on a MediaWiki\\n\";\n+\t\t\tprint STDOUT \"error $remote only master allowed\\n\";\n+\t\t\tnext;\n+\t\t}\n+\t\tif (mw_push_revision($local, $remote)) {\n+\t\t\t$pushed = 1;\n+\t\t}\n+\t}\n+\n+\t# Notify Git that the push is done\n+\tprint STDOUT \"\\n\";\n+\n+\tif ($pushed) {\n+\t\tprint STDERR \"Just pushed some revisions to MediaWiki.\\n\";\n+\t\tprint STDERR \"The pushed revisions now have to be re-imported, and your current branch\\n\";\n+\t\tprint STDERR \"needs to be updated with these re-imported commits. You can do this with\\n\";\n+\t\tprint STDERR \"\\n\";\n+\t\tprint STDERR \" git pull --rebase\\n\";\n+\t\tprint STDERR \"\\n\";\n+\t}\n+}\n+\n+sub mw_push_revision {\n+\tmy $local = shift;\n+\tmy $remote = shift; # actually, this has to be \"refs/heads/master\" at this point.\n+\tmy $last_local_revid = get_last_local_revision();\n+\tprint STDERR \".\\n\"; # Finish sentence started by get_last_local_revision()\n+\tmy $last_remote_revid = get_last_remote_revision();\n+\n+\t# Get sha1 of commit pointed by local HEAD\n+\tmy $HEAD_sha1 = run_git(\"rev-parse $local 2>/dev/null\"); chomp($HEAD_sha1);\n+\t# Get sha1 of commit pointed by remotes/$remotename/master\n+\tmy $remoteorigin_sha1 = run_git(\"rev-parse refs/remotes/$remotename/master 2>/dev/null\");\n+\tchomp($remoteorigin_sha1);\n+\n+\tif ($last_local_revid > 0 &&\n+\t $last_local_revid < $last_remote_revid) {\n+\t\treturn error_non_fast_forward($remote);\n+\t}\n+\n+\tif ($HEAD_sha1 eq $remoteorigin_sha1) {\n+\t\t# nothing to push\n+\t\treturn 0;\n+\t}\n+\n+\t# Get every commit in between HEAD and refs/remotes/origin/master,\n+\t# including HEAD and refs/remotes/origin/master\n+\tmy @commit_pairs = ();\n+\tif ($last_local_revid > 0) {\n+\t\tmy $parsed_sha1 = $remoteorigin_sha1;\n+\t\t# Find a path from last MediaWiki commit to pushed commit\n+\t\twhile ($parsed_sha1 ne $HEAD_sha1) {\n+\t\t\tmy @commit_info = grep(/^$parsed_sha1/, split(/\\n/, run_git(\"rev-list --children $local\")));\n+\t\t\tif (!@commit_info) {\n+\t\t\t\treturn error_non_fast_forward($remote);\n+\t\t\t}\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $commit_info[0]);\n+\t\t\t# $commit_info_split[1] is the sha1 of the commit to export\n+\t\t\t# $commit_info_split[0] is the sha1 of its direct child\n+\t\t\tpush(@commit_pairs, \\@commit_info_split);\n+\t\t\t$parsed_sha1 = $commit_info_split[1];\n+\t\t}\n+\t} else {\n+\t\t# No remote mediawiki revision. Export the whole\n+\t\t# history (linearized with --first-parent)\n+\t\tprint STDERR \"Warning: no common ancestor, pushing complete history\\n\";\n+\t\tmy $history = run_git(\"rev-list --first-parent --children $local\");\n+\t\tmy @history = split('\\n', $history);\n+\t\t@history = @history[1..$#history];\n+\t\tforeach my $line (reverse @history) {\n+\t\t\tmy @commit_info_split = split(/ |\\n/, $line);\n+\t\t\tpush(@commit_pairs, \\@commit_info_split);\n+\t\t}\n+\t}\n+\n+\tforeach my $commit_info_split (@commit_pairs) {\n+\t\tmy $sha1_child = @{$commit_info_split}[0];\n+\t\tmy $sha1_commit = @{$commit_info_split}[1];\n+\t\tmy $diff_infos = run_git(\"diff-tree -r --raw -z $sha1_child $sha1_commit\");\n+\t\t# TODO: we could detect rename, and encode them with a #redirect on the wiki.\n+\t\t# TODO: for now, it's just a delete+add\n+\t\tmy @diff_info_list = split(/\\0/, $diff_infos);\n+\t\t# Keep the first line of the commit message as mediawiki comment for the revision\n+\t\tmy $commit_msg = (split(/\\n/, run_git(\"show --pretty=format:\\\"%s\\\" $sha1_commit\")))[0];\n+\t\tchomp($commit_msg);\n+\t\t# Push every blob\n+\t\twhile (@diff_info_list) {\n+\t\t\t# git diff-tree -z gives an output like\n+\t\t\t# \\0\\0\n+\t\t\t# \\0\\0\n+\t\t\t# and we've split on \\0.\n+\t\t\tmy $info = shift(@diff_info_list);\n+\t\t\tmy $file = shift(@diff_info_list);\n+\t\t\tmw_push_file($info, $file, $commit_msg);\n+\t\t}\n+\t}\n+\n+\tprint STDOUT \"ok $remote\\n\";\n+\treturn 1;\n+}\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt\nnew file mode 100644\nindex 0000000..4d211f5\n--- /dev/null\n+++ b/contrib/mw-to-git/git-remote-mediawiki.txt\n@@ -0,0 +1,7 @@\n+Git-Mediawiki is a project which aims the creation of a gate\n+between git and mediawiki, allowing git users to push and pull\n+objects from mediawiki just as one would do with a classic git\n+repository thanks to remote-helpers.\n+\n+For more information, visit the wiki at\n+https://github.com/Bibzball/Git-Mediawiki/wiki\n-- \n1.7.7.rc0.75.g56f27\n"},{"id":"174682","messageId":"1314896096-17544-3-git-send-email-Matthieu.Moy@imag.fr","threadId":"28226","inReplyTo":"1314896096-17544-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 2/2] git-remote-mediawiki: allow push to set MediaWiki metadata","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2011-09-01T16:54:56Z","receivedAt":"2011-09-01T16:54:56Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Push can not set the commit note \"mediawiki_revision:\" and update the\nremote reference. This avoids having to \"git pull --rebase\" after each\npush, and is probably more natural. Make it the default, but let it be\nconfigurable with mediawiki.dumbPush or remote..dumbPush.\n\nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 35 ++++++++++++++++++++++++++++++-\n 1 files changed, 33 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex d4d1198..0ba88de 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -84,6 +84,27 @@ my $shallow_import = run_git(\"config --get --bool remote.\". $remotename .\".shall\n chomp($shallow_import);\n $shallow_import = ($shallow_import eq \"true\");\n \n+# Dumb push: don't update notes and mediawiki ref to reflect the last push.\n+#\n+# Configurable with mediawiki.dumbPush, or per-remote with\n+# remote..dumbPush.\n+#\n+# This means the user will have to re-import the just-pushed\n+# revisions. On the other hand, this means that the Git revisions\n+# corresponding to MediaWiki revisions are all imported from the wiki,\n+# regardless of whether they were initially created in Git or from the\n+# web interface, hence all users will get the same history (i.e. if\n+# the push from Git to MediaWiki loses some information, everybody\n+# will get the history with information lost). If the import is\n+# deterministic, this means everybody gets the same sha1 for each\n+# MediaWiki revision.\n+my $dumb_push = run_git(\"config --get --bool remote.$remotename.dumbPush\");\n+unless ($dumb_push) {\n+\t$dumb_push = run_git(\"config --get --bool mediawiki.dumbPush\");\n+}\n+chomp($dumb_push);\n+$dumb_push = ($dumb_push eq \"true\");\n+\n my $wiki_name = $url;\n $wiki_name =~ s/[^\\/]*:\\/\\///;\n \n@@ -598,6 +619,9 @@ sub mw_push_file {\n \tmy $complete_file_name = shift;\n \t# Commit message\n \tmy $summary = shift;\n+\t# MediaWiki revision number. Keep the previous one by default,\n+\t# in case there's no edit to perform.\n+\tmy $newrevid = shift;\n \n \tmy $new_sha1 = $diff_info_split[3];\n \tmy $old_sha1 = $diff_info_split[2];\n@@ -631,10 +655,12 @@ sub mw_push_file {\n \t\t\t\t }) || die 'Fatal: Error ' .\n \t\t\t\t $mediawiki->{error}->{code} .\n \t\t\t\t ' from mediwiki: ' . $mediawiki->{error}->{details};\n+\t\t$newrevid = $result->{edit}->{newrevid};\n \t\tprint STDERR \"Pushed file : $new_sha1 - $title\\n\";\n \t} else {\n \t\tprint STDERR \"$complete_file_name not a mediawiki file (Not pushable on this version of git-remote-mediawiki).\\n\"\n \t}\n+\treturn $newrevid;\n }\n \n sub mw_push {\n@@ -666,7 +692,7 @@ sub mw_push {\n \t# Notify Git that the push is done\n \tprint STDOUT \"\\n\";\n \n-\tif ($pushed) {\n+\tif ($pushed && $dumb_push) {\n \t\tprint STDERR \"Just pushed some revisions to MediaWiki.\\n\";\n \t\tprint STDERR \"The pushed revisions now have to be re-imported, and your current branch\\n\";\n \t\tprint STDERR \"needs to be updated with these re-imported commits. You can do this with\\n\";\n@@ -682,6 +708,7 @@ sub mw_push_revision {\n \tmy $last_local_revid = get_last_local_revision();\n \tprint STDERR \".\\n\"; # Finish sentence started by get_last_local_revision()\n \tmy $last_remote_revid = get_last_remote_revision();\n+\tmy $mw_revision = $last_remote_revid;\n \n \t# Get sha1 of commit pointed by local HEAD\n \tmy $HEAD_sha1 = run_git(\"rev-parse $local 2>/dev/null\"); chomp($HEAD_sha1);\n@@ -747,7 +774,11 @@ sub mw_push_revision {\n \t\t\t# and we've split on \\0.\n \t\t\tmy $info = shift(@diff_info_list);\n \t\t\tmy $file = shift(@diff_info_list);\n-\t\t\tmw_push_file($info, $file, $commit_msg);\n+\t\t\t$mw_revision = mw_push_file($info, $file, $commit_msg, $mw_revision);\n+\t\t}\n+\t\tunless ($dumb_push) {\n+\t\t\trun_git(\"notes --ref=$remotename/mediawiki add -m \\\"mediawiki_revision: $mw_revision\\\" $sha1_commit\");\n+\t\t\trun_git(\"update-ref -m \\\"Git-MediaWiki push\\\" refs/mediawiki/$remotename/master $sha1_commit $sha1_child\");\n \t\t}\n \t}\n \n-- \n1.7.7.rc0.75.g56f27\n"},{"id":"174685","messageId":"CAGdFq_iqNxeRhQUfY0BVBNWqDq01i9MJ_2STC0MjiF0p=rrBGg@mail.gmail.com","threadId":"28226","inReplyTo":"1314895778-17482-2-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH 2/2 v2] (short) documentation for the testgit remote helper","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-09-01T16:59:19Z","receivedAt":"2011-09-01T16:59:19Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Thu, Sep 1, 2011 at 18:49, Matthieu Moy wrote:\n> While it's not a command meant to be used by actual users (hence, not\n> mentionned in git(1)), this command is a very precious help for\n> remote-helpers authors.\n>\n> The best place for such technical doc is the source code, but users may\n> not find it without a link in a manpage.\n>\n> Signed-off-by: Matthieu Moy \n\nAcked-by: Sverre Rabbelier \n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174687","messageId":"CAGdFq_gc6scerXf6PgbDE-89nmaTo8z49L2fOJ2E_m42_7ywfQ@mail.gmail.com","threadId":"28226","inReplyTo":"1314895778-17482-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH 1/2 v2] Documentation/git-remote-helpers: explain how import works with multiple refs","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-09-01T16:59:31Z","receivedAt":"2011-09-01T16:59:31Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Thu, Sep 1, 2011 at 18:49, Matthieu Moy wrote:\n> This is important for two reasons:\n>\n> * when two \"import\" lines follow each other, only one \"done\" command\n>  should be issued in the fast-import stream, not one per \"import\".\n>\n> * The blank line terminating an import command should not be confused\n>  with the one terminating the sequence of commands.\n>\n> While we're there, illustrate the corresponding explanation for push\n> batches with an example.\n>\n> Signed-off-by: Matthieu Moy \n\nAcked-by: Sverre Rabbelier \n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"174718","messageId":"20110901231734.GA21945@elie","threadId":"28226","inReplyTo":"1314809222-30528-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH] git-remote-helpers.txt: explain how import works with multiple refs","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-09-01T23:17:34Z","receivedAt":"2011-09-01T23:17:34Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Matthieu,\n\nMatthieu Moy wrote:\n\n> --- a/Documentation/git-remote-helpers.txt\n> +++ b/Documentation/git-remote-helpers.txt\n> @@ -241,7 +241,22 @@ Supported if the helper has the \"fetch\" capability.\n> 'push' +:::\n> \tPushes the given local commit or branch to the\n> \tremote branch described by . A batch sequence of\n> -\tone or more push commands is terminated with a blank line.\n> +\tone or more 'push' commands is terminated with a blank line\n> +\t(if there is only one reference to push, a single 'push' command\n> +\tis followed by a blank line). For example, the following would\n> +\tbe two batches of 'push', the first asking the remote-helper\n> +\tto push the local ref 'master' to the remote ref 'master' and\n> +\tthe local 'HEAD' to the remote 'branch', and the second\n> +\tasking to push ref 'foo' to ref 'bar' (forced update requested\n> +\tby the '+').\n> ++\n> +------------\n> +push refs/heads/master:refs/heads/master\n> +push HEAD:refs/heads/branch\n> +\\n\n> +push +refs/heads/foo:refs/heads/bar\n> +\\n\n> +------------\n\nProbably examples like this could go in a later EXAMPLES section.\n\nAt first I was worried about this not actually working, thinking\n\"push\" might have traditionally had the newline-ends-command-stream\nsemantics that \"connect\" has. But the push codepath does not set the\nno_disconnect_req flag, so I was worrying in vain. :) The stream\npassed to the helper ends with two newlines when git pushes.\n\n> +\n> Zero or more protocol options may be entered after the last 'push'\n> command, before the batch's terminating blank line.\n> @@ -266,6 +281,11 @@ Supported if the helper has the \"push\" capability.\n> Especially useful for interoperability with a foreign versioning\n> system.\n> +\n> +Just like 'push', a batch sequence of one or more 'import' is\n> +terminated with a blank line. For each batch of 'import', the remote\n> +helper should produce a fast-import stream terminated by a 'done'\n> +command.\n> ++\n> Supported if the helper has the \"import\" capability.\n\nThis documents v1.7.7-rc0~61^2~3 (transport-helper: change import\nsemantics, 2011-07-16). Such a documentation fix was desperately\nneeded; thank you!\n\nFor what it's worth, with or without a change to prevent the example\nfrom interrupting the flow of description,\nAcked-by: Jonathan Nieder \n"},{"id":"174720","messageId":"20110901234446.GA25817@elie","threadId":"28226","inReplyTo":"vpqobz53ig3.fsf@bauges.imag.fr","subject":"Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-09-01T23:44:46Z","receivedAt":"2011-09-01T23:44:46Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Matthieu,\n\nMatthieu Moy wrote:\n> Jonathan Nieder writes:\n\n>> Here's an old attempt to make the documentation a little easier to read,\n>> and hopefully also to add to.\n>\n> Thanks, that helps. I wish we had all this earlier ;-).\n\nGlad you like it.\n\n>> +Git sends the remote helper a list of commands on standard input, one\n>> +per line. The first command is always the 'capabilities' command, \n>\n> Do we want to set this in stone?\n\nYes, I think we should set it in stone. Helpers can use that as a\nsanity check to know they are actually being used as a remote\nhelper. And specifying it makes the protocol more concrete.\n\n> Wouldn't a Git implementation calling\n> \"option\" before \"capabilities\" be correct?\n\nAt that point git doesn't know if the helper implements the \"option\"\ncapability.\n\n>> +Capabilities\n>> +~~~~~~~~~~~~\n>\n> (perhaps name the section \"Overview of Capabilities\"?)\n\nYes, good idea.\n\n[...]\n> Since this \"Capabilities\" section is meant to be an overview, I'd\n> shorten this to\n>\n> +'refspec' ::\n> +\tThis modifies the 'import' capability, allowing the produced\n> +\tfast-import stream to modify refs in a private namespace\n> +\tinstead of writing to refs/heads or refs/remotes directly.\n>\n> and drop the detailed explanation here.\n\nAnother good idea.\n\n>> +Capabilities for Fetching\n>> +~~~~~~~~~~~~~~~~~~~~~~~~~\n> [...]\n>> +'refspec' ::\n>> +\tThis modifies the 'import' capability.\n>\n> Since this would be the \"detailed explanation\" part, this is the one\n> readers will read more carefully, so I'd put the recommandation right\n> here:\n>\n> +\tIt is recommended that all importers providing the 'import'\n> +\tcapability use this.\n\nYet another. :)\n\n> and of course, keep this:\n>\n>> ++\n>> +A helper advertising\n>> +`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`\n>> +in its capabilities is saying that, when it handles\n>> +`import refs/heads/topic`, the stream it outputs will update the\n>> +`refs/svn/origin/branches/topic` ref.\n>> ++\n>> +This capability can be advertised multiple times. The first\n>> +applicable refspec takes precedence. The left-hand of refspecs\n>> +advertised with this capability must cover all refs reported by\n>> +the list command. If no 'refspec' capability is advertised,\n>> +there is an implied `refspec {asterisk}:{asterisk}`.\n>> +\n\nThanks much! Will try to find time to resend later today.\n"},{"id":"174784","messageId":"vpqty8tc3rt.fsf@bauges.imag.fr","threadId":"28226","inReplyTo":"20110901231734.GA21945@elie","subject":"Re: [PATCH] git-remote-helpers.txt: explain how import works with multiple refs","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-09-03T10:35:18Z","receivedAt":"2011-09-03T10:35:18Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jonathan Nieder writes:\n\n>> +\tis followed by a blank line). For example, the following would\n>> +\tbe two batches of 'push', the first asking the remote-helper\n>> +\tto push the local ref 'master' to the remote ref 'master' and\n>> +\tthe local 'HEAD' to the remote 'branch', and the second\n>> +\tasking to push ref 'foo' to ref 'bar' (forced update requested\n>> +\tby the '+').\n>> ++\n>> +------------\n>> +push refs/heads/master:refs/heads/master\n>> +push HEAD:refs/heads/branch\n>> +\\n\n>> +push +refs/heads/foo:refs/heads/bar\n>> +\\n\n>> +------------\n>\n> Probably examples like this could go in a later EXAMPLES section.\n\nI think having a few examples within the explanations helps the reader\nvisualize what commands look like, and understand better further\nexplanations.\n\nPlus, the example explains quickly the + which isn't documented\notherwise (but people reading this probably already know what a refspec\nis and what the + normally means in this context so it's probably\nsufficient to explain it as a side remark in an example)\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"}]}