threads / patch / 31033

patch, 8 partsgit-remote-mediawiki: actually send empty comment when they're empty

Subject: [PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

## tl;dr

33 messages between Jul 16, 2012 and Jul 23, 2012. Diffs are folded; open one to read it.

replies: 32people: 4as markdown or json

Matthieu Moy· Jul 16, 2012, 12:00 UTC · lore

[PATCH 0/8] git-remote-mediawiki: fixes, optimizations, and progress report

After the recent improvements to git-remote-mediawiki, I did a bit of real-life testing, by importing https://ensiwiki.ensimag.fr/ and https://git.wiki.kernel.org/. It turned out we had a few blocking bugs and a grossly unoptimized algorithm on push. This patch serie makes git-remote-mediawiki usable on these wikis (although it's still a bit slow).

My initial patch serie also contained a patch "git-remote-mediawiki: replace TODO-list in comment by appropriate link" that replaced the TODO-list in the comments of the file with just

 # Gateway between Git and MediaWiki.
 # Documentation & bugtracker: https://github.com/moy/Git-Mediawiki/

but I'm holding it for now as there seems to be interest in changing the homepage, so the link may not be appropriate.

Matthieu Moy (8):
  git-remote-mediawiki: don't split namespaces with spaces
  git-remote-mediawiki: actually send empty comment when they're empty
  git-remote-mediawiki: make mediafiles export optional
  git-remote-mediawiki: get rid of O(N^2) loop
  git-remote-mediawiki: use --force when adding notes
  git-remote-mediawiki: show progress information when listing pages
  git-remote-mediawiki: show progress information when getting last
    remote revision
  git-remote-mediawiki: properly deal with invalid remote revisions
 contrib/mw-to-git/git-remote-mediawiki | 80 +++++++++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 16 deletions(-)
-- 
1.7.11.1.30.g7e1baf9.dirty
Matthieu Moy· Jul 16, 2012, 12:00 UTC · re: Matthieu Moy · lore

[PATCH 1/8] git-remote-mediawiki: don't split namespaces with spaces

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +1 −1
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index accd70a..a6ad8cf 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1275,7 +1275,7 @@ sub get_mw_namespace_id {
 		# Look at configuration file, if the record for that namespace is
 		# already cached. Namespaces are stored in form:
 		# "Name_of_namespace:Id_namespace", ex.: "File:6".
-		my @temp = split(/[ \n]/, run_git("config --get-all remote."
+		my @temp = split(/[\n]/, run_git("config --get-all remote."
 						. $remotename .".namespaceCache"));
 		chomp(@temp);
 		foreach my $ns (@temp) {
-- 
1.7.11.1.30.g7e1baf9.dirty
Matthieu Moy· Jul 16, 2012, 12:00 UTC · re: Matthieu Moy · lore
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +8 −1
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index a6ad8cf..a2da52f 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
 # used to reflect file creation or deletion in diff.
 use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
 
+# Used on Git's side to reflect empty edit messages on the wiki
+use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
+
 my $remotename = $ARGV[0];
 my $url = $ARGV[1];
 
@@ -935,7 +938,7 @@ sub mw_import_revids {
 
 		my %commit;
 		$commit{author} = $rev->{user} || 'Anonymous';
-		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+		$commit{comment} = $rev->{comment} || EMPTY_MESSAGE;
 		$commit{title} = mediawiki_smudge_filename($page_title);
 		$commit{mw_revision} = $rev->{revid};
 		$commit{content} = mediawiki_smudge($rev->{'*'});
@@ -1050,6 +1053,10 @@ sub mw_push_file {
 	my $oldrevid = shift;
 	my $newrevid;
 
+	if ($summary eq EMPTY_MESSAGE) {
+		$summary = '';
+	}
+
 	my $new_sha1 = $diff_info_split[3];
 	my $old_sha1 = $diff_info_split[2];
 	my $page_created = ($old_sha1 eq NULL_SHA1);
-- 
1.7.11.1.30.g7e1baf9.dirty
Junio C Hamano· Jul 16, 2012, 18:13 UTC · re: Matthieu Moy · lore

Re: [PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

Matthieu Moy <Matthieu.Moy@imag.fr> writes:
Show 16 quoted lines
> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
> ---
>  contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
> index a6ad8cf..a2da52f 100755
> --- a/contrib/mw-to-git/git-remote-mediawiki
> +++ b/contrib/mw-to-git/git-remote-mediawiki
> @@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
>  # used to reflect file creation or deletion in diff.
>  use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
>  
> +# Used on Git's side to reflect empty edit messages on the wiki
> +use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
> +

Is there a reason why this sentinel value cannot be a real empty string?

Not that I am complaining, as I find it highly unlikely for a message stored in mw to be exactly "*Empty MediaWiki Message*", but just wondering.

Show 23 quoted lines
>  my $remotename = $ARGV[0];
>  my $url = $ARGV[1];
>  
> @@ -935,7 +938,7 @@ sub mw_import_revids {
>  
>  		my %commit;
>  		$commit{author} = $rev->{user} || 'Anonymous';
> -		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
> +		$commit{comment} = $rev->{comment} || EMPTY_MESSAGE;
>  		$commit{title} = mediawiki_smudge_filename($page_title);
>  		$commit{mw_revision} = $rev->{revid};
>  		$commit{content} = mediawiki_smudge($rev->{'*'});
> @@ -1050,6 +1053,10 @@ sub mw_push_file {
>  	my $oldrevid = shift;
>  	my $newrevid;
>  
> +	if ($summary eq EMPTY_MESSAGE) {
> +		$summary = '';
> +	}
> +
>  	my $new_sha1 = $diff_info_split[3];
>  	my $old_sha1 = $diff_info_split[2];
>  	my $page_created = ($old_sha1 eq NULL_SHA1);
Matthieu Moy· Jul 16, 2012, 19:06 UTC · re: Junio C Hamano · lore

Re: [PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

Junio C Hamano <gitster@pobox.com> writes:
Show 21 quoted lines
> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>
>> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
>> ---
>>  contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
>> index a6ad8cf..a2da52f 100755
>> --- a/contrib/mw-to-git/git-remote-mediawiki
>> +++ b/contrib/mw-to-git/git-remote-mediawiki
>> @@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
>>  # used to reflect file creation or deletion in diff.
>>  use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
>>  
>> +# Used on Git's side to reflect empty edit messages on the wiki
>> +use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
>> +
>
> Is there a reason why this sentinel value cannot be a real empty
> string?

That would mean having an empty commit message on the Git side, which is against the use (although possible with fast-import). The import turns empty mediawiki messages (which are very common in my experience) into this string to provide a nicer-looking history.

It's not like EMPTY_CONTENT, which is explicitely forbidden on the MediaWiki side, hence really cannot be the empty string.

Ideally, EMPTY_MESSAGE could be configurable, and the empty string could be an acceptable value to make this conversion optional.

(note that the string was already there, my patch only makes it a constant to be able to use it in two distinct places).

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
Junio C Hamano· Jul 16, 2012, 19:17 UTC · re: Matthieu Moy · lore

Re: [PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:
Show 26 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
>
>> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>>
>>> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
>>> ---
>>>  contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
>>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
>>> index a6ad8cf..a2da52f 100755
>>> --- a/contrib/mw-to-git/git-remote-mediawiki
>>> +++ b/contrib/mw-to-git/git-remote-mediawiki
>>> @@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
>>>  # used to reflect file creation or deletion in diff.
>>>  use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
>>>  
>>> +# Used on Git's side to reflect empty edit messages on the wiki
>>> +use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
>>> +
>>
>> Is there a reason why this sentinel value cannot be a real empty
>> string?
>
> That would mean having an empty commit message on the Git side, which is
> against the use (although possible with fast-import).
It also is possible with "commit --allow-empty-message", no?
> (note that the string was already there, my patch only makes it a
> constant to be able to use it in two distinct places).

It was clear from the patch text without looking beyond context, but after all I thought this was meant to be a clean-up series, and I simply wondered why mapping empty to some magic string was not considered to be something to be cleaned up. That's all.

I obviously do not care that deeply either way.
Matthieu Moy· Jul 16, 2012, 12:00 UTC · re: Matthieu Moy · lore

[PATCH 3/8] git-remote-mediawiki: make mediafiles export optional

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +22 −2
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index a2da52f..8e46e4e 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -66,11 +66,16 @@ chomp(@tracked_pages);
 my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
 chomp(@tracked_categories);
 
-# Import media files too.
+# Import media files on pull
 my $import_media = run_git("config --get --bool remote.". $remotename .".mediaimport");
 chomp($import_media);
 $import_media = ($import_media eq "true");
 
+# Export media files on push
+my $export_media = run_git("config --get --bool remote.". $remotename .".mediaexport");
+chomp($export_media);
+$export_media = !($export_media eq "false");
+
 my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
 # Note: mwPassword is discourraged. Use the credential system instead.
 my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
@@ -1068,6 +1073,11 @@ sub mw_push_file {
 		$extension = "";
 	}
 	if ($extension eq "mw") {
+		my $ns = get_mw_namespace_id_for_page($complete_file_name);
+		if ($ns && $ns == get_mw_namespace_id("File") && (!$export_media)) {
+			print STDERR "Ignoring media file related page: $complete_file_name\n";
+			return ($oldrevid, "ok");
+		}
 		my $file_content;
 		if ($page_deleted) {
 			# Deleting a page usually requires
@@ -1107,10 +1117,12 @@ sub mw_push_file {
 		}
 		$newrevid = $result->{edit}->{newrevid};
 		print STDERR "Pushed file: $new_sha1 - $title\n";
-	} else {
+	} elsif ($export_media) {
 		$newrevid = mw_upload_file($complete_file_name, $new_sha1,
 					   $extension, $page_deleted,
 					   $summary);
+	} else {
+		print STDERR "Ignoring media file $title\n";
 	}
 	$newrevid = ($newrevid or $oldrevid);
 	return ($newrevid, "ok");
@@ -1328,3 +1340,11 @@ sub get_mw_namespace_id {
 		die "No such namespace $name on MediaWiki.";
 	}
 }
+
+sub get_mw_namespace_id_for_page {
+	if (my ($namespace) = $_[0] =~ /^([^:]*):/) {
+		return get_mw_namespace_id($namespace);
+	} else {
+		return;
+	}
+}
-- 
1.7.11.1.30.g7e1baf9.dirty
Junio C Hamano· Jul 16, 2012, 18:15 UTC · re: Matthieu Moy · lore

Re: [PATCH 3/8] git-remote-mediawiki: make mediafiles export optional

Matthieu Moy <Matthieu.Moy@imag.fr> writes:
> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>

It would have been nicer to hint why people may want to omit mediafiles from their export under what condition somewhere in the documentation or at least in the proposed commit log message.

Matthieu Moy· Jul 16, 2012, 19:40 UTC · re: Junio C Hamano · lore

Re: [PATCH 3/8] git-remote-mediawiki: make mediafiles export optional

Junio C Hamano <gitster@pobox.com> writes:
Show 7 quoted lines
> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>
>> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
>
> It would have been nicer to hint why people may want to omit
> mediafiles from their export under what condition somewhere in the
> documentation or at least in the proposed commit log message.

The real use-case for which I implemented is a bit particular (described below), but I'll resend with a more general explanation.

For the curious: I have a wiki with ~2Gb of mediafiles, and I want to secure it without particular access to the server. I import the wiki on a desktop machine with a large disk, and re-export it to another webserver with a small disk quota. Importing mediafiles is good because it gives me a backup, but exporting them is not an option as my quota on the webserver is too small.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
Matthieu Moy· Jul 16, 2012, 12:00 UTC · re: Matthieu Moy · lore

[PATCH 4/8] git-remote-mediawiki: get rid of O(N^2) loop

The algorithm to find a path from the local revision to the remote one was calling "git rev-list" and parsing its output N times. Run rev-list only once, and fill a hashtable with the result to optimize the body of the loop.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +18 −7
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 8e46e4e..f9c0cc6 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1196,16 +1196,27 @@ sub mw_push_revision {
 	if ($last_local_revid > 0) {
 		my $parsed_sha1 = $remoteorigin_sha1;
 		# Find a path from last MediaWiki commit to pushed commit
+		print STDERR "Computing path from local to remote ...\n";
+		my @local_ancestry = split(/\n/, run_git("rev-list --boundary --children $local ^$parsed_sha1"));
+		my %local_ancestry;
+		foreach my $line (@local_ancestry) {
+			if (my ($parent, $child) = $line =~ m/^-?([a-f0-9]+) ([a-f0-9]+)/) {
+				$local_ancestry{$parent} = $child;
+				if ($parent eq $parsed_sha1 || $child eq $parsed_sha1) {
+					print STDERR "$parent -> $child\n";
+				}
+			} elsif (!$line =~ m/^([a-f0-9]+)/) {
+				die "Unexpected output from git rev-list: $line";
+			}
+		}
 		while ($parsed_sha1 ne $HEAD_sha1) {
-			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));
-			if (!@commit_info) {
+			my $child = $local_ancestry{$parsed_sha1};
+			if (!$child) {
+				printf STDERR "Cannot find a path in history from remote commit to last commit\n";
 				return error_non_fast_forward($remote);
 			}
-			my @commit_info_split = split(/ |\n/, $commit_info[0]);
-			# $commit_info_split[1] is the sha1 of the commit to export
-			# $commit_info_split[0] is the sha1 of its direct child
-			push(@commit_pairs, \@commit_info_split);
-			$parsed_sha1 = $commit_info_split[1];
+			push(@commit_pairs, [$parsed_sha1, $child]);
+			$parsed_sha1 = $child;
 		}
 	} else {
 		# No remote mediawiki revision. Export the whole
-- 
1.7.11.1.30.g7e1baf9.dirty
Junio C Hamano· Jul 16, 2012, 18:19 UTC · re: Matthieu Moy · lore

Re: [PATCH 4/8] git-remote-mediawiki: get rid of O(N^2) loop

Matthieu Moy <Matthieu.Moy@imag.fr> writes:
> The algorithm to find a path from the local revision to the remote one
> was calling "git rev-list" and parsing its output N times. Run rev-list
> only once, and fill a hashtable with the result to optimize the body of
> the loop.

Good thinking. I wonder if it would further reduce the overhead if you stop using --children and do this using --parents instead, as you will be reading the parsed_sha1..local range either way yourself anyway.

Show 44 quoted lines
> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
> ---
>  contrib/mw-to-git/git-remote-mediawiki | 25 ++++++++++++++++++-------
>  1 file changed, 18 insertions(+), 7 deletions(-)
>
> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
> index 8e46e4e..f9c0cc6 100755
> --- a/contrib/mw-to-git/git-remote-mediawiki
> +++ b/contrib/mw-to-git/git-remote-mediawiki
> @@ -1196,16 +1196,27 @@ sub mw_push_revision {
>  	if ($last_local_revid > 0) {
>  		my $parsed_sha1 = $remoteorigin_sha1;
>  		# Find a path from last MediaWiki commit to pushed commit
> +		print STDERR "Computing path from local to remote ...\n";
> +		my @local_ancestry = split(/\n/, run_git("rev-list --boundary --children $local ^$parsed_sha1"));
> +		my %local_ancestry;
> +		foreach my $line (@local_ancestry) {
> +			if (my ($parent, $child) = $line =~ m/^-?([a-f0-9]+) ([a-f0-9]+)/) {
> +				$local_ancestry{$parent} = $child;
> +				if ($parent eq $parsed_sha1 || $child eq $parsed_sha1) {
> +					print STDERR "$parent -> $child\n";
> +				}
> +			} elsif (!$line =~ m/^([a-f0-9]+)/) {
> +				die "Unexpected output from git rev-list: $line";
> +			}
> +		}
>  		while ($parsed_sha1 ne $HEAD_sha1) {
> -			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));
> -			if (!@commit_info) {
> +			my $child = $local_ancestry{$parsed_sha1};
> +			if (!$child) {
> +				printf STDERR "Cannot find a path in history from remote commit to last commit\n";
>  				return error_non_fast_forward($remote);
>  			}
> -			my @commit_info_split = split(/ |\n/, $commit_info[0]);
> -			# $commit_info_split[1] is the sha1 of the commit to export
> -			# $commit_info_split[0] is the sha1 of its direct child
> -			push(@commit_pairs, \@commit_info_split);
> -			$parsed_sha1 = $commit_info_split[1];
> +			push(@commit_pairs, [$parsed_sha1, $child]);
> +			$parsed_sha1 = $child;
>  		}
>  	} else {
>  		# No remote mediawiki revision. Export the whole
Matthieu Moy· Jul 16, 2012, 19:31 UTC · re: Junio C Hamano · lore

Re: [PATCH 4/8] git-remote-mediawiki: get rid of O(N^2) loop

Junio C Hamano <gitster@pobox.com> writes:
Show 11 quoted lines
> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>
>> The algorithm to find a path from the local revision to the remote one
>> was calling "git rev-list" and parsing its output N times. Run rev-list
>> only once, and fill a hashtable with the result to optimize the body of
>> the loop.
>
> Good thinking.  I wonder if it would further reduce the overhead if
> you stop using --children and do this using --parents instead, as
> you will be reading the parsed_sha1..local range either way yourself
> anyway.

It is possible, yes. I'll resend a version with --parents, but this probably doesn't change the performance much: what we really need is for Git to prune dead-ends in the subgraph, to make sure we find a path without having to backtrack (i.e. we need parent rewriting history simplification), so Git has to do something a bit clever anyway.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
Matthieu Moy· Jul 16, 2012, 12:00 UTC · re: Matthieu Moy · lore

[PATCH 5/8] git-remote-mediawiki: use --force when adding notes

When notes are created to record a push, it normally doesn't exist yet. However, when a push is interrupted and then restarted, it may happen that a commit already has notes attached, and we want to reflect the newly created remote revision, hence use 'git notes add -f' to override the existing one

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +1 −1
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index f9c0cc6..f56f9b0 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1264,7 +1264,7 @@ sub mw_push_revision {
 			}
 		}
 		unless ($dumb_push) {
-			run_git("notes --ref=$remotename/mediawiki add -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
+			run_git("notes --ref=$remotename/mediawiki add -f -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
 			run_git("update-ref -m \"Git-MediaWiki push\" refs/mediawiki/$remotename/master $sha1_commit $sha1_child");
 		}
 	}
-- 
1.7.11.1.30.g7e1baf9.dirty
Matthieu Moy· Jul 16, 2012, 12:00 UTC · re: Matthieu Moy · lore

[PATCH 6/8] git-remote-mediawiki: show progress information when listing pages

Initial phases of push and pull with git-remote-mediawiki can be long on a large wiki. Let the user know what's going on.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 3 +++
 1 file changed, 3 insertions(+)
Show changes to contrib/mw-to-git/git-remote-mediawiki +3 −0
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index f56f9b0..755c9d0 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -371,6 +371,8 @@ sub get_mw_first_pages {
 sub get_mw_pages {
 	mw_connect_maybe();
 
+	print STDERR "Listing pages on remote wiki...\n";
+
 	my %pages; # hash on page titles to avoid duplicates
 	my $user_defined;
 	if (@tracked_pages) {
@@ -394,6 +396,7 @@ sub get_mw_pages {
 			get_all_mediafiles(\%pages);
 		}
 	}
+	print STDERR (scalar keys %pages) . " pages found.\n";
 	return %pages;
 }
 
-- 
1.7.11.1.30.g7e1baf9.dirty
Matthieu Moy· Jul 16, 2012, 12:00 UTC · re: Matthieu Moy · lore

[PATCH 7/8] git-remote-mediawiki: show progress information when getting last remote revision

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 ++
 1 file changed, 2 insertions(+)
Show changes to contrib/mw-to-git/git-remote-mediawiki +2 −0
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 755c9d0..cf467ac 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -579,6 +579,8 @@ sub get_last_remote_revision {
 
 	my $max_rev_num = 0;
 
+	print STDERR "Getting last revision id on tracked pages...\n";
+
 	foreach my $page (@pages) {
 		my $id = $page->{pageid};
 
-- 
1.7.11.1.30.g7e1baf9.dirty
Matthieu Moy· Jul 16, 2012, 12:00 UTC · re: Matthieu Moy · lore

[PATCH 8/8] git-remote-mediawiki: properly deal with invalid remote revisions

Some wiki, including https://git.wiki.kernel.org/ have invalid revision numbers (i.e. the actual revision numbers are non-contiguous). Don't die when encountering one.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +9 −4
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index cf467ac..80a07cc 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -910,6 +910,10 @@ sub mw_import_revids {
 	my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
 
 	foreach my $pagerevid (@$revision_ids) {
+	        # Count page even if we skip it, since we display
+		# $n/$total and $total includes skipped pages.
+		$n++;
+
 		# fetch the content of the pages
 		my $query = {
 			action => 'query',
@@ -924,6 +928,11 @@ sub mw_import_revids {
 			die "Failed to retrieve modified page for revision $pagerevid";
 		}
 
+		if (defined($result->{query}->{badrevids}->{$pagerevid})) {
+			# The revision id does not exist on the remote wiki.
+			next;
+		}
+
 		if (!defined($result->{query}->{pages})) {
 			die "Invalid revision $pagerevid.";
 		}
@@ -932,10 +941,6 @@ sub mw_import_revids {
 		my $result_page = $result_pages[0];
 		my $rev = $result_pages[0]->{revisions}->[0];
 
-	        # Count page even if we skip it, since we display
-		# $n/$total and $total includes skipped pages.
-		$n++;
-
 		my $page_title = $result_page->{title};
 
 		if (!exists($pages->{$page_title})) {
-- 
1.7.11.1.30.g7e1baf9.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore

[PATCH 0/8 v2] git-remote-mediawiki: fixes, optimizations, and progress report

Changes since v1: more explanations in commit message for "make mediafiles export optional", and use --parents instead of --children in "get rid of O(N^2) loop".

Matthieu Moy (8):
  git-remote-mediawiki: don't split namespaces with spaces
  git-remote-mediawiki: actually send empty comment when they're empty
  git-remote-mediawiki: make mediafiles export optional
  git-remote-mediawiki: get rid of O(N^2) loop
  git-remote-mediawiki: use --force when adding notes
  git-remote-mediawiki: show progress information when listing pages
  git-remote-mediawiki: show progress information when getting last
    remote revision
  git-remote-mediawiki: properly deal with invalid remote revisions
 contrib/mw-to-git/git-remote-mediawiki | 79 +++++++++++++++++++++++++++-------
 1 file changed, 63 insertions(+), 16 deletions(-)
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore

[PATCH 1/8] git-remote-mediawiki: don't split namespaces with spaces

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +1 −1
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index accd70a..a6ad8cf 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1275,7 +1275,7 @@ sub get_mw_namespace_id {
 		# Look at configuration file, if the record for that namespace is
 		# already cached. Namespaces are stored in form:
 		# "Name_of_namespace:Id_namespace", ex.: "File:6".
-		my @temp = split(/[ \n]/, run_git("config --get-all remote."
+		my @temp = split(/[\n]/, run_git("config --get-all remote."
 						. $remotename .".namespaceCache"));
 		chomp(@temp);
 		foreach my $ns (@temp) {
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +8 −1
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index a6ad8cf..a2da52f 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
 # used to reflect file creation or deletion in diff.
 use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
 
+# Used on Git's side to reflect empty edit messages on the wiki
+use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
+
 my $remotename = $ARGV[0];
 my $url = $ARGV[1];
 
@@ -935,7 +938,7 @@ sub mw_import_revids {
 
 		my %commit;
 		$commit{author} = $rev->{user} || 'Anonymous';
-		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+		$commit{comment} = $rev->{comment} || EMPTY_MESSAGE;
 		$commit{title} = mediawiki_smudge_filename($page_title);
 		$commit{mw_revision} = $rev->{revid};
 		$commit{content} = mediawiki_smudge($rev->{'*'});
@@ -1050,6 +1053,10 @@ sub mw_push_file {
 	my $oldrevid = shift;
 	my $newrevid;
 
+	if ($summary eq EMPTY_MESSAGE) {
+		$summary = '';
+	}
+
 	my $new_sha1 = $diff_info_split[3];
 	my $old_sha1 = $diff_info_split[2];
 	my $page_created = ($old_sha1 eq NULL_SHA1);
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore

[PATCH 3/8] git-remote-mediawiki: make mediafiles export optional

It is possible to use git-remote-mediawiki on a tree with both .mw files and other files. Before git-remote-mediawiki learnt how to export mediafiles, such mixed trees allowed the user to maintain both the wiki and other files for the same project in the same repository. With the newly added support for exporting mediafiles, pushing such mixed trees would upload unrelated files as mediafiles, which may not be desired.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +22 −2
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index a2da52f..8e46e4e 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -66,11 +66,16 @@ chomp(@tracked_pages);
 my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
 chomp(@tracked_categories);
 
-# Import media files too.
+# Import media files on pull
 my $import_media = run_git("config --get --bool remote.". $remotename .".mediaimport");
 chomp($import_media);
 $import_media = ($import_media eq "true");
 
+# Export media files on push
+my $export_media = run_git("config --get --bool remote.". $remotename .".mediaexport");
+chomp($export_media);
+$export_media = !($export_media eq "false");
+
 my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
 # Note: mwPassword is discourraged. Use the credential system instead.
 my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
@@ -1068,6 +1073,11 @@ sub mw_push_file {
 		$extension = "";
 	}
 	if ($extension eq "mw") {
+		my $ns = get_mw_namespace_id_for_page($complete_file_name);
+		if ($ns && $ns == get_mw_namespace_id("File") && (!$export_media)) {
+			print STDERR "Ignoring media file related page: $complete_file_name\n";
+			return ($oldrevid, "ok");
+		}
 		my $file_content;
 		if ($page_deleted) {
 			# Deleting a page usually requires
@@ -1107,10 +1117,12 @@ sub mw_push_file {
 		}
 		$newrevid = $result->{edit}->{newrevid};
 		print STDERR "Pushed file: $new_sha1 - $title\n";
-	} else {
+	} elsif ($export_media) {
 		$newrevid = mw_upload_file($complete_file_name, $new_sha1,
 					   $extension, $page_deleted,
 					   $summary);
+	} else {
+		print STDERR "Ignoring media file $title\n";
 	}
 	$newrevid = ($newrevid or $oldrevid);
 	return ($newrevid, "ok");
@@ -1328,3 +1340,11 @@ sub get_mw_namespace_id {
 		die "No such namespace $name on MediaWiki.";
 	}
 }
+
+sub get_mw_namespace_id_for_page {
+	if (my ($namespace) = $_[0] =~ /^([^:]*):/) {
+		return get_mw_namespace_id($namespace);
+	} else {
+		return;
+	}
+}
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore

[PATCH 4/8] git-remote-mediawiki: get rid of O(N^2) loop

The algorithm to find a path from the local revision to the remote one was calling "git rev-list" and parsing its output N times. Run rev-list only once, and fill a hashtable with the result to optimize the body of the loop.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +17 −7
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 8e46e4e..fb1e9e0 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1196,16 +1196,26 @@ sub mw_push_revision {
 	if ($last_local_revid > 0) {
 		my $parsed_sha1 = $remoteorigin_sha1;
 		# Find a path from last MediaWiki commit to pushed commit
+		print STDERR "Computing path from local to remote ...\n";
+		my @local_ancestry = split(/\n/, run_git("rev-list --boundary --parents $local ^$parsed_sha1"));
+		my %local_ancestry;
+		foreach my $line (@local_ancestry) {
+			if (my ($child, $parents) = $line =~ m/^-?([a-f0-9]+) ([a-f0-9 ]+)/) {
+				foreach my $parent (split(' ', $parents)) {
+					$local_ancestry{$parent} = $child;
+				}
+			} elsif (!$line =~ m/^([a-f0-9]+)/) {
+				die "Unexpected output from git rev-list: $line";
+			}
+		}
 		while ($parsed_sha1 ne $HEAD_sha1) {
-			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));
-			if (!@commit_info) {
+			my $child = $local_ancestry{$parsed_sha1};
+			if (!$child) {
+				printf STDERR "Cannot find a path in history from remote commit to last commit\n";
 				return error_non_fast_forward($remote);
 			}
-			my @commit_info_split = split(/ |\n/, $commit_info[0]);
-			# $commit_info_split[1] is the sha1 of the commit to export
-			# $commit_info_split[0] is the sha1 of its direct child
-			push(@commit_pairs, \@commit_info_split);
-			$parsed_sha1 = $commit_info_split[1];
+			push(@commit_pairs, [$parsed_sha1, $child]);
+			$parsed_sha1 = $child;
 		}
 	} else {
 		# No remote mediawiki revision. Export the whole
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore

[PATCH 5/8] git-remote-mediawiki: use --force when adding notes

When notes are created to record a push, it normally doesn't exist yet. However, when a push is interrupted and then restarted, it may happen that a commit already has notes attached, and we want to reflect the newly created remote revision, hence use 'git notes add -f' to override the existing one

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +1 −1
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index fb1e9e0..517a4db 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1263,7 +1263,7 @@ sub mw_push_revision {
 			}
 		}
 		unless ($dumb_push) {
-			run_git("notes --ref=$remotename/mediawiki add -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
+			run_git("notes --ref=$remotename/mediawiki add -f -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
 			run_git("update-ref -m \"Git-MediaWiki push\" refs/mediawiki/$remotename/master $sha1_commit $sha1_child");
 		}
 	}
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore

[PATCH 6/8] git-remote-mediawiki: show progress information when listing pages

Initial phases of push and pull with git-remote-mediawiki can be long on a large wiki. Let the user know what's going on.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 3 +++
 1 file changed, 3 insertions(+)
Show changes to contrib/mw-to-git/git-remote-mediawiki +3 −0
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 517a4db..729a0bc 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -371,6 +371,8 @@ sub get_mw_first_pages {
 sub get_mw_pages {
 	mw_connect_maybe();
 
+	print STDERR "Listing pages on remote wiki...\n";
+
 	my %pages; # hash on page titles to avoid duplicates
 	my $user_defined;
 	if (@tracked_pages) {
@@ -394,6 +396,7 @@ sub get_mw_pages {
 			get_all_mediafiles(\%pages);
 		}
 	}
+	print STDERR (scalar keys %pages) . " pages found.\n";
 	return %pages;
 }
 
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore

[PATCH 7/8] git-remote-mediawiki: show progress information when getting last remote revision

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 ++
 1 file changed, 2 insertions(+)
Show changes to contrib/mw-to-git/git-remote-mediawiki +2 −0
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 729a0bc..8badff6 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -579,6 +579,8 @@ sub get_last_remote_revision {
 
 	my $max_rev_num = 0;
 
+	print STDERR "Getting last revision id on tracked pages...\n";
+
 	foreach my $page (@pages) {
 		my $id = $page->{pageid};
 
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 16, 2012, 19:46 UTC · re: Matthieu Moy · lore

[PATCH 8/8] git-remote-mediawiki: properly deal with invalid remote revisions

Some wiki, including https://git.wiki.kernel.org/ have invalid revision numbers (i.e. the actual revision numbers are non-contiguous). Don't die when encountering one.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
Show changes to contrib/mw-to-git/git-remote-mediawiki +9 −4
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 8badff6..5eab96b 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -910,6 +910,10 @@ sub mw_import_revids {
 	my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
 
 	foreach my $pagerevid (@$revision_ids) {
+	        # Count page even if we skip it, since we display
+		# $n/$total and $total includes skipped pages.
+		$n++;
+
 		# fetch the content of the pages
 		my $query = {
 			action => 'query',
@@ -924,6 +928,11 @@ sub mw_import_revids {
 			die "Failed to retrieve modified page for revision $pagerevid";
 		}
 
+		if (defined($result->{query}->{badrevids}->{$pagerevid})) {
+			# The revision id does not exist on the remote wiki.
+			next;
+		}
+
 		if (!defined($result->{query}->{pages})) {
 			die "Invalid revision $pagerevid.";
 		}
@@ -932,10 +941,6 @@ sub mw_import_revids {
 		my $result_page = $result_pages[0];
 		my $rev = $result_pages[0]->{revisions}->[0];
 
-	        # Count page even if we skip it, since we display
-		# $n/$total and $total includes skipped pages.
-		$n++;
-
 		my $page_title = $result_page->{title};
 
 		if (!exists($pages->{$page_title})) {
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 17, 2012, 14:05 UTC · re: Matthieu Moy · lore

[PATCH 0/2] git-remote-mediawiki: two more fixes

These patches can be added to branch mm/mediawiki-usability in pu.

For the curious: I originally had patch 2/2 in my serie, but mistakenly discarded it during a rebase. While recovering it and testing it, I found a typo fixed by patch 1/2.

Matthieu Moy (2):
  git-remote-mediawiki: fix incorrect test usage in test
  git-remote-mediawiki: allow page names with a ':'
 contrib/mw-to-git/git-remote-mediawiki      | 49 +++++++++++++++++++----------
 contrib/mw-to-git/t/push-pull-tests.sh      |  2 +-
 contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh | 20 ++++++++++++
 3 files changed, 54 insertions(+), 17 deletions(-)
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 17, 2012, 14:05 UTC · re: Matthieu Moy · lore

[PATCH 1/2] git-remote-mediawiki: fix incorrect test usage in test

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/t/push-pull-tests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to contrib/mw-to-git/t/push-pull-tests.sh +1 −1
diff --git a/contrib/mw-to-git/t/push-pull-tests.sh b/contrib/mw-to-git/t/push-pull-tests.sh
index 6692a0f..9da2dc5 100644
--- a/contrib/mw-to-git/t/push-pull-tests.sh
+++ b/contrib/mw-to-git/t/push-pull-tests.sh
@@ -104,7 +104,7 @@ test_push_pull () {
 			git push
 		) &&
 
-		test ! wiki_page_exist Foo
+		test_must_fail wiki_page_exist Foo
 	'
 
 	test_expect_success 'Merge conflict expected and solving it' '
-- 
1.7.11.2.258.g5ff3cdf.dirty
Matthieu Moy· Jul 17, 2012, 14:06 UTC · re: Matthieu Moy · lore

[PATCH 2/2] git-remote-mediawiki: allow page names with a ':'

Traditionnally, pages named Foo:Bar are page 'Bar' in namespace 'Foo'. However, it is also possible to call a page Foo:Bar if 'Foo' is not a namespace. In this case, the actual name of the page is 'Foo:Bar', in the main namespace. Since we can't tell with only the filename, query the wiki for a namespace 'Foo' in these cases, but deal with the case where no such namespace is found.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki      | 49 +++++++++++++++++++----------
 contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh | 20 ++++++++++++
 2 files changed, 53 insertions(+), 16 deletions(-)
Show changes to 2 files +53 −16

contrib/mw-to-git/git-remote-mediawiki, contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index d6be188..47d4878 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -955,8 +955,11 @@ sub mw_import_revids {
 		# Differentiates classic pages and media files.
 		my ($namespace, $filename) = $page_title =~ /^([^:]*):(.*)$/;
 		my %mediafile;
-		if ($namespace && get_mw_namespace_id($namespace) == get_mw_namespace_id("File")) {
-			%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});
+		if ($namespace) {
+			my $id = get_mw_namespace_id($namespace);
+			if ($id && $id == get_mw_namespace_id("File")) {
+				%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});
+			}
 		}
 		# If this is a revision of the media page for new version
 		# of a file do one common commit for both file and media page.
@@ -1306,7 +1309,11 @@ sub get_mw_namespace_id {
 		chomp(@temp);
 		foreach my $ns (@temp) {
 			my ($n, $id) = split(/:/, $ns);
-			$namespace_id{$n} = $id;
+			if ($id eq 'notANameSpace') {
+				$namespace_id{$n} = {is_namespace => 0};
+			} else {
+				$namespace_id{$n} = {is_namespace => 1, id => $id};
+			}
 			$cached_mw_namespace_id{$n} = 1;
 		}
 	}
@@ -1324,28 +1331,38 @@ sub get_mw_namespace_id {
 
 	        while (my ($id, $ns) = each(%{$result->{query}->{namespaces}})) {
 	                if (defined($ns->{id}) && defined($ns->{canonical})) {
-				$namespace_id{$ns->{canonical}} = $ns->{id};
+				$namespace_id{$ns->{canonical}} = {is_namespace => 1, id => $ns->{id}};
 				if ($ns->{'*'}) {
 					# alias (e.g. french Fichier: as alias for canonical File:)
-					$namespace_id{$ns->{'*'}} = $ns->{id};
+					$namespace_id{$ns->{'*'}} = {is_namespace => 1, id => $ns->{id}};
 				}
 			}
 	        }
 	}
 
-	my $id = $namespace_id{$name};
+	my $ns = $namespace_id{$name};
+	my $id;
 
-	if (defined $id) {
-		# Store explicitely requested namespaces on disk
-		if (!exists $cached_mw_namespace_id{$name}) {
-			run_git("config --add remote.". $remotename
-				.".namespaceCache \"". $name .":". $id ."\"");
-			$cached_mw_namespace_id{$name} = 1;
-		}
-		return $id;
-	} else {
-		die "No such namespace $name on MediaWiki.";
+	unless (defined $ns) {
+		print STDERR "No such namespace $name on MediaWiki.\n";
+		$ns = {is_namespace => 0};
+		$namespace_id{$name} = $ns;
+	}
+
+	if ($ns->{is_namespace}) {
+		$id = $ns->{id};
+	}
+
+	# Store "notANameSpace" as special value for inexisting namespaces
+	my $store_id = ($id || 'notANameSpace');
+
+	# Store explicitely requested namespaces on disk
+	if (!exists $cached_mw_namespace_id{$name}) {
+		run_git("config --add remote.". $remotename
+			.".namespaceCache \"". $name .":". $store_id ."\"");
+		$cached_mw_namespace_id{$name} = 1;
 	}
+	return $id;
 }
 
 sub get_mw_namespace_id_for_page {
diff --git a/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh b/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh
index 8635878..246d47d 100755
--- a/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh
+++ b/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh
@@ -169,6 +169,26 @@ test_expect_failure 'special character at the begining of file name from mw to g
 	test_path_is_file mw_dir_11/[char_2
 '
 
+test_expect_success 'Pull page with title containing ":" other than namespace separator' '
+	wiki_editpage Foo:Bar content false &&
+	(
+		cd mw_dir_11 &&
+		git pull
+	) &&
+	test_path_is_file mw_dir_11/Foo:Bar.mw
+'
+
+test_expect_success 'Push page with title containing ":" other than namespace separator' '
+	(
+		cd mw_dir_11 &&
+		echo content >NotANameSpace:Page.mw &&
+		git add NotANameSpace:Page.mw &&
+		git commit -m "add page with colon" &&
+		git push
+	) &&
+	wiki_page_exist NotANameSpace:Page
+'
+
 test_expect_success 'test of correct formating for file name from mw to git' '
 	wiki_reset &&
 	git clone mediawiki::'"$WIKI_URL"' mw_dir_12 &&
-- 
1.7.11.2.258.g5ff3cdf.dirty
Dan Johnson· Jul 20, 2012, 21:11 UTC · re: Matthieu Moy · lore

Re: [PATCH 2/2] git-remote-mediawiki: allow page names with a ':'

On Tue, Jul 17, 2012 at 10:06 AM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
Show 6 quoted lines
> Traditionnally, pages named Foo:Bar are page 'Bar' in namespace 'Foo'.
> However, it is also possible to call a page Foo:Bar if 'Foo' is not a
> namespace. In this case, the actual name of the page is 'Foo:Bar', in the
> main namespace. Since we can't tell with only the filename, query the
> wiki for a namespace 'Foo' in these cases, but deal with the case where
> no such namespace is found.

Might not be worth fixing, and it's just a typo in the commit message, but: s/Traditionnally/Traditionally/ ?

-- 
-Dan
Matthieu Moy· Jul 23, 2012, 07:42 UTC · re: Dan Johnson · lore

Re: [PATCH 2/2] git-remote-mediawiki: allow page names with a ':'

Dan Johnson <computerdruid@gmail.com> writes:
Show 11 quoted lines
> On Tue, Jul 17, 2012 at 10:06 AM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
>> Traditionnally, pages named Foo:Bar are page 'Bar' in namespace 'Foo'.
>> However, it is also possible to call a page Foo:Bar if 'Foo' is not a
>> namespace. In this case, the actual name of the page is 'Foo:Bar', in the
>> main namespace. Since we can't tell with only the filename, query the
>> wiki for a namespace 'Foo' in these cases, but deal with the case where
>> no such namespace is found.
>
> Might not be worth fixing, and it's just a typo in the commit message, but:
> s/Traditionnally/Traditionally/
> ?

Thanks, but the topic is in next already, so it's too late to fix the commit message.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

← back to recent threads