threads / patch / 47064

patchremote-mediawiki: limit filenames to legal

Subject: [PATCH] remote-mediawiki: limit filenames to legal

## tl;dr

6 messages between Oct 29, 2017 and Oct 30, 2017. Diffs are folded; open one to read it.

replies: 5people: 2as markdown or json

Antoine Beaupré· Oct 29, 2017, 16:37 UTC · lore

mediawiki pages can have names longer than NAME_MAX (generally 255) characters, which will fail on checkout. we simply strip out extra characters, which may mean one page's content will overwrite another (the last editing winning).

ideally, we would do a more clever system to find unique names, but
that would be more difficult and error prone for a situation that
should rarely happen in the first place.
---
 contrib/mw-to-git/git-remote-mediawiki.perl | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Show changes to contrib/mw-to-git/git-remote-mediawiki.perl +2 −1

diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl
index e7f857c1a..58870d197 100755
--- a/contrib/mw-to-git/git-remote-mediawiki.perl
+++ b/contrib/mw-to-git/git-remote-mediawiki.perl
@@ -18,6 +18,7 @@ use Git::Mediawiki qw(clean_filename smudge_filename connect_maybe
 					EMPTY HTTP_CODE_OK);
 use DateTime::Format::ISO8601;
 use warnings;
+use POSIX;
 
 # By default, use UTF-8 to communicate with Git and the user
 binmode STDERR, ':encoding(UTF-8)';
@@ -703,7 +704,7 @@ sub import_file_revision {
 		%mediafile = %{$mediafile};
 	}
 
-	my $title = $commit{title};
+	my $title = substr($commit{title}, 0, NAME_MAX);
 	my $comment = $commit{comment};
 	my $content = $commit{content};
 	my $author = $commit{author};

-- 
2.11.0

Antoine Beaupré· Oct 29, 2017, 18:10 UTC · re: Antoine Beaupré · lore

[PATCH v2] remote-mediawiki: limit filenames to legal

ideally, we would do a more clever system to find unique names, but that would be more difficult and error prone for a situation that should rarely happen in the first place.

Signed-off-by: Antoine Beaupré <anarcat@debian.org>
---
 contrib/mw-to-git/Git/Mediawiki.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Show changes to contrib/mw-to-git/Git/Mediawiki.pm +2 −1

diff --git a/contrib/mw-to-git/Git/Mediawiki.pm b/contrib/mw-to-git/Git/Mediawiki.pm
index d13c4dfa7..c9f22680a 100644
--- a/contrib/mw-to-git/Git/Mediawiki.pm
+++ b/contrib/mw-to-git/Git/Mediawiki.pm
@@ -2,6 +2,7 @@ package Git::Mediawiki;
 
 use 5.008;
 use strict;
+use POSIX;
 use Git;
 
 BEGIN {
@@ -52,7 +53,7 @@ sub smudge_filename {
 	$filename =~ s/ /_/g;
 	# Decode forbidden characters encoded in clean_filename
 	$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf('%c', hex($1))/ge;
-	return $filename;
+	return substr($filename, 0, NAME_MAX-3);
 }
 
 sub connect_maybe {

-- 
2.11.0

Matthieu Moy· Oct 30, 2017, 10:34 UTC · re: Antoine Beaupré · lore

Re: [PATCH v2] remote-mediawiki: limit filenames to legal

Antoine Beaupré <anarcat@debian.org> writes:

Show 6 quoted lines

> @@ -52,7 +53,7 @@ sub smudge_filename {
>  	$filename =~ s/ /_/g;
>  	# Decode forbidden characters encoded in clean_filename
>  	$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf('%c', hex($1))/ge;
> -	return $filename;
> +	return substr($filename, 0, NAME_MAX-3);

There's a request to allow a configurable extension (.mediawiki would help importing in some wikis, see https://github.com/Git-Mediawiki/Git-Mediawiki/issues/42). You should at least make this stg like length(".mw") so that the next search&replace for ".mw" finds this.

Also, note that your solution works for using Git-Mediawiki in a read-only way, but if you start modifying and pushing such files, you'll get into trouble. It probably makes sense to issue a warnign in such case.

Regards,

-- 
Matthieu Moy
https://matthieu-moy.fr/

Antoine Beaupré· Oct 30, 2017, 12:31 UTC · re: Matthieu Moy · lore

Re: [PATCH v2] remote-mediawiki: limit filenames to legal

On 2017-10-30 11:34:11, Matthieu Moy wrote:

Show 14 quoted lines

> Antoine Beaupré <anarcat@debian.org> writes:
>
>> @@ -52,7 +53,7 @@ sub smudge_filename {
>>  	$filename =~ s/ /_/g;
>>  	# Decode forbidden characters encoded in clean_filename
>>  	$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf('%c', hex($1))/ge;
>> -	return $filename;
>> +	return substr($filename, 0, NAME_MAX-3);
>
> There's a request to allow a configurable extension (.mediawiki would
> help importing in some wikis, see
> https://github.com/Git-Mediawiki/Git-Mediawiki/issues/42). You should at
> least make this stg like length(".mw") so that the next search&replace
> for ".mw" finds this.

I believe I did that in v3.

> Also, note that your solution works for using Git-Mediawiki in a
> read-only way, but if you start modifying and pushing such files, you'll
> get into trouble. It probably makes sense to issue a warnign in such
> case.

True. I didn't consider that, but then again the patch is not a regression: you couldn't have pushed those repos in the first place anyways...

A.

-- 
The history of any one part of the earth, like the life of a soldier,
consists of long periods of boredom and short periods of terror.
                       - British geologist Derek V. Ager

Antoine Beaupré· Oct 29, 2017, 18:15 UTC · re: Antoine Beaupré · lore

sorry for the noise here, but the original patch didn't fix the length in the right place. v2 fixed it in the library properly, but i forgot to also include the length of the suffix. this should be good to go...

Antoine Beaupré· Oct 29, 2017, 18:15 UTC · re: Antoine Beaupré · lore

[PATCH v3] remote-mediawiki: limit filenames to legal

ideally, we would do a more clever system to find unique names, but that would be more difficult and error prone for a situation that should rarely happen in the first place.

Signed-off-by: Antoine Beaupré <anarcat@debian.org>
---
 contrib/mw-to-git/Git/Mediawiki.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Show changes to contrib/mw-to-git/Git/Mediawiki.pm +2 −1

diff --git a/contrib/mw-to-git/Git/Mediawiki.pm b/contrib/mw-to-git/Git/Mediawiki.pm
index d13c4dfa7..917d9e2d3 100644
--- a/contrib/mw-to-git/Git/Mediawiki.pm
+++ b/contrib/mw-to-git/Git/Mediawiki.pm
@@ -2,6 +2,7 @@ package Git::Mediawiki;
 
 use 5.008;
 use strict;
+use POSIX;
 use Git;
 
 BEGIN {
@@ -52,7 +53,7 @@ sub smudge_filename {
 	$filename =~ s/ /_/g;
 	# Decode forbidden characters encoded in clean_filename
 	$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf('%c', hex($1))/ge;
-	return $filename;
+	return substr($filename, 0, NAME_MAX-length('.mw'));
 }
 
 sub connect_maybe {

-- 
2.11.0