{"thread":{"id":"47063","subject":"[PATCH 4/4] remote-mediawiki: allow using (Main) as a namespace and skip special namespaces","startedAt":"2017-10-29T16:14:47Z","lastAt":"2017-11-08T02:07:38Z","messageCount":78,"participants":["Antoine Beaupré","Eric Sunshine","Kevin","Matthieu Moy","Junio C Hamano","Thomas Adam"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"331274","messageId":"20171029160857.29460-5-anarcat@debian.org","threadId":"47063","inReplyTo":"20171029160857.29460-1-anarcat@debian.org","subject":"[PATCH 4/4] remote-mediawiki: allow using (Main) as a namespace and skip special namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-29T16:08:57Z","receivedAt":"2017-10-29T16:14:47Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Reviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 31 +++++++++++++++++++----------\n 1 file changed, 21 insertions(+), 10 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 07cc74bac..ccefde4dc 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,16 +264,27 @@ sub get_mw_tracked_categories {\n \n sub get_mw_tracked_namespaces {\n my $pages = shift;\n- foreach my $local_namespace (@tracked_namespaces) {\n- my $mw_pages = $mediawiki->list( {\n- action => 'query',\n- list => 'allpages',\n- apnamespace => get_mw_namespace_id($local_namespace),\n- aplimit => 'max' } )\n- || die $mediawiki->{error}->{code} . ': '\n- . $mediawiki->{error}->{details} . \"\\n\";\n- foreach my $page (@{$mw_pages}) {\n- $pages->{$page->{title}} = $page;\n+ foreach my $local_namespace (sort @tracked_namespaces) {\n+ my ($mw_pages, $namespace_id);\n+ if ($local_namespace eq \"(Main)\") {\n+ $namespace_id = 0;\n+ } else {\n+ $namespace_id = get_mw_namespace_id($local_namespace);\n+ }\n+ if ($namespace_id >= 0) {\n+ if ($mw_pages = $mediawiki->list( {\n+ action => 'query',\n+ list => 'allpages',\n+ apnamespace => $namespace_id,\n+ aplimit => 'max' } )) {\n+ print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n+ foreach my $page (@{$mw_pages}) {\n+ $pages->{$page->{title}} = $page;\n+ }\n+ } else {\n+ warn $mediawiki->{error}->{code} . ': '\n+ . $mediawiki->{error}->{details} . \"\\n\";\n+ }\n }\n }\n return;\n-- \n2.11.0\n\n"},{"id":"331275","messageId":"20171029160857.29460-2-anarcat@debian.org","threadId":"47063","inReplyTo":"20171029160857.29460-1-anarcat@debian.org","subject":"[PATCH 1/4] remote-mediawiki: add namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-29T16:08:54Z","receivedAt":"2017-10-29T16:14:50Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Kevin \n\nthis introduces a new remote.origin.namespaces argument that is a\nspace-separated list of namespaces. the list of pages extract is then\ntaken from all the specified namespaces.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 34 +++++++++++++++++++++++++++--\n 1 file changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex e7f857c1a..1c5e39831 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -17,6 +17,7 @@ use Git;\n use Git::Mediawiki qw(clean_filename smudge_filename connect_maybe\n \t\t\t\t\tEMPTY HTTP_CODE_OK);\n use DateTime::Format::ISO8601;\n+use Scalar::Util;\n use warnings;\n \n # By default, use UTF-8 to communicate with Git and the user\n@@ -63,6 +64,10 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.categories\"));\n chomp(@tracked_categories);\n \n+# Just like @tracked_categories, but for MediaWiki namespaces.\n+my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+chomp(@tracked_namespaces);\n+\n # Import media files on pull\n my $import_media = run_git(\"config --get --bool remote.${remotename}.mediaimport\");\n chomp($import_media);\n@@ -256,6 +261,23 @@ sub get_mw_tracked_categories {\n \treturn;\n }\n \n+sub get_mw_tracked_namespaces {\n+ my $pages = shift;\n+ foreach my $local_namespace (@tracked_namespaces) {\n+ my $mw_pages = $mediawiki->list( {\n+ action => 'query',\n+ list => 'allpages',\n+ apnamespace => get_mw_namespace_id($local_namespace),\n+ aplimit => 'max' } )\n+ || die $mediawiki->{error}->{code} . ': '\n+ . $mediawiki->{error}->{details} . \"\\n\";\n+ foreach my $page (@{$mw_pages}) {\n+ $pages->{$page->{title}} = $page;\n+ }\n+ }\n+ return;\n+}\n+\n sub get_mw_all_pages {\n \tmy $pages = shift;\n \t# No user-provided list, get the list of pages from the API.\n@@ -319,6 +341,10 @@ sub get_mw_pages {\n \t\t$user_defined = 1;\n \t\tget_mw_tracked_categories(\\%pages);\n \t}\n+ if (@tracked_namespaces) {\n+ $user_defined = 1;\n+ get_mw_tracked_namespaces(\\%pages);\n+ }\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n@@ -1263,7 +1289,6 @@ my %cached_mw_namespace_id;\n sub get_mw_namespace_id {\n \t$mediawiki = connect_maybe($mediawiki, $remotename, $url);\n \tmy $name = shift;\n-\n \tif (!exists $namespace_id{$name}) {\n \t\t# Look at configuration file, if the record for that namespace is\n \t\t# already cached. Namespaces are stored in form:\n@@ -1331,7 +1356,12 @@ sub get_mw_namespace_id {\n sub get_mw_namespace_id_for_page {\n \tmy $namespace = shift;\n \tif ($namespace =~ /^([^:]*):/) {\n-\t\treturn get_mw_namespace_id($namespace);\n+\t\tmy ($ns, $id) = split(/:/, $namespace);\n+\t\tif (Scalar::Util::looks_like_number($id)) {\n+\t\t\treturn get_mw_namespace_id($ns);\n+\t\t} else{\n+\t\t\treturn\n+\t\t}\n \t} else {\n \t\treturn;\n \t}\n-- \n2.11.0\n\n"},{"id":"331276","messageId":"20171029160857.29460-4-anarcat@debian.org","threadId":"47063","inReplyTo":"20171029160857.29460-1-anarcat@debian.org","subject":"[PATCH 3/4] remote-mediawiki: show known namespace choices on failure","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-29T16:08:56Z","receivedAt":"2017-10-29T16:14:52Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"if we fail to find a requested namespace, we should tell the user\nwhich ones we know about, since we already do. this allows users to\nfeetch all namespaces by specifying a dummy namespace, failing, then\ncopying the list of namespaces in the config.\n\neventually, we should have a flag that allows fetching all namespaces\nautomatically.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex fc48846a1..07cc74bac 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -1334,7 +1334,9 @@ sub get_mw_namespace_id {\n \tmy $id;\n \n \tif (!defined $ns) {\n-\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n+\t\tmy @namespaces = sort keys %namespace_id;\n+\t\tfor (@namespaces) { s/ /_/g; }\n+\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki, known namespaces: @namespaces.\\n\";\n \t\t$ns = {is_namespace => 0};\n \t\t$namespace_id{$name} = $ns;\n \t}\n-- \n2.11.0\n\n"},{"id":"331277","messageId":"20171029160857.29460-3-anarcat@debian.org","threadId":"47063","inReplyTo":"20171029160857.29460-1-anarcat@debian.org","subject":"[PATCH 2/4] remote-mediawiki: allow fetching namespaces with spaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-29T16:08:55Z","receivedAt":"2017-10-29T16:14:54Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Ingo Ruhnke \n\nwe still want to use spaces as separators in the config, but we should\nallow the user to specify namespaces with spaces, so we use underscore\nfor this.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 1c5e39831..fc48846a1 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -66,6 +66,7 @@ chomp(@tracked_categories);\n \n # Just like @tracked_categories, but for MediaWiki namespaces.\n my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+for (@tracked_namespaces) { s/_/ /g; }\n chomp(@tracked_namespaces);\n \n # Import media files on pull\n-- \n2.11.0\n\n"},{"id":"331278","messageId":"20171029160857.29460-1-anarcat@debian.org","threadId":"47063","inReplyTo":null,"subject":"[PATCH 0/4] WIP: git-remote-media wiki namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-29T16:08:53Z","receivedAt":"2017-10-29T16:14:58Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Hi,\n\nFor a few years now, work has been happening in a [GitHub issue] to\nimprove git's support for MediaWiki sites, which are implemented in\nthe contrib/mw-to-git/ module, mostly visible in the\ngit-remote-mediawiki command.\n\n [GitHub issue]: https://github.com/Git-Mediawiki/Git-Mediawiki/issues/10\n\nThis specific patchset adds support for namespaces in\nMediaWiki. Without this, it is impossible to fetch pages outside the\n\"(Main)\" namespace (e.g. Talk pages or \"meta\"). Namespaces are heavily\nused on many wikis and this seems like an essential feature to have.\n\nI have been hesitant in pushing those patches here because I know how\nstrict the git community is regarding patchsets and I was afraid they\nwould just get shot down, especially because there are no unit tests\nfor the new functionality. Obviously, doing unit tests against a full\nMediaWiki instance isn't exactly trivial. Even though the contrib\nmodule features a test suite and a way to install MediaWiki, I haven't\nhad the chance to test this yet, so unit tests are still missing. This\nis the main reason why this is marked WIP.\n\nI have tried to follow the patch submission guide, but I believe this\nis my first Git patch, so please be gentle. Any review would be\ngreatly appreciated and I hope this can be eventually merged in. This\nwork is also available on GitHub:\n\nhttps://github.com/anarcat/git/tree/mediawiki-namespaces\n\nThanks in advance,\n\nA.\n\n"},{"id":"331282","messageId":"CAPig+cSmfJ2Uv21Q4DgJNoy6Ywj7GWPJa6qq0YL9Kar6Q74a_Q@mail.gmail.com","threadId":"47063","inReplyTo":"20171029160857.29460-2-anarcat@debian.org","subject":"Re: [PATCH 1/4] remote-mediawiki: add namespace support","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-10-29T17:24:03Z","receivedAt":"2017-10-29T17:24:10Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n> From: Kevin \n>\n> this introduces a new remote.origin.namespaces argument that is a\n\ns/this/This/\n\n> space-separated list of namespaces. the list of pages extract is then\n\ns/the/The/\n\n> taken from all the specified namespaces.\n>\n> Reviewed-by: Antoine Beaupré \n> Signed-off-by: Antoine Beaupré \n> ---\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -1331,7 +1356,12 @@ sub get_mw_namespace_id {\n> sub get_mw_namespace_id_for_page {\n> my $namespace = shift;\n> if ($namespace =~ /^([^:]*):/) {\n\nThis is not a new issue, but why capture if $1 is never referenced in\nthe code below?\n\n> - return get_mw_namespace_id($namespace);\n> + my ($ns, $id) = split(/:/, $namespace);\n> + if (Scalar::Util::looks_like_number($id)) {\n> + return get_mw_namespace_id($ns);\n\nSo, the idea is that if the input has form \"something:number\", then\nyou want to look up \"something\" as a namespace name. Anything else\n(such as \"something:foobar\") is not considered a valid page reference.\nRight?\n\n> + } else{\n\nMissing space before open brace.\n\n> + return\n\nNot required, but missing semi-colon.\n\n> + }\n> } else {\n> return;\n> }\n\nThe multiple 'return's are a bit messy. Perhaps collapse the entire\nfunction to something like this:\n\n sub get_mw_namespace_id_for_page {\n my $arg = shift;\n if ($arg =~ /^([^:]+):\\d+$/) {\n return get_mw_namespace_id($1);\n }\n return undef;\n }\n\nThen, you don't need even need Scalar::Util::looks_like_number()\n(unless, I suppose, the incoming number is expected to be something\nother than simple digits).\n\nIn fact, it may be that the intent of the original code *was* meant to\ndo exactly the same as shown in my example above, but that the person\nwho wrote it accidentally typed:\n\n return get_mw_namespace_id($namespace);\n\ninstead of the intended:\n\n return get_mw_namespace_id($1);\n\nSo, a minimal fix would be simply to change $namespace to $1.\nTightening the regex as I did in my example would be a bonus (though\nprobably ought to be a separate patch).\n"},{"id":"331283","messageId":"CAPig+cSkwD35wK7Z_qZZusKWWBC5CU6AnbF9g_g907uqpeYQsw@mail.gmail.com","threadId":"47063","inReplyTo":"20171029160857.29460-4-anarcat@debian.org","subject":"Re: [PATCH 3/4] remote-mediawiki: show known namespace choices on failure","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-10-29T17:34:31Z","receivedAt":"2017-10-29T17:34:37Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n> if we fail to find a requested namespace, we should tell the user\n\ns/if/If/\n\n> which ones we know about, since we already do. this allows users to\n\ns/this/This/\n\nNot sure what \", since we already do\" means here.\n\n> feetch all namespaces by specifying a dummy namespace, failing, then\n\ns/feetch/fetch/\n\n> copying the list of namespaces in the config.\n>\n> eventually, we should have a flag that allows fetching all namespaces\n> automatically.\n>\n> Reviewed-by: Antoine Beaupré \n> Signed-off-by: Antoine Beaupré \n> ---\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -1334,7 +1334,9 @@ sub get_mw_namespace_id {\n> my $id;\n>\n> if (!defined $ns) {\n> - print {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n> + my @namespaces = sort keys %namespace_id;\n> + for (@namespaces) { s/ /_/g; }\n> + print {*STDERR} \"No such namespace ${name} on MediaWiki, known namespaces: @namespaces.\\n\";\n\nProbably want to drop the terminating \".\" in the error message.\n\n> $ns = {is_namespace => 0};\n> $namespace_id{$name} = $ns;\n> }\n"},{"id":"331289","messageId":"87a809959p.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"CAPig+cSmfJ2Uv21Q4DgJNoy6Ywj7GWPJa6qq0YL9Kar6Q74a_Q@mail.gmail.com","subject":"Re: [PATCH 1/4] remote-mediawiki: add namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-29T18:29:38Z","receivedAt":"2017-10-29T18:39:49Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-10-29 13:24:03, Eric Sunshine wrote:\n> On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n>> From: Kevin \n>>\n>> this introduces a new remote.origin.namespaces argument that is a\n>\n> s/this/This/\n\nack.\n\n>> space-separated list of namespaces. the list of pages extract is then\n>\n> s/the/The/\n\nack.\n\n>> taken from all the specified namespaces.\n>>\n>> Reviewed-by: Antoine Beaupré \n>> Signed-off-by: Antoine Beaupré \n>> ---\n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> @@ -1331,7 +1356,12 @@ sub get_mw_namespace_id {\n>> sub get_mw_namespace_id_for_page {\n>> my $namespace = shift;\n>> if ($namespace =~ /^([^:]*):/) {\n>\n> This is not a new issue, but why capture if $1 is never referenced in\n> the code below?\n\nmeh, i dunno.\n\n>> - return get_mw_namespace_id($namespace);\n>> + my ($ns, $id) = split(/:/, $namespace);\n>> + if (Scalar::Util::looks_like_number($id)) {\n>> + return get_mw_namespace_id($ns);\n>\n> So, the idea is that if the input has form \"something:number\", then\n> you want to look up \"something\" as a namespace name. Anything else\n> (such as \"something:foobar\") is not considered a valid page reference.\n> Right?\n\nfrankly, i have no idea what's going on here.\n\n>> + } else{\n>\n> Missing space before open brace.\n\nright.\n\n>> + return\n>\n> Not required, but missing semi-colon.\n\nok.\n\n>> + }\n>> } else {\n>> return;\n>> }\n>\n> The multiple 'return's are a bit messy. Perhaps collapse the entire\n> function to something like this:\n>\n> sub get_mw_namespace_id_for_page {\n> my $arg = shift;\n> if ($arg =~ /^([^:]+):\\d+$/) {\n> return get_mw_namespace_id($1);\n> }\n> return undef;\n> }\n>\n> Then, you don't need even need Scalar::Util::looks_like_number()\n> (unless, I suppose, the incoming number is expected to be something\n> other than simple digits).\n>\n> In fact, it may be that the intent of the original code *was* meant to\n> do exactly the same as shown in my example above, but that the person\n> who wrote it accidentally typed:\n>\n> return get_mw_namespace_id($namespace);\n>\n> instead of the intended:\n>\n> return get_mw_namespace_id($1);\n>\n> So, a minimal fix would be simply to change $namespace to $1.\n> Tightening the regex as I did in my example would be a bonus (though\n> probably ought to be a separate patch).\n\nso while i'm happy to just copy-paste your code in there, that's kind of\na sensitive area of the code, as it was originally used only in the\nupload procedure, which I haven't tested at all. so i'm hesitant in just\nmerging that in as is.\n\ni don't understand why or how this even works, to be honest: page names\ndon't necessarily look like numbers, in fact, they generally don't. i\ndon't understand why the patch submitted here even touches that function\nat all, considering that the function is only used on uploads. I just\ncargo-culted it from the original issue...\n\nsigh.\n\na.\n\n-- \nC'est trop facile quand les guerres sont finies\nD'aller gueuler que c'était la dernière\nAmis bourgeois vous me faites envie\nNe voyez vous pas donc point vos cimetières?\n - Jaques Brel\n"},{"id":"331290","messageId":"877evd9561.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"CAPig+cSkwD35wK7Z_qZZusKWWBC5CU6AnbF9g_g907uqpeYQsw@mail.gmail.com","subject":"Re: [PATCH 3/4] remote-mediawiki: show known namespace choices on failure","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-29T18:31:50Z","receivedAt":"2017-10-29T18:39:52Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-10-29 13:34:31, Eric Sunshine wrote:\n> On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n>> if we fail to find a requested namespace, we should tell the user\n>\n> s/if/If/\n\nfixed.\n\n>> which ones we know about, since we already do. this allows users to\n>\n> s/this/This/\n>\n> Not sure what \", since we already do\" means here.\n\nwe already have fetched the mapping, fixed.\n\n>> feetch all namespaces by specifying a dummy namespace, failing, then\n>\n> s/feetch/fetch/\n\nfixed.\n\n>> copying the list of namespaces in the config.\n>>\n>> eventually, we should have a flag that allows fetching all namespaces\n>> automatically.\n>>\n>> Reviewed-by: Antoine Beaupré \n>> Signed-off-by: Antoine Beaupré \n>> ---\n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> @@ -1334,7 +1334,9 @@ sub get_mw_namespace_id {\n>> my $id;\n>>\n>> if (!defined $ns) {\n>> - print {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n>> + my @namespaces = sort keys %namespace_id;\n>> + for (@namespaces) { s/ /_/g; }\n>> + print {*STDERR} \"No such namespace ${name} on MediaWiki, known namespaces: @namespaces.\\n\";\n>\n> Probably want to drop the terminating \".\" in the error message.\n\nmeh... i just respected what was already there, but it's true it can be\nerror-prone when copy-pasting, so removed.\n\na.\n-- \nA ballot is like a bullet. You don't throw your ballots until you see\na target, and if that target is not within your reach, keep your\nballot in your pocket.\n - Malcom X\n"},{"id":"331293","messageId":"CAPig+cSTp1Udo6xXk5-L6MpWBdiy4sPO__NcND03-89EvRgLHQ@mail.gmail.com","threadId":"47063","inReplyTo":"20171029160857.29460-5-anarcat@debian.org","subject":"Re: [PATCH 4/4] remote-mediawiki: allow using (Main) as a namespace and skip special namespaces","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-10-29T19:49:28Z","receivedAt":"2017-10-29T19:49:35Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n> Subject: remote-mediawiki: allow using (Main) as a namespace and skip special namespaces\n\nThis patch is more difficult to review than it perhaps ought to be\nsince it is making multiple unrelated changes.\n\nIt's not clear from the description what special namespaces are and\nwhy they need to be skipped. It's also not clear why (Main) is\nspecial. Perhaps the commit message(s) could explain these issues in\nmore detail.\n\nTo simplify review and make it easier to gauge what it going on, it\nmight make sense to split this patch into at least two: one which\nskips \"special namespaces\", and one which gives special treatment to\n(Main).\n\nMore below...\n\n> Reviewed-by: Antoine Beaupré \n> Signed-off-by: Antoine Beaupré \n> ---\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -264,16 +264,27 @@ sub get_mw_tracked_categories {\n> sub get_mw_tracked_namespaces {\n> my $pages = shift;\n> - foreach my $local_namespace (@tracked_namespaces) {\n> - my $mw_pages = $mediawiki->list( {\n> - action => 'query',\n> - list => 'allpages',\n> - apnamespace => get_mw_namespace_id($local_namespace),\n> - aplimit => 'max' } )\n> - || die $mediawiki->{error}->{code} . ': '\n> - . $mediawiki->{error}->{details} . \"\\n\";\n> - foreach my $page (@{$mw_pages}) {\n> - $pages->{$page->{title}} = $page;\n> + foreach my $local_namespace (sort @tracked_namespaces) {\n> + my ($mw_pages, $namespace_id);\n> + if ($local_namespace eq \"(Main)\") {\n> + $namespace_id = 0;\n> + } else {\n> + $namespace_id = get_mw_namespace_id($local_namespace);\n> + }\n> + if ($namespace_id >= 0) {\n\nThis may be problematic since get_mw_namespace_id() may return undef\nrather than a number, in which case Perl will complain. Since the code\nskips the $mediawiki query altogether when it encounters \"(Main)\", you\ncould fix this problem and simplify the code overall by simply\nskipping the bulk of the foreach loop body instead of mucking around\nwith $namespace_id. For instance:\n\n foreach my $local_namespace (sort @tracked_namespaces) {\n next if ($local_namespace eq \"(Main)\");\n ...normal processing...\n }\n\n> + if ($mw_pages = $mediawiki->list( {\n> + action => 'query',\n> + list => 'allpages',\n> + apnamespace => $namespace_id,\n> + aplimit => 'max' } )) {\n> + print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n\nThe original code did not emit this diagnostic but the new code does\nso unconditionally. Is this just leftover debugging code or is\nintended that all users should see this information all the time?\n\n> + foreach my $page (@{$mw_pages}) {\n> + $pages->{$page->{title}} = $page;\n> + }\n> + } else {\n> + warn $mediawiki->{error}->{code} . ': '\n> + . $mediawiki->{error}->{details} . \"\\n\";\n\nI guess this is the part which \"skips special namespaces\". The\noriginal code die()'d but this merely warns. Aside from these \"special\nnamespaces\", are there genuine cases when the $mediawiki query would\nreturn an error, and which should indeed die(), or is warning\nappropriate for all $mediawiki query error cases?\n\n> + }\n> }\n> }\n> return;\n"},{"id":"331294","messageId":"CAPig+cTySXEWGxSUTxQXmHz7rR_nTbf+ALciZahJ2Ks6EmSqzg@mail.gmail.com","threadId":"47063","inReplyTo":"87a809959p.fsf@curie.anarc.at","subject":"Re: [PATCH 1/4] remote-mediawiki: add namespace support","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-10-29T20:07:21Z","receivedAt":"2017-10-29T20:07:28Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 2:29 PM, Antoine Beaupré wrote:\n> On 2017-10-29 13:24:03, Eric Sunshine wrote:\n>> On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n>> So, the idea is that if the input has form \"something:number\", then\n>> you want to look up \"something\" as a namespace name. Anything else\n>> (such as \"something:foobar\") is not considered a valid page reference.\n>> Right?\n>\n> frankly, i have no idea what's going on here.\n>\n>> The multiple 'return's are a bit messy. Perhaps collapse the entire\n>> function to something like this:\n>>\n>> sub get_mw_namespace_id_for_page {\n>> my $arg = shift;\n>> if ($arg =~ /^([^:]+):\\d+$/) {\n>> return get_mw_namespace_id($1);\n>> }\n>> return undef;\n>> }\n>>\n>> In fact, it may be that the intent of the original code *was* meant to\n>> do exactly the same as shown in my example above, but that the person\n>> who wrote it accidentally typed:\n>>\n>> return get_mw_namespace_id($namespace);\n>>\n>> instead of the intended:\n>>\n>> return get_mw_namespace_id($1);\n>>\n>> So, a minimal fix would be simply to change $namespace to $1.\n>> Tightening the regex as I did in my example would be a bonus (though\n>> probably ought to be a separate patch).\n>\n> so while i'm happy to just copy-paste your code in there, that's kind of\n> a sensitive area of the code, as it was originally used only in the\n> upload procedure, which I haven't tested at all. so i'm hesitant in just\n> merging that in as is.\n\nI don't think there's a need to copy/paste my example code. If you\ninstead make the minimal suggested fix, then the resulting code will\nbe effectively equivalent to my example (minus the tighter regex).\n\n> i don't understand why or how this even works, to be honest: page names\n> don't necessarily look like numbers, in fact, they generally don't. i\n> don't understand why the patch submitted here even touches that function\n> at all, considering that the function is only used on uploads. I just\n> cargo-culted it from the original issue...\n\nI, myself, am not familiar with or a user of Mediawiki or with the Git\nbridging, and I don't know what page names look like, but I'm pretty\nwell convinced from reading both the existing code and this patch that\nthe changes to get_mw_namespace_id_for_page() are really just a bug\nfix to that function. My interpretation is that the function really\nwas intended to strip the \":id\" portion of \"name:id\" before calling\nget_mw_namespace_id(); the fact that the original code neglects to do\nso seems just an oversight. The fact that the regex uses capturing\nparentheses implies strongly that it was indeed the intention to use\n$1 in the call to get_mw_namespace_id(). Unlike the \"fix\" in the patch\nyou posted from Kevin, which is perhaps unnecessarily complicated, the\nfix I suggested above is about a minimal as possible. That is,\nchanging:\n\n return get_mw_namespace_id($namespace);\n\nto:\n\n return get_mw_namespace_id($1);\n\nshould achieve the same result. (It could be made more robust by\ntightening the regex as in my example, but that's a separate topic,\nnot needed just to get the function to work as intended.)\n"},{"id":"331298","messageId":"6ed0d8d4-243a-7afc-bd37-a6f71004e195@ki-ai.org","threadId":"47063","inReplyTo":"CAPig+cTySXEWGxSUTxQXmHz7rR_nTbf+ALciZahJ2Ks6EmSqzg@mail.gmail.com","subject":"Re: [PATCH 1/4] remote-mediawiki: add namespace support","fromName":"Kevin","fromEmail":"kevin@ki-ai.org","sentAt":"2017-10-29T23:08:00Z","receivedAt":"2017-10-29T23:14:49Z","isPatch":true,"sender":{"key":"kevin@ki-ai.org","avatar":null},"body":"So I shared the patch some time ago (~2 years). Surprisingly its just\nnow getting attention. I guess some renewed interest in using mediawiki\nwith git. Myself, however, am no longer using mediawiki. Nor am I\ncompletely clear on what the reasons were for using some variable or\nanother a couple of years ago. So... the best of luck, sorry I couldn't\nbe more helpful.\n\nEric Sunshine:\n> On Sun, Oct 29, 2017 at 2:29 PM, Antoine Beaupré wrote:\n>> On 2017-10-29 13:24:03, Eric Sunshine wrote:\n>>> On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n>>> So, the idea is that if the input has form \"something:number\", then\n>>> you want to look up \"something\" as a namespace name. Anything else\n>>> (such as \"something:foobar\") is not considered a valid page reference.\n>>> Right?\n>>\n>> frankly, i have no idea what's going on here.\n>>\n>>> The multiple 'return's are a bit messy. Perhaps collapse the entire\n>>> function to something like this:\n>>>\n>>> sub get_mw_namespace_id_for_page {\n>>> my $arg = shift;\n>>> if ($arg =~ /^([^:]+):\\d+$/) {\n>>> return get_mw_namespace_id($1);\n>>> }\n>>> return undef;\n>>> }\n>>>\n>>> In fact, it may be that the intent of the original code *was* meant to\n>>> do exactly the same as shown in my example above, but that the person\n>>> who wrote it accidentally typed:\n>>>\n>>> return get_mw_namespace_id($namespace);\n>>>\n>>> instead of the intended:\n>>>\n>>> return get_mw_namespace_id($1);\n>>>\n>>> So, a minimal fix would be simply to change $namespace to $1.\n>>> Tightening the regex as I did in my example would be a bonus (though\n>>> probably ought to be a separate patch).\n>>\n>> so while i'm happy to just copy-paste your code in there, that's kind of\n>> a sensitive area of the code, as it was originally used only in the\n>> upload procedure, which I haven't tested at all. so i'm hesitant in just\n>> merging that in as is.\n> \n> I don't think there's a need to copy/paste my example code. If you\n> instead make the minimal suggested fix, then the resulting code will\n> be effectively equivalent to my example (minus the tighter regex).\n> \n>> i don't understand why or how this even works, to be honest: page names\n>> don't necessarily look like numbers, in fact, they generally don't. i\n>> don't understand why the patch submitted here even touches that function\n>> at all, considering that the function is only used on uploads. I just\n>> cargo-culted it from the original issue...\n> \n> I, myself, am not familiar with or a user of Mediawiki or with the Git\n> bridging, and I don't know what page names look like, but I'm pretty\n> well convinced from reading both the existing code and this patch that\n> the changes to get_mw_namespace_id_for_page() are really just a bug\n> fix to that function. My interpretation is that the function really\n> was intended to strip the \":id\" portion of \"name:id\" before calling\n> get_mw_namespace_id(); the fact that the original code neglects to do\n> so seems just an oversight. The fact that the regex uses capturing\n> parentheses implies strongly that it was indeed the intention to use\n> $1 in the call to get_mw_namespace_id(). Unlike the \"fix\" in the patch\n> you posted from Kevin, which is perhaps unnecessarily complicated, the\n> fix I suggested above is about a minimal as possible. That is,\n> changing:\n> \n> return get_mw_namespace_id($namespace);\n> \n> to:\n> \n> return get_mw_namespace_id($1);\n> \n> should achieve the same result. (It could be made more robust by\n> tightening the regex as in my example, but that's a separate topic,\n> not needed just to get the function to work as intended.)\n> \n\n"},{"id":"331305","messageId":"874lqh8jwr.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"CAPig+cSTp1Udo6xXk5-L6MpWBdiy4sPO__NcND03-89EvRgLHQ@mail.gmail.com","subject":"Re: [PATCH 4/4] remote-mediawiki: allow using (Main) as a namespace and skip special namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:11:00Z","receivedAt":"2017-10-30T02:11:07Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-10-29 15:49:28, Eric Sunshine wrote:\n> On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n>> Subject: remote-mediawiki: allow using (Main) as a namespace and skip special namespaces\n>\n> This patch is more difficult to review than it perhaps ought to be\n> since it is making multiple unrelated changes.\n>\n> It's not clear from the description what special namespaces are and\n> why they need to be skipped. It's also not clear why (Main) is\n> special. Perhaps the commit message(s) could explain these issues in\n> more detail.\n>\n> To simplify review and make it easier to gauge what it going on, it\n> might make sense to split this patch into at least two: one which\n> skips \"special namespaces\", and one which gives special treatment to\n> (Main).\n\nAgreed, I'll try to do that.\n\n> More below...\n>\n>> Reviewed-by: Antoine Beaupré \n>> Signed-off-by: Antoine Beaupré \n>> ---\n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> @@ -264,16 +264,27 @@ sub get_mw_tracked_categories {\n>> sub get_mw_tracked_namespaces {\n>> my $pages = shift;\n>> - foreach my $local_namespace (@tracked_namespaces) {\n>> - my $mw_pages = $mediawiki->list( {\n>> - action => 'query',\n>> - list => 'allpages',\n>> - apnamespace => get_mw_namespace_id($local_namespace),\n>> - aplimit => 'max' } )\n>> - || die $mediawiki->{error}->{code} . ': '\n>> - . $mediawiki->{error}->{details} . \"\\n\";\n>> - foreach my $page (@{$mw_pages}) {\n>> - $pages->{$page->{title}} = $page;\n>> + foreach my $local_namespace (sort @tracked_namespaces) {\n>> + my ($mw_pages, $namespace_id);\n>> + if ($local_namespace eq \"(Main)\") {\n>> + $namespace_id = 0;\n>> + } else {\n>> + $namespace_id = get_mw_namespace_id($local_namespace);\n>> + }\n>> + if ($namespace_id >= 0) {\n>\n> This may be problematic since get_mw_namespace_id() may return undef\n> rather than a number, in which case Perl will complain. Since the code\n> skips the $mediawiki query altogether when it encounters \"(Main)\", you\n> could fix this problem and simplify the code overall by simply\n> skipping the bulk of the foreach loop body instead of mucking around\n> with $namespace_id. For instance:\n>\n> foreach my $local_namespace (sort @tracked_namespaces) {\n> next if ($local_namespace eq \"(Main)\");\n> ...normal processing...\n> }\n\nAh yes. I see your point but it doesn't actually skip the query when it\nencouters main ($namespace_id >= 0).\n\n>> + if ($mw_pages = $mediawiki->list( {\n>> + action => 'query',\n>> + list => 'allpages',\n>> + apnamespace => $namespace_id,\n>> + aplimit => 'max' } )) {\n>> + print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n>\n> The original code did not emit this diagnostic but the new code does\n> so unconditionally. Is this just leftover debugging code or is\n> intended that all users should see this information all the time?\n\nThis is a known issue that permeates the whole remote at this point, and\nit is quite annoying.\n\nhttps://github.com/Git-Mediawiki/Git-Mediawiki/issues/30\n\nI have, however, considered it useful to include this to show progress\nas it can take a while to fetch all namespace information...\n\nObviously, once we figure out how to silence this stuff (ie. how to\nrecognize -q), it should be silenced like everything else, but until\nthen I think it's quite useful.\n\n>> + foreach my $page (@{$mw_pages}) {\n>> + $pages->{$page->{title}} = $page;\n>> + }\n>> + } else {\n>> + warn $mediawiki->{error}->{code} . ': '\n>> + . $mediawiki->{error}->{details} . \"\\n\";\n>\n> I guess this is the part which \"skips special namespaces\". The\n> original code die()'d but this merely warns. Aside from these \"special\n> namespaces\", are there genuine cases when the $mediawiki query would\n> return an error, and which should indeed die(), or is warning\n> appropriate for all $mediawiki query error cases?\n\nMaybe I didn't get the indentation right, but this } else { is for query\nfailures, *not* the if ($namespace_id < 0). So < 0 is just silently\nskipped.\n\nThe original code was die()'ing on failures, but I think that's a\nmistake: we should fetch what we can and warn on the failures. That\nallows the user to fix multiple problems at once instead of having to\nrerun the script repeatedly.\n\nA.\n\n-- \nLe féminisme n'a jamais tué personne\nLe machisme tue tous les jours.\n - Benoîte Groulx\n"},{"id":"331307","messageId":"871sll8jqv.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"6ed0d8d4-243a-7afc-bd37-a6f71004e195@ki-ai.org","subject":"Re: [PATCH 1/4] remote-mediawiki: add namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:14:32Z","receivedAt":"2017-10-30T02:14:37Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-10-29 23:08:00, Kevin wrote:\n> So I shared the patch some time ago (~2 years). Surprisingly its just\n> now getting attention. I guess some renewed interest in using mediawiki\n> with git.\n\nI think what's happening is that someone (ie. me :p) figured it was\nabout frigging time to actually send those patches to the git mailing\nlist. ;) And I'm glad we're seeing such good reviews, so thanks Eric for\nthat... \n\n> Myself, however, am no longer using mediawiki. Nor am I\n> completely clear on what the reasons were for using some variable or\n> another a couple of years ago. So... the best of luck, sorry I couldn't\n> be more helpful.\n\nThat's too bad, but thanks for the feedback anyways. :)\n\nFrankly, I'm tempted to just completely remove the\nget_mw_namespace_id_for_page hunk - it's completely unrelated to the\nrest of the patch.\n\nCould that be a bugfix for a separate issue that crept up in your\npatchset? For example this?\n\nhttps://github.com/Git-Mediawiki/Git-Mediawiki/issues/43\n\nA.\n-- \nThat's one of the remarkable things about life: it's never so bad that\nit can't get worse.\n - Calvin\n"},{"id":"331309","messageId":"87y3nt73uy.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"CAPig+cSTp1Udo6xXk5-L6MpWBdiy4sPO__NcND03-89EvRgLHQ@mail.gmail.com","subject":"Re: [PATCH 4/4] remote-mediawiki: allow using (Main) as a namespace and skip special namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:43:01Z","receivedAt":"2017-10-30T02:43:07Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-10-29 15:49:28, Eric Sunshine wrote:\n[...]\n>> Reviewed-by: Antoine Beaupré \n>> Signed-off-by: Antoine Beaupré \n>> ---\n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> @@ -264,16 +264,27 @@ sub get_mw_tracked_categories {\n>> sub get_mw_tracked_namespaces {\n>> my $pages = shift;\n>> - foreach my $local_namespace (@tracked_namespaces) {\n>> - my $mw_pages = $mediawiki->list( {\n>> - action => 'query',\n>> - list => 'allpages',\n>> - apnamespace => get_mw_namespace_id($local_namespace),\n>> - aplimit => 'max' } )\n>> - || die $mediawiki->{error}->{code} . ': '\n>> - . $mediawiki->{error}->{details} . \"\\n\";\n>> - foreach my $page (@{$mw_pages}) {\n>> - $pages->{$page->{title}} = $page;\n>> + foreach my $local_namespace (sort @tracked_namespaces) {\n>> + my ($mw_pages, $namespace_id);\n>> + if ($local_namespace eq \"(Main)\") {\n>> + $namespace_id = 0;\n>> + } else {\n>> + $namespace_id = get_mw_namespace_id($local_namespace);\n>> + }\n>> + if ($namespace_id >= 0) {\n>\n> This may be problematic since get_mw_namespace_id() may return undef\n> rather than a number, in which case Perl will complain.\n\n[...]\n\nActually, get_mw_namespace_id() doesn't seem like it can return undef -\ndid you mistake it with get_mw_namespace_id_for_page()?\n\nA.\n\n-- \nUncompromising war resistance and refusal to do military service under\nany circumstances.\n - Albert Einstein\n"},{"id":"331311","messageId":"20171030025142.19421-2-anarcat@debian.org","threadId":"47063","inReplyTo":"20171030025142.19421-1-anarcat@debian.org","subject":"[PATCH 1/7] remote-mediawiki: add namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:51:36Z","receivedAt":"2017-10-30T02:51:55Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Kevin \n\nThis introduces a new remote.origin.namespaces argument that is a\nspace-separated list of namespaces. The list of pages extract is then\ntaken from all the specified namespaces.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex e7f857c1a..5ffb57595 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -63,6 +63,10 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.categories\"));\n chomp(@tracked_categories);\n \n+# Just like @tracked_categories, but for MediaWiki namespaces.\n+my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+chomp(@tracked_namespaces);\n+\n # Import media files on pull\n my $import_media = run_git(\"config --get --bool remote.${remotename}.mediaimport\");\n chomp($import_media);\n@@ -256,6 +260,23 @@ sub get_mw_tracked_categories {\n \treturn;\n }\n \n+sub get_mw_tracked_namespaces {\n+ my $pages = shift;\n+ foreach my $local_namespace (@tracked_namespaces) {\n+ my $mw_pages = $mediawiki->list( {\n+ action => 'query',\n+ list => 'allpages',\n+ apnamespace => get_mw_namespace_id($local_namespace),\n+ aplimit => 'max' } )\n+ || die $mediawiki->{error}->{code} . ': '\n+ . $mediawiki->{error}->{details} . \"\\n\";\n+ foreach my $page (@{$mw_pages}) {\n+ $pages->{$page->{title}} = $page;\n+ }\n+ }\n+ return;\n+}\n+\n sub get_mw_all_pages {\n \tmy $pages = shift;\n \t# No user-provided list, get the list of pages from the API.\n@@ -319,6 +340,10 @@ sub get_mw_pages {\n \t\t$user_defined = 1;\n \t\tget_mw_tracked_categories(\\%pages);\n \t}\n+\tif (@tracked_namespaces) {\n+\t\t$user_defined = 1;\n+\t\tget_mw_tracked_namespaces(\\%pages);\n+\t}\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n-- \n2.11.0\n\n"},{"id":"331312","messageId":"20171030025142.19421-4-anarcat@debian.org","threadId":"47063","inReplyTo":"20171030025142.19421-1-anarcat@debian.org","subject":"[PATCH 3/7] remote-mediawiki: show known namespace choices on failure","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:51:38Z","receivedAt":"2017-10-30T02:51:57Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"If we fail to find a requested namespace, we should tell the user\nwhich ones we know about, since those were already fetched. This\nallows users to fetch all namespaces by specifying a dummy namespace,\nfailing, then copying the list of namespaces in the config.\n\nEventually, we should have a flag that allows fetching all namespaces\nautomatically.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex a1d783789..e7616e1a2 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -1334,7 +1334,9 @@ sub get_mw_namespace_id {\n \tmy $id;\n \n \tif (!defined $ns) {\n-\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n+\t\tmy @namespaces = sort keys %namespace_id;\n+\t\tfor (@namespaces) { s/ /_/g; }\n+\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki, known namespaces: @namespaces\\n\";\n \t\t$ns = {is_namespace => 0};\n \t\t$namespace_id{$name} = $ns;\n \t}\n-- \n2.11.0\n\n"},{"id":"331313","messageId":"20171030025142.19421-6-anarcat@debian.org","threadId":"47063","inReplyTo":"20171030025142.19421-1-anarcat@debian.org","subject":"[PATCH 5/7] remote-mediawiki: support fetching from (Main) namespace","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:51:40Z","receivedAt":"2017-10-30T02:51:59Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"When we specify a list of namespaces to fetch from, by default the MW\nAPI will not fetch from the default namespace, refered to as \"(Main)\"\nin the documentation:\n\nhttps://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces\n\nI haven't found a way to address that \"(Main)\" namespace when getting\nthe namespace ids: indeed, when listing namespaces, there is no\n\"canonical\" field for the main namespace, although there is a \"*\"\nfield that is set to \"\" (empty). So in theory, we could specify the\nempty namespace to get the main namespace, but that would make\nspecifying namespaces harder for the user: we would need to teach\nusers about the \"empty\" default namespace. It would also make the code\nmore complicated: we'd need to parse quotes in the configuration.\n\nSo we simply override the query here and allow the user to specify\n\"(Main)\" since that is the publicly documented name.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 5c85e64b6..2c2a7367b 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,9 +264,14 @@ sub get_mw_tracked_categories {\n sub get_mw_tracked_namespaces {\n my $pages = shift;\n foreach my $local_namespace (@tracked_namespaces) {\n- my $namespace_id = get_mw_namespace_id($local_namespace);\n+ my ($namespace_id, $mw_pages);\n+ if ($local_namespace eq \"(Main)\") {\n+ $namespace_id = 0;\n+ } else {\n+ $namespace_id = get_mw_namespace_id($local_namespace);\n+ }\n next if $namespace_id < 0; # virtual namespaces don't support allpages\n- my $mw_pages = $mediawiki->list( {\n+ $mw_pages = $mediawiki->list( {\n action => 'query',\n list => 'allpages',\n apnamespace => $namespace_id,\n-- \n2.11.0\n\n"},{"id":"331314","messageId":"20171030025142.19421-7-anarcat@debian.org","threadId":"47063","inReplyTo":"20171030025142.19421-1-anarcat@debian.org","subject":"[PATCH 6/7] remote-mediawiki: process namespaces in order","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:51:41Z","receivedAt":"2017-10-30T02:52:01Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Ideally, we'd process them in numeric order since that is more\nlogical, but we can't do that yet since this is where we find the\nnumeric identifiers in the first place. Lexicographic order is a good\ncompromise.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 2c2a7367b..5199af6f6 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -263,7 +263,7 @@ sub get_mw_tracked_categories {\n \n sub get_mw_tracked_namespaces {\n my $pages = shift;\n- foreach my $local_namespace (@tracked_namespaces) {\n+ foreach my $local_namespace (sort @tracked_namespaces) {\n my ($namespace_id, $mw_pages);\n if ($local_namespace eq \"(Main)\") {\n $namespace_id = 0;\n-- \n2.11.0\n\n"},{"id":"331315","messageId":"20171030025142.19421-8-anarcat@debian.org","threadId":"47063","inReplyTo":"20171030025142.19421-1-anarcat@debian.org","subject":"[PATCH 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:51:42Z","receivedAt":"2017-10-30T02:52:05Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Without this, the fetch process seems hanged while we fetch page\nlistings across the namespaces. Obviously, it should be possible to\nsilence this with -q, but that's an issue already present everywhere\nin the code and should be fixed separately:\n\nhttps://github.com/Git-Mediawiki/Git-Mediawiki/issues/30\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 5199af6f6..61e6dd798 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -278,6 +278,7 @@ sub get_mw_tracked_namespaces {\n aplimit => 'max' } )\n || die $mediawiki->{error}->{code} . ': '\n . $mediawiki->{error}->{details} . \"\\n\";\n+ print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n foreach my $page (@{$mw_pages}) {\n $pages->{$page->{title}} = $page;\n }\n-- \n2.11.0\n\n"},{"id":"331316","messageId":"20171030025142.19421-5-anarcat@debian.org","threadId":"47063","inReplyTo":"20171030025142.19421-1-anarcat@debian.org","subject":"[PATCH 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:51:39Z","receivedAt":"2017-10-30T02:52:08Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Virtual namespaces do not correspond to pages in the database and are\nautomatically generated by MediaWiki. It makes little sense,\ntherefore, to fetch pages from those namespaces and the MW API doesn't\nsupport listing those pages.\n\nAccording to the documentation, those virtual namespaces are currently\n\"Special\" (-1) and \"Media\" (-2) but we treat all negative namespaces\nas \"virtual\" as a future-proofing mechanism.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex e7616e1a2..5c85e64b6 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,10 +264,12 @@ sub get_mw_tracked_categories {\n sub get_mw_tracked_namespaces {\n my $pages = shift;\n foreach my $local_namespace (@tracked_namespaces) {\n+ my $namespace_id = get_mw_namespace_id($local_namespace);\n+ next if $namespace_id < 0; # virtual namespaces don't support allpages\n my $mw_pages = $mediawiki->list( {\n action => 'query',\n list => 'allpages',\n- apnamespace => get_mw_namespace_id($local_namespace),\n+ apnamespace => $namespace_id,\n aplimit => 'max' } )\n || die $mediawiki->{error}->{code} . ': '\n . $mediawiki->{error}->{details} . \"\\n\";\n-- \n2.11.0\n\n"},{"id":"331317","messageId":"20171030025142.19421-1-anarcat@debian.org","threadId":"47063","inReplyTo":"20171029160857.29460-2-anarcat@debian.org","subject":"[PATCH v2 0/7] remote-mediawiki: add namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:51:35Z","receivedAt":"2017-10-30T02:52:10Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"This patch series tries to integrate all the feedback received in the\nrecent review from Eric Sunshine. It completely removes the confusing\nchanges to get_mw_namespace_id_for_page() because I believe they are\nunrelated to the namespace support.\n\nI also split up the last patch in 4 different patches for clarity and\nfixed the vocabulary (it's \"virtual\" namespaces, not \"special\", which\nis a specific namespace).\n\nI left that die() in there because it makes the code a little cleaner\nand I'm lazy.\n\nThanks again for the good feedback!\n"},{"id":"331318","messageId":"20171030025142.19421-3-anarcat@debian.org","threadId":"47063","inReplyTo":"20171030025142.19421-1-anarcat@debian.org","subject":"[PATCH 2/7] remote-mediawiki: allow fetching namespaces with spaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T02:51:37Z","receivedAt":"2017-10-30T02:52:12Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Ingo Ruhnke \n\nwe still want to use spaces as separators in the config, but we should\nallow the user to specify namespaces with spaces, so we use underscore\nfor this.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 5ffb57595..a1d783789 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -65,6 +65,7 @@ chomp(@tracked_categories);\n \n # Just like @tracked_categories, but for MediaWiki namespaces.\n my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+for (@tracked_namespaces) { s/_/ /g; }\n chomp(@tracked_namespaces);\n \n # Import media files on pull\n-- \n2.11.0\n\n"},{"id":"331323","messageId":"CAPig+cTJTK8jSA+_hr2A8=t6CQ+CTjTuzvA24NTTEPEhMy9Jrw@mail.gmail.com","threadId":"47063","inReplyTo":"874lqh8jwr.fsf@curie.anarc.at","subject":"Re: [PATCH 4/4] remote-mediawiki: allow using (Main) as a namespace and skip special namespaces","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-10-30T03:49:03Z","receivedAt":"2017-10-30T03:49:09Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 10:11 PM, Antoine Beaupré wrote:\n> On 2017-10-29 15:49:28, Eric Sunshine wrote:\n>> On Sun, Oct 29, 2017 at 12:08 PM, Antoine Beaupré wrote:\n>>> + foreach my $local_namespace (sort @tracked_namespaces) {\n>>> + my ($mw_pages, $namespace_id);\n>>> + if ($local_namespace eq \"(Main)\") {\n>>> + $namespace_id = 0;\n>>> + } else {\n>>> + $namespace_id = get_mw_namespace_id($local_namespace);\n>>> + }\n>>> + if ($namespace_id >= 0) {\n>>\n>> This may be problematic since get_mw_namespace_id() may return undef\n>> rather than a number, in which case Perl will complain. Since the code\n>> skips the $mediawiki query altogether when it encounters \"(Main)\", you\n>> could fix this problem and simplify the code overall by simply\n>> skipping the bulk of the foreach loop body instead of mucking around\n>> with $namespace_id. For instance:\n>>\n>> foreach my $local_namespace (sort @tracked_namespaces) {\n>> next if ($local_namespace eq \"(Main)\");\n>> ...normal processing...\n>> }\n>\n> Ah yes. I see your point but it doesn't actually skip the query when it\n> encouters main ($namespace_id >= 0).\n\nAh, yes, you're right. My brain glossed over the '=' in '>=' for some reason.\n"},{"id":"331324","messageId":"CAPig+cSMRG0_kPpgFv5hco9Lk9jfYmJm-+rkQaw6vNQTdoEg7Q@mail.gmail.com","threadId":"47063","inReplyTo":"87y3nt73uy.fsf@curie.anarc.at","subject":"Re: [PATCH 4/4] remote-mediawiki: allow using (Main) as a namespace and skip special namespaces","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-10-30T03:52:16Z","receivedAt":"2017-10-30T03:52:22Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 10:43 PM, Antoine Beaupré wrote:\n> On 2017-10-29 15:49:28, Eric Sunshine wrote:\n>> This may be problematic since get_mw_namespace_id() may return undef\n>> rather than a number, in which case Perl will complain.\n>\n> Actually, get_mw_namespace_id() doesn't seem like it can return undef -\n> did you mistake it with get_mw_namespace_id_for_page()?\n\nHmm, no. What I see in the function is this:\n\n my $id;\n ...\n if ($ns->{is_namespace}) {\n $id = $ns->{id};\n }\n ...\n return $id;\n\nSo, $id starts undefined and is assigned only conditionally before\nbeing returned, but perhaps I'm missing some subtlety.\n"},{"id":"331339","messageId":"q7h98tfszzp5.fsf@orange.lip.ens-lyon.fr","threadId":"47063","inReplyTo":"20171029160857.29460-1-anarcat@debian.org","subject":"Re: [PATCH 0/4] WIP: git-remote-media wiki namespace support","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@univ-lyon1.fr","sentAt":"2017-10-30T10:40:06Z","receivedAt":"2017-10-30T10:40:12Z","isPatch":true,"sender":{"key":"matthieu.moy@univ-lyon1.fr","avatar":"https://gravatar.com/avatar/8ab83b763226bd297b59ddd1463a8bfc852924577191233f73a953b86888fe0c?d=mp&s=160"},"body":"Antoine Beaupré writes:\n\n> Obviously, doing unit tests against a full MediaWiki instance isn't\n> exactly trivial.\n\nNot trivial, but doable: there is all the infrastructure to do so in t/:\ninstall-wiki.sh to automatically install Mediawiki, and then a testsuite\nthat interacts with it.\n\nThis has been written under the assumption that the developer had a\nlighttpd instance running on localhost, but this can probably be adapted\nto run on Travis-CI (install lighttpd & Mediawiki in the install: part,\nand run the tests afterwards), so that anyone can run the tests by just\nsubmitting a pull-request to Git-Mediawiki.\n\nIf you are to work more on Git-Mediawiki, don't underestimate the\nusefullness of the testsuite (for example, Git-Mediawiki was developped\nagainst a prehistoric version of Mediawiki, the testsuite can help\nensuring it still works on the lastest version), nor the fun of playing\nwith install scripts and CI systems ;-).\n\nCheers,\n\n-- \nMatthieu Moy\nhttps://matthieu-moy.fr/\n"},{"id":"331340","messageId":"q7h93760zzk7.fsf@orange.lip.ens-lyon.fr","threadId":"47063","inReplyTo":"20171029160857.29460-2-anarcat@debian.org","subject":"Re: [PATCH 1/4] remote-mediawiki: add namespace support","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@univ-lyon1.fr","sentAt":"2017-10-30T10:43:04Z","receivedAt":"2017-10-30T10:43:10Z","isPatch":true,"sender":{"key":"matthieu.moy@univ-lyon1.fr","avatar":"https://gravatar.com/avatar/8ab83b763226bd297b59ddd1463a8bfc852924577191233f73a953b86888fe0c?d=mp&s=160"},"body":"Antoine Beaupré writes:\n\n> @@ -319,6 +341,10 @@ sub get_mw_pages {\n> \t\t$user_defined = 1;\n> \t\tget_mw_tracked_categories(\\%pages);\n> \t}\n> + if (@tracked_namespaces) {\n> + $user_defined = 1;\n> + get_mw_tracked_namespaces(\\%pages);\n> + }\n> \tif (!$user_defined) {\n> \t\tget_mw_all_pages(\\%pages);\n> \t}\n\nSpace Vs tabs indent issue (I have tab-width = 8, you probably have 4\nand this \"if\" looks underindented).\n\n-- \nMatthieu Moy\nhttps://matthieu-moy.fr/\n"},{"id":"331342","messageId":"87she07rt5.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"CAPig+cSMRG0_kPpgFv5hco9Lk9jfYmJm-+rkQaw6vNQTdoEg7Q@mail.gmail.com","subject":"Re: [PATCH 4/4] remote-mediawiki: allow using (Main) as a namespace and skip special namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T12:17:58Z","receivedAt":"2017-10-30T12:18:06Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-10-29 23:52:16, Eric Sunshine wrote:\n> On Sun, Oct 29, 2017 at 10:43 PM, Antoine Beaupré wrote:\n>> On 2017-10-29 15:49:28, Eric Sunshine wrote:\n>>> This may be problematic since get_mw_namespace_id() may return undef\n>>> rather than a number, in which case Perl will complain.\n>>\n>> Actually, get_mw_namespace_id() doesn't seem like it can return undef -\n>> did you mistake it with get_mw_namespace_id_for_page()?\n>\n> Hmm, no. What I see in the function is this:\n>\n> my $id;\n> ...\n> if ($ns->{is_namespace}) {\n> $id = $ns->{id};\n> }\n> ...\n> return $id;\n>\n> So, $id starts undefined and is assigned only conditionally before\n> being returned, but perhaps I'm missing some subtlety.\n\nAh yes, you're probably right there.\n\n-- \nDuring the initial stage of the struggle, the oppressed, instead of\nstriving for liberation, tend themselves to become oppressors The very\nstructure of their thought has been conditioned by the contradictions of\nthe concrete, existential situation by which they were shaped. Their\nideal is to be men; but for them, to be men is to be oppressors. This is\ntheir model of humanity.\n - Paulo Freire, Pedagogy of the Oppressed\n"},{"id":"331343","messageId":"87po947roj.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"q7h98tfszzp5.fsf@orange.lip.ens-lyon.fr","subject":"Re: [PATCH 0/4] WIP: git-remote-media wiki namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-10-30T12:20:44Z","receivedAt":"2017-10-30T12:20:49Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-10-30 11:40:06, Matthieu Moy wrote:\n> Antoine Beaupré writes:\n>\n>> Obviously, doing unit tests against a full MediaWiki instance isn't\n>> exactly trivial.\n>\n> Not trivial, but doable: there is all the infrastructure to do so in t/:\n> install-wiki.sh to automatically install Mediawiki, and then a testsuite\n> that interacts with it.\n>\n> This has been written under the assumption that the developer had a\n> lighttpd instance running on localhost, but this can probably be adapted\n> to run on Travis-CI (install lighttpd & Mediawiki in the install: part,\n> and run the tests afterwards), so that anyone can run the tests by just\n> submitting a pull-request to Git-Mediawiki.\n>\n> If you are to work more on Git-Mediawiki, don't underestimate the\n> usefullness of the testsuite (for example, Git-Mediawiki was developped\n> against a prehistoric version of Mediawiki, the testsuite can help\n> ensuring it still works on the lastest version), nor the fun of playing\n> with install scripts and CI systems ;-).\n\nHello!\n\nGlad to hear from you. :)\n\nSo I actually tried install-wiki.sh, and it \"failed to start lighttpd\"\nand told me to see logs. I couldn't find them and stopped there...\n\nIt would be great to hook this up into CI somewhere, but I suspect it\nisn't considering how it doesn't actually work out of the box.\n\nI'm hoping we can still do things and fix some things without going\nthrough that trouble, but I recognize it would be better to have unit\ntests operational.\n\nHonestly, I would prefer just having this thing work and not have to\nwork on it. :) I have lots of things on my plate and I'm just scratching\nan itch on this one - some backup script broke and I am trying to fix\nit. Once it works, my work is done, so unfortunately I cannot lead that\nproject (but I'd be happy to help when I can of course).\n\nA.\n\n-- \nThe greatest tragedy in mankind's entire history may be the hijacking of\nmorality by religion.\n - Arthur C. Clarke\n"},{"id":"331566","messageId":"CAPig+cSRrXj9BdgM6Q2OdBZtYJOVOnfo4BaG30C-_ug3QunPfQ@mail.gmail.com","threadId":"47063","inReplyTo":"20171030025142.19421-5-anarcat@debian.org","subject":"Re: [PATCH 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-01T13:52:09Z","receivedAt":"2017-11-01T13:52:15Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 10:51 PM, Antoine Beaupré wrote:\n> Virtual namespaces do not correspond to pages in the database and are\n> automatically generated by MediaWiki. It makes little sense,\n> therefore, to fetch pages from those namespaces and the MW API doesn't\n> support listing those pages.\n>\n> According to the documentation, those virtual namespaces are currently\n> \"Special\" (-1) and \"Media\" (-2) but we treat all negative namespaces\n> as \"virtual\" as a future-proofing mechanism.\n\nThis patch makes more sense now with the additional commentary.\nThanks. More below.\n\n> Signed-off-by: Antoine Beaupré \n> ---\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> index e7616e1a2..5c85e64b6 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -264,10 +264,12 @@ sub get_mw_tracked_categories {\n> sub get_mw_tracked_namespaces {\n> my $pages = shift;\n> foreach my $local_namespace (@tracked_namespaces) {\n> + my $namespace_id = get_mw_namespace_id($local_namespace);\n> + next if $namespace_id < 0; # virtual namespaces don't support allpages\n\nSince (it appears) that get_mw_namespace_id() can return undef, you\nprobably still need to take that into account before performing a\nnumeric comparison:\n\n next if !$namespace_id || $namespace_id < 0;\n\n> my $mw_pages = $mediawiki->list( {\n> action => 'query',\n> list => 'allpages',\n> - apnamespace => get_mw_namespace_id($local_namespace),\n> + apnamespace => $namespace_id,\n> aplimit => 'max' } )\n> || die $mediawiki->{error}->{code} . ': '\n> . $mediawiki->{error}->{details} . \"\\n\";\n"},{"id":"331575","messageId":"87wp3a54nj.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"CAPig+cSRrXj9BdgM6Q2OdBZtYJOVOnfo4BaG30C-_ug3QunPfQ@mail.gmail.com","subject":"Re: [PATCH 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-01T16:45:36Z","receivedAt":"2017-11-01T16:45:42Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-11-01 09:52:09, Eric Sunshine wrote:\n> On Sun, Oct 29, 2017 at 10:51 PM, Antoine Beaupré wrote:\n>> Virtual namespaces do not correspond to pages in the database and are\n>> automatically generated by MediaWiki. It makes little sense,\n>> therefore, to fetch pages from those namespaces and the MW API doesn't\n>> support listing those pages.\n>>\n>> According to the documentation, those virtual namespaces are currently\n>> \"Special\" (-1) and \"Media\" (-2) but we treat all negative namespaces\n>> as \"virtual\" as a future-proofing mechanism.\n>\n> This patch makes more sense now with the additional commentary.\n> Thanks. More below.\n>\n>> Signed-off-by: Antoine Beaupré \n>> ---\n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> index e7616e1a2..5c85e64b6 100755\n>> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n>> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> @@ -264,10 +264,12 @@ sub get_mw_tracked_categories {\n>> sub get_mw_tracked_namespaces {\n>> my $pages = shift;\n>> foreach my $local_namespace (@tracked_namespaces) {\n>> + my $namespace_id = get_mw_namespace_id($local_namespace);\n>> + next if $namespace_id < 0; # virtual namespaces don't support allpages\n>\n> Since (it appears) that get_mw_namespace_id() can return undef, you\n> probably still need to take that into account before performing a\n> numeric comparison:\n>\n> next if !$namespace_id || $namespace_id < 0;\n\nI would argue that this bug exists already elsewhere in the code - no\nerror handling exists there... Furthermore, it should be !defined()\nbecause it can be 0.\n\nIt might still worth fixing this, but I'm not sure what the process is\nhere - in the latest \"what's cooking\" Junio said this patchset would be\nmerged in \"next\". Should I reroll the patchset to fix this or not?\n\nA.\n\n-- \nN'aimer qu'un seul est barbarie, car c'est au détriment de tous les\nautres. Fût-ce l'amour de Dieu.\n - Nietzsche, \"Par delà le bien et le mal\"\n"},{"id":"331599","messageId":"CAPig+cTX1kBCk-phodTanU1dmwjM_2TNevKyGvdCWonqhEU5Dg@mail.gmail.com","threadId":"47063","inReplyTo":"20171030025142.19421-6-anarcat@debian.org","subject":"Re: [PATCH 5/7] remote-mediawiki: support fetching from (Main) namespace","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-01T19:56:51Z","receivedAt":"2017-11-01T19:56:57Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 10:51 PM, Antoine Beaupré wrote:\n> When we specify a list of namespaces to fetch from, by default the MW\n> API will not fetch from the default namespace, refered to as \"(Main)\"\n> in the documentation:\n>\n> https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces\n>\n> I haven't found a way to address that \"(Main)\" namespace when getting\n> the namespace ids: indeed, when listing namespaces, there is no\n> \"canonical\" field for the main namespace, although there is a \"*\"\n> field that is set to \"\" (empty). So in theory, we could specify the\n> empty namespace to get the main namespace, but that would make\n> specifying namespaces harder for the user: we would need to teach\n> users about the \"empty\" default namespace. It would also make the code\n> more complicated: we'd need to parse quotes in the configuration.\n>\n> So we simply override the query here and allow the user to specify\n> \"(Main)\" since that is the publicly documented name.\n\nThanks, this explanation makes the patch a lot clearer. More below...\n\n> Signed-off-by: Antoine Beaupré \n> ---\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -264,9 +264,14 @@ sub get_mw_tracked_categories {\n> sub get_mw_tracked_namespaces {\n> my $pages = shift;\n> foreach my $local_namespace (@tracked_namespaces) {\n> - my $namespace_id = get_mw_namespace_id($local_namespace);\n> + my ($namespace_id, $mw_pages);\n> + if ($local_namespace eq \"(Main)\") {\n> + $namespace_id = 0;\n> + } else {\n> + $namespace_id = get_mw_namespace_id($local_namespace);\n> + }\n\nI meant to ask this in the previous round, but with the earlier patch\nmixing several distinct changes into one, I plumb forgot: Would it\nmake sense to move this \"(Main)\" special case into\nget_mw_namespace_id() itself? After all, that function is all about\ndetermining an ID associated with a name, and \"(Main)\" is a name.\n\n> next if $namespace_id < 0; # virtual namespaces don't support allpages\n> - my $mw_pages = $mediawiki->list( {\n> + $mw_pages = $mediawiki->list( {\n\nWhy did the \"my\" of $my_pages get moved up to the top of the foreach\nloop? I can't seem to see any reason for it. Is this an unrelated\nchange accidentally included in this patch?\n\n> action => 'query',\n> list => 'allpages',\n> apnamespace => $namespace_id,\n> --\n"},{"id":"331601","messageId":"CAPig+cSKxaMkaHMKwRWB716RZ1=yBWFDaWJFFCNa29TUc+rpNQ@mail.gmail.com","threadId":"47063","inReplyTo":"20171030025142.19421-7-anarcat@debian.org","subject":"Re: [PATCH 6/7] remote-mediawiki: process namespaces in order","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-01T19:59:16Z","receivedAt":"2017-11-01T19:59:21Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 10:51 PM, Antoine Beaupré wrote:\n> Ideally, we'd process them in numeric order since that is more\n> logical, but we can't do that yet since this is where we find the\n> numeric identifiers in the first place. Lexicographic order is a good\n> compromise.\n\nThe reader of this commit message is left with the question: Why is\nthis change needed? Is it for the benefit of a human eventually seeing\nthe output? Is it because a subsequent patch requires a certain order?\n\n> Signed-off-by: Antoine Beaupré \n> ---\n> contrib/mw-to-git/git-remote-mediawiki.perl | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> index 2c2a7367b..5199af6f6 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -263,7 +263,7 @@ sub get_mw_tracked_categories {\n>\n> sub get_mw_tracked_namespaces {\n> my $pages = shift;\n> - foreach my $local_namespace (@tracked_namespaces) {\n> + foreach my $local_namespace (sort @tracked_namespaces) {\n> my ($namespace_id, $mw_pages);\n> if ($local_namespace eq \"(Main)\") {\n> $namespace_id = 0;\n> --\n"},{"id":"331602","messageId":"CAPig+cR9h-DS3NLzHVdnad420HtN3Afg+1BHBZBzG1EG0=U45g@mail.gmail.com","threadId":"47063","inReplyTo":"20171030025142.19421-8-anarcat@debian.org","subject":"Re: [PATCH 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-01T20:01:20Z","receivedAt":"2017-11-01T20:01:25Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Oct 29, 2017 at 10:51 PM, Antoine Beaupré wrote:\n> Without this, the fetch process seems hanged while we fetch page\n> listings across the namespaces. Obviously, it should be possible to\n> silence this with -q, but that's an issue already present everywhere\n> in the code and should be fixed separately:\n>\n> https://github.com/Git-Mediawiki/Git-Mediawiki/issues/30\n\nUnlike the previous round, this commit message makes it clear that\nthis new printed message is indeed intentional. Thanks.\n\n> Signed-off-by: Antoine Beaupré \n> ---\n> contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> index 5199af6f6..61e6dd798 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -278,6 +278,7 @@ sub get_mw_tracked_namespaces {\n> aplimit => 'max' } )\n> || die $mediawiki->{error}->{code} . ': '\n> . $mediawiki->{error}->{details} . \"\\n\";\n> + print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n> foreach my $page (@{$mw_pages}) {\n> $pages->{$page->{title}} = $page;\n> }\n> --\n> 2.11.0\n"},{"id":"331635","messageId":"xmqqvaitsc9z.fsf@gitster.mtv.corp.google.com","threadId":"47063","inReplyTo":"87wp3a54nj.fsf@curie.anarc.at","subject":"Re: [PATCH 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-02T01:24:40Z","receivedAt":"2017-11-02T01:24:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Antoine Beaupré writes:\n\n> It might still worth fixing this, but I'm not sure what the process is\n> here - in the latest \"what's cooking\" Junio said this patchset would be\n> merged in \"next\". Should I reroll the patchset to fix this or not?\n\nThe process is for you (the contributor of the topic) to yell at me,\n\"don't merge it yet, there still are updates to come\".\n\nThat message _may_ come to late, in which case we may have to go\nincremental, but I usually try to leave at least a few days between\nthe time I mark a topic as \"will merge\" and the time I actually do\nthe merge, for this exact reason.\n\nThanks.\n"},{"id":"331726","messageId":"8760as5qfl.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"CAPig+cTX1kBCk-phodTanU1dmwjM_2TNevKyGvdCWonqhEU5Dg@mail.gmail.com","subject":"Re: [PATCH 5/7] remote-mediawiki: support fetching from (Main) namespace","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:19:42Z","receivedAt":"2017-11-02T21:19:53Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-11-01 15:56:51, Eric Sunshine wrote:\n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> @@ -264,9 +264,14 @@ sub get_mw_tracked_categories {\n>> sub get_mw_tracked_namespaces {\n>> my $pages = shift;\n>> foreach my $local_namespace (@tracked_namespaces) {\n>> - my $namespace_id = get_mw_namespace_id($local_namespace);\n>> + my ($namespace_id, $mw_pages);\n>> + if ($local_namespace eq \"(Main)\") {\n>> + $namespace_id = 0;\n>> + } else {\n>> + $namespace_id = get_mw_namespace_id($local_namespace);\n>> + }\n>\n> I meant to ask this in the previous round, but with the earlier patch\n> mixing several distinct changes into one, I plumb forgot: Would it\n> make sense to move this \"(Main)\" special case into\n> get_mw_namespace_id() itself? After all, that function is all about\n> determining an ID associated with a name, and \"(Main)\" is a name.\n\nRight. At first sight, I agree: get_mw_namespace_id should do the right\nthing. But then, I look at the code of that function, and it strikes me\nas ... well... really hard to actually do this the right way.\n\nIn fact, I suspect that passing \"\" to get_mw_namespace_id would actually\ndo the right thing. The problem, as I explained before, is that passing\nthat in the configuration is pretty hard: it would needlessly complicate\nthe configuration setting, so I think it's a fair shortcut to do it\nhere.\n\n>> next if $namespace_id < 0; # virtual namespaces don't support allpages\n>> - my $mw_pages = $mediawiki->list( {\n>> + $mw_pages = $mediawiki->list( {\n>\n> Why did the \"my\" of $my_pages get moved up to the top of the foreach\n> loop? I can't seem to see any reason for it. Is this an unrelated\n> change accidentally included in this patch?\n\nJust a habit of declaring functions at the beginning of a block. Maybe\nit's because I'm old? :)\n\nI'll reroll a last patchset with those fixes.\n\nA.\n\n-- \nOne of the strongest motives that leads men to art and science is\nescape from everyday life with its painful crudity and hopeless\ndreariness. Such men make this cosmos and its construction the pivot\nof their emotional life, in order to find the peace and security which\nthey cannot find in the narrow whirlpool of personal experience.\n - Albert Einstein\n"},{"id":"331728","messageId":"87375w5qdw.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"xmqqvaitsc9z.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:20:43Z","receivedAt":"2017-11-02T21:20:48Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-11-02 10:24:40, Junio C Hamano wrote:\n> Antoine Beaupré writes:\n>\n>> It might still worth fixing this, but I'm not sure what the process is\n>> here - in the latest \"what's cooking\" Junio said this patchset would be\n>> merged in \"next\". Should I reroll the patchset to fix this or not?\n>\n> The process is for you (the contributor of the topic) to yell at me,\n> \"don't merge it yet, there still are updates to come\".\n\nYELL! \"don't merge it yet, there still are updates to come\". :)\n\n> That message _may_ come to late, in which case we may have to go\n> incremental, but I usually try to leave at least a few days between\n> the time I mark a topic as \"will merge\" and the time I actually do\n> the merge, for this exact reason.\n\nAwesome, thanks for the update.\n\ni'll roll a v4 with the last tweaks, hopefully that will be the last.\n\na.\n\n-- \nHow inappropriate to call this planet 'Earth' when it is quite clearly\n'Ocean'.\n - Arthur C. Clarke\n"},{"id":"331729","messageId":"20171102212518.1601-1-anarcat@debian.org","threadId":"47063","inReplyTo":"20171030025142.19421-1-anarcat@debian.org","subject":"[PATCH v3 0/7] remote-mediawiki: namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:25:11Z","receivedAt":"2017-11-02T21:25:31Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"This should be the final roll of patches for namespace support. I\nincluded the undef check even though that problem occurs elsewhere in\nthe code. I also removed the needless \"my\" move.\n\nHopefully that should be the last in the queue!\n\n"},{"id":"331730","messageId":"20171102212518.1601-2-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v3 1/7] remote-mediawiki: add namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:25:12Z","receivedAt":"2017-11-02T21:25:38Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Kevin \n\nThis introduces a new remote.origin.namespaces argument that is a\nspace-separated list of namespaces. The list of pages extract is then\ntaken from all the specified namespaces.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex e7f857c1a..5ffb57595 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -63,6 +63,10 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.categories\"));\n chomp(@tracked_categories);\n \n+# Just like @tracked_categories, but for MediaWiki namespaces.\n+my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+chomp(@tracked_namespaces);\n+\n # Import media files on pull\n my $import_media = run_git(\"config --get --bool remote.${remotename}.mediaimport\");\n chomp($import_media);\n@@ -256,6 +260,23 @@ sub get_mw_tracked_categories {\n \treturn;\n }\n \n+sub get_mw_tracked_namespaces {\n+ my $pages = shift;\n+ foreach my $local_namespace (@tracked_namespaces) {\n+ my $mw_pages = $mediawiki->list( {\n+ action => 'query',\n+ list => 'allpages',\n+ apnamespace => get_mw_namespace_id($local_namespace),\n+ aplimit => 'max' } )\n+ || die $mediawiki->{error}->{code} . ': '\n+ . $mediawiki->{error}->{details} . \"\\n\";\n+ foreach my $page (@{$mw_pages}) {\n+ $pages->{$page->{title}} = $page;\n+ }\n+ }\n+ return;\n+}\n+\n sub get_mw_all_pages {\n \tmy $pages = shift;\n \t# No user-provided list, get the list of pages from the API.\n@@ -319,6 +340,10 @@ sub get_mw_pages {\n \t\t$user_defined = 1;\n \t\tget_mw_tracked_categories(\\%pages);\n \t}\n+\tif (@tracked_namespaces) {\n+\t\t$user_defined = 1;\n+\t\tget_mw_tracked_namespaces(\\%pages);\n+\t}\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n-- \n2.11.0\n\n"},{"id":"331731","messageId":"20171102212518.1601-3-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v3 2/7] remote-mediawiki: allow fetching namespaces with spaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:25:13Z","receivedAt":"2017-11-02T21:25:44Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Ingo Ruhnke \n\nwe still want to use spaces as separators in the config, but we should\nallow the user to specify namespaces with spaces, so we use underscore\nfor this.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 5ffb57595..a1d783789 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -65,6 +65,7 @@ chomp(@tracked_categories);\n \n # Just like @tracked_categories, but for MediaWiki namespaces.\n my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+for (@tracked_namespaces) { s/_/ /g; }\n chomp(@tracked_namespaces);\n \n # Import media files on pull\n-- \n2.11.0\n\n"},{"id":"331732","messageId":"20171102212518.1601-4-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v3 3/7] remote-mediawiki: show known namespace choices on failure","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:25:14Z","receivedAt":"2017-11-02T21:25:45Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"If we fail to find a requested namespace, we should tell the user\nwhich ones we know about, since those were already fetched. This\nallows users to fetch all namespaces by specifying a dummy namespace,\nfailing, then copying the list of namespaces in the config.\n\nEventually, we should have a flag that allows fetching all namespaces\nautomatically.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex a1d783789..e7616e1a2 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -1334,7 +1334,9 @@ sub get_mw_namespace_id {\n \tmy $id;\n \n \tif (!defined $ns) {\n-\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n+\t\tmy @namespaces = sort keys %namespace_id;\n+\t\tfor (@namespaces) { s/ /_/g; }\n+\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki, known namespaces: @namespaces\\n\";\n \t\t$ns = {is_namespace => 0};\n \t\t$namespace_id{$name} = $ns;\n \t}\n-- \n2.11.0\n\n"},{"id":"331733","messageId":"20171102212518.1601-5-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v3 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:25:15Z","receivedAt":"2017-11-02T21:25:47Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Virtual namespaces do not correspond to pages in the database and are\nautomatically generated by MediaWiki. It makes little sense,\ntherefore, to fetch pages from those namespaces and the MW API doesn't\nsupport listing those pages.\n\nAccording to the documentation, those virtual namespaces are currently\n\"Special\" (-1) and \"Media\" (-2) but we treat all negative namespaces\nas \"virtual\" as a future-proofing mechanism.\n\nReviewed-by: Eric Sunshine \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex e7616e1a2..21fb2e302 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,10 +264,13 @@ sub get_mw_tracked_categories {\n sub get_mw_tracked_namespaces {\n my $pages = shift;\n foreach my $local_namespace (@tracked_namespaces) {\n+ my $namespace_id = get_mw_namespace_id($local_namespace);\n+ # virtual namespaces don't support allpages\n+ next if !defined($namespace_id) || $namespace_id < 0;\n my $mw_pages = $mediawiki->list( {\n action => 'query',\n list => 'allpages',\n- apnamespace => get_mw_namespace_id($local_namespace),\n+ apnamespace => $namespace_id,\n aplimit => 'max' } )\n || die $mediawiki->{error}->{code} . ': '\n . $mediawiki->{error}->{details} . \"\\n\";\n-- \n2.11.0\n\n"},{"id":"331734","messageId":"20171102212518.1601-6-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v3 5/7] remote-mediawiki: support fetching from (Main) namespace","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:25:16Z","receivedAt":"2017-11-02T21:25:49Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"When we specify a list of namespaces to fetch from, by default the MW\nAPI will not fetch from the default namespace, refered to as \"(Main)\"\nin the documentation:\n\nhttps://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces\n\nI haven't found a way to address that \"(Main)\" namespace when getting\nthe namespace ids: indeed, when listing namespaces, there is no\n\"canonical\" field for the main namespace, although there is a \"*\"\nfield that is set to \"\" (empty). So in theory, we could specify the\nempty namespace to get the main namespace, but that would make\nspecifying namespaces harder for the user: we would need to teach\nusers about the \"empty\" default namespace. It would also make the code\nmore complicated: we'd need to parse quotes in the configuration.\n\nSo we simply override the query here and allow the user to specify\n\"(Main)\" since that is the publicly documented name.\n\nReviewed-by: Eric Sunshine \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 21fb2e302..898541a9f 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,7 +264,12 @@ sub get_mw_tracked_categories {\n sub get_mw_tracked_namespaces {\n my $pages = shift;\n foreach my $local_namespace (@tracked_namespaces) {\n- my $namespace_id = get_mw_namespace_id($local_namespace);\n+ my $namespace_id;\n+ if ($local_namespace eq \"(Main)\") {\n+ $namespace_id = 0;\n+ } else {\n+ $namespace_id = get_mw_namespace_id($local_namespace);\n+ }\n # virtual namespaces don't support allpages\n next if !defined($namespace_id) || $namespace_id < 0;\n my $mw_pages = $mediawiki->list( {\n-- \n2.11.0\n\n"},{"id":"331735","messageId":"20171102212518.1601-8-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v3 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:25:18Z","receivedAt":"2017-11-02T21:25:52Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Without this, the fetch process seems hanged while we fetch page\nlistings across the namespaces. Obviously, it should be possible to\nsilence this with -q, but that's an issue already present everywhere\nin the code and should be fixed separately:\n\nhttps://github.com/Git-Mediawiki/Git-Mediawiki/issues/30\n\nReviewed-by: Eric Sunshine \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex f53e638cf..dc43a950b 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -279,6 +279,7 @@ sub get_mw_tracked_namespaces {\n aplimit => 'max' } )\n || die $mediawiki->{error}->{code} . ': '\n . $mediawiki->{error}->{details} . \"\\n\";\n+ print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n foreach my $page (@{$mw_pages}) {\n $pages->{$page->{title}} = $page;\n }\n-- \n2.11.0\n\n"},{"id":"331736","messageId":"20171102212518.1601-7-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v3 6/7] remote-mediawiki: process namespaces in order","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T21:25:17Z","receivedAt":"2017-11-02T21:25:54Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Ideally, we'd process them in numeric order since that is more\nlogical, but we can't do that yet since this is where we find the\nnumeric identifiers in the first place. Lexicographic order is a good\ncompromise.\n\nReviewed-by: Eric Sunshine \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 898541a9f..f53e638cf 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -263,7 +263,7 @@ sub get_mw_tracked_categories {\n \n sub get_mw_tracked_namespaces {\n my $pages = shift;\n- foreach my $local_namespace (@tracked_namespaces) {\n+ foreach my $local_namespace (sort @tracked_namespaces) {\n my $namespace_id;\n if ($local_namespace eq \"(Main)\") {\n $namespace_id = 0;\n-- \n2.11.0\n\n"},{"id":"331738","messageId":"20171102221807.juon446b6zfw64cq@laptop.local","threadId":"47063","inReplyTo":"20171102212518.1601-8-anarcat@debian.org","subject":"Re: [PATCH v3 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Thomas Adam","fromEmail":"thomas@xteddy.org","sentAt":"2017-11-02T22:18:07Z","receivedAt":"2017-11-02T22:18:17Z","isPatch":true,"sender":{"key":"thomas@xteddy.org","avatar":"https://gravatar.com/avatar/e7256db4738e501e5d2e84f00bb0bd99503165729573848a03330301fc2adc4a?d=mp&s=160"},"body":"Hi,\n\nOn Thu, Nov 02, 2017 at 05:25:18PM -0400, Antoine Beaupré wrote:\n> + print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n\nHow is this any different to using warn()? I appreciate you're using a\nglobbed filehandle, but it seems superfluous to me.\n\nKindly,\nThomas\n"},{"id":"331740","messageId":"87vais48rg.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"20171102221807.juon446b6zfw64cq@laptop.local","subject":"Re: [PATCH v3 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T22:26:43Z","receivedAt":"2017-11-02T22:26:48Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-11-02 22:18:07, Thomas Adam wrote:\n> Hi,\n>\n> On Thu, Nov 02, 2017 at 05:25:18PM -0400, Antoine Beaupré wrote:\n>> + print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n>\n> How is this any different to using warn()? I appreciate you're using a\n> globbed filehandle, but it seems superfluous to me.\n\nIt's what is used everywhere in the module, I'm just tagging along.\n\nThis was discussed before: there's an issue about cleaning up the\nmessaging in that module, that can be fixed separately.\n\nA.\n-- \nN'aimer qu'un seul est barbarie, car c'est au détriment de tous les\nautres. Fût-ce l'amour de Dieu.\n - Nietzsche, \"Par delà le bien et le mal\"\n"},{"id":"331741","messageId":"20171102223102.j3niifmupcs77p3r@laptop.local","threadId":"47063","inReplyTo":"87vais48rg.fsf@curie.anarc.at","subject":"Re: [PATCH v3 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Thomas Adam","fromEmail":"thomas@xteddy.org","sentAt":"2017-11-02T22:31:02Z","receivedAt":"2017-11-02T22:31:10Z","isPatch":true,"sender":{"key":"thomas@xteddy.org","avatar":"https://gravatar.com/avatar/e7256db4738e501e5d2e84f00bb0bd99503165729573848a03330301fc2adc4a?d=mp&s=160"},"body":"On Thu, Nov 02, 2017 at 06:26:43PM -0400, Antoine Beaupré wrote:\n> On 2017-11-02 22:18:07, Thomas Adam wrote:\n> > Hi,\n> >\n> > On Thu, Nov 02, 2017 at 05:25:18PM -0400, Antoine Beaupré wrote:\n> >> + print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n> >\n> > How is this any different to using warn()? I appreciate you're using a\n> > globbed filehandle, but it seems superfluous to me.\n> \n> It's what is used everywhere in the module, I'm just tagging along.\n> \n> This was discussed before: there's an issue about cleaning up the\n> messaging in that module, that can be fixed separately.\n\nUnderstood. That should happen sooner rather than later.\n\n-- Thomas Adam\n"},{"id":"331742","messageId":"CAPig+cR0jG65LbopxqPpidaaNUSTRq9tboZpv0RPWyWUkSEGUw@mail.gmail.com","threadId":"47063","inReplyTo":"20171102212518.1601-5-anarcat@debian.org","subject":"Re: [PATCH v3 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-02T22:43:00Z","receivedAt":"2017-11-02T22:43:08Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Nov 2, 2017 at 5:25 PM, Antoine Beaupré wrote:\n> Virtual namespaces do not correspond to pages in the database and are\n> automatically generated by MediaWiki. It makes little sense,\n> therefore, to fetch pages from those namespaces and the MW API doesn't\n> support listing those pages.\n>\n> According to the documentation, those virtual namespaces are currently\n> \"Special\" (-1) and \"Media\" (-2) but we treat all negative namespaces\n> as \"virtual\" as a future-proofing mechanism.\n>\n> Reviewed-by: Eric Sunshine \n\nIt probably would be best to omit this Reviewed-by: since it was not\nprovided explicitly. More importantly, I'm neither a user of nor\nfamiliar with MediaWiki or its API, so a Reviewed-by: from me has\nlittle or no value. Probably best would be for someone such as\nMatthieu to give his Reviewed-by: if he so desires.\n\n> Signed-off-by: Antoine Beaupré \n"},{"id":"331743","messageId":"CAPig+cRtz7xuBy7QUgr_BHomH+txs1ynxdV9Z28xu1CDVW+uuQ@mail.gmail.com","threadId":"47063","inReplyTo":"20171102212518.1601-6-anarcat@debian.org","subject":"Re: [PATCH v3 5/7] remote-mediawiki: support fetching from (Main) namespace","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-02T22:48:23Z","receivedAt":"2017-11-02T22:48:28Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Nov 2, 2017 at 5:25 PM, Antoine Beaupré wrote:\n> When we specify a list of namespaces to fetch from, by default the MW\n> API will not fetch from the default namespace, refered to as \"(Main)\"\n> in the documentation:\n>\n> https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces\n>\n> I haven't found a way to address that \"(Main)\" namespace when getting\n> the namespace ids: indeed, when listing namespaces, there is no\n> \"canonical\" field for the main namespace, although there is a \"*\"\n> field that is set to \"\" (empty). So in theory, we could specify the\n> empty namespace to get the main namespace, but that would make\n> specifying namespaces harder for the user: we would need to teach\n> users about the \"empty\" default namespace. It would also make the code\n> more complicated: we'd need to parse quotes in the configuration.\n>\n> So we simply override the query here and allow the user to specify\n> \"(Main)\" since that is the publicly documented name.\n>\n> Reviewed-by: Eric Sunshine \n\nAs with the previous patch, it would be best to drop this Reviewed-by:\nsince it has no value with my name attached to it and was not provided\nexplicitly.\n\n> Signed-off-by: Antoine Beaupré \n"},{"id":"331744","messageId":"CAPig+cRTaGb=WNkyTT4btnhf+H+9SJC7N5hF+KxJrmy55Oi-gg@mail.gmail.com","threadId":"47063","inReplyTo":"20171102212518.1601-7-anarcat@debian.org","subject":"Re: [PATCH v3 6/7] remote-mediawiki: process namespaces in order","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-02T22:49:21Z","receivedAt":"2017-11-02T22:49:26Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Nov 2, 2017 at 5:25 PM, Antoine Beaupré wrote:\n> Ideally, we'd process them in numeric order since that is more\n> logical, but we can't do that yet since this is where we find the\n> numeric identifiers in the first place. Lexicographic order is a good\n> compromise.\n>\n> Reviewed-by: Eric Sunshine \n\nDitto: It would be best to drop this Reviewed-by: since it has no\nvalue with my name attached to it and was not provided explicitly.\n\n> Signed-off-by: Antoine Beaupré \n"},{"id":"331745","messageId":"CAPig+cRFLv46hw6Mn__o5VLi5B9edA26zD_Pfca6A9xe4a6bJQ@mail.gmail.com","threadId":"47063","inReplyTo":"20171102212518.1601-8-anarcat@debian.org","subject":"Re: [PATCH v3 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-02T22:50:19Z","receivedAt":"2017-11-02T22:50:24Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Nov 2, 2017 at 5:25 PM, Antoine Beaupré wrote:\n> Without this, the fetch process seems hanged while we fetch page\n> listings across the namespaces. Obviously, it should be possible to\n> silence this with -q, but that's an issue already present everywhere\n> in the code and should be fixed separately:\n>\n> https://github.com/Git-Mediawiki/Git-Mediawiki/issues/30\n>\n> Reviewed-by: Eric Sunshine \n\nDitto: It would be best to drop this Reviewed-by: since it has no\nvalue with my name attached to it and was not provided explicitly.\n\n> Signed-off-by: Antoine Beaupré \n"},{"id":"331746","messageId":"87r2tg47hh.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"CAPig+cR0jG65LbopxqPpidaaNUSTRq9tboZpv0RPWyWUkSEGUw@mail.gmail.com","subject":"Re: [PATCH v3 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T22:54:18Z","receivedAt":"2017-11-02T22:54:24Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-11-02 18:43:00, Eric Sunshine wrote:\n> On Thu, Nov 2, 2017 at 5:25 PM, Antoine Beaupré wrote:\n>> Virtual namespaces do not correspond to pages in the database and are\n>> automatically generated by MediaWiki. It makes little sense,\n>> therefore, to fetch pages from those namespaces and the MW API doesn't\n>> support listing those pages.\n>>\n>> According to the documentation, those virtual namespaces are currently\n>> \"Special\" (-1) and \"Media\" (-2) but we treat all negative namespaces\n>> as \"virtual\" as a future-proofing mechanism.\n>>\n>> Reviewed-by: Eric Sunshine \n>\n> It probably would be best to omit this Reviewed-by: since it was not\n> provided explicitly. More importantly, I'm neither a user of nor\n> familiar with MediaWiki or its API, so a Reviewed-by: from me has\n> little or no value. Probably best would be for someone such as\n> Matthieu to give his Reviewed-by: if he so desires.\n\nAlright, I was wondering what the process was for those. I didn't want\nto leave your contributions by the wayside...\n\nI'll wait a little while longer for more feedback and then resend\nwithout those. unless...\n\n@junio: my github repo has the branch without those Reviewed-by tags,\niirc. so if you can to merge from there, that will keep me from sending\nyet another pile of patches for such a trivial change...\n\na.\n\n-- \nSemantics is the gravity of abstraction.\n"},{"id":"331747","messageId":"87o9ok46qu.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"20171102223102.j3niifmupcs77p3r@laptop.local","subject":"Re: [PATCH v3 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-02T23:10:17Z","receivedAt":"2017-11-02T23:10:23Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-11-02 22:31:02, Thomas Adam wrote:\n> On Thu, Nov 02, 2017 at 06:26:43PM -0400, Antoine Beaupré wrote:\n>> On 2017-11-02 22:18:07, Thomas Adam wrote:\n>> > Hi,\n>> >\n>> > On Thu, Nov 02, 2017 at 05:25:18PM -0400, Antoine Beaupré wrote:\n>> >> + print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n>> >\n>> > How is this any different to using warn()? I appreciate you're using a\n>> > globbed filehandle, but it seems superfluous to me.\n>> \n>> It's what is used everywhere in the module, I'm just tagging along.\n>> \n>> This was discussed before: there's an issue about cleaning up the\n>> messaging in that module, that can be fixed separately.\n>\n> Understood. That should happen sooner rather than later.\n\nActually, is there a standard way to do this in git with Perl\nextensions? I know about \"option verbosity N\" but how should I translate\nthis into Perl? Carp? Warn? Log::Any? Log4perl?\n\nRecommendations welcome...\n\nA.\n\n-- \nSi Dieu existe, j'espère qu'Il a une excuse valable\n - Daniel Pennac\n"},{"id":"331822","messageId":"20171104095740.hhdgcqgtyjejmqjm@laptop.local","threadId":"47063","inReplyTo":"87o9ok46qu.fsf@curie.anarc.at","subject":"Re: [PATCH v3 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Thomas Adam","fromEmail":"thomas@xteddy.org","sentAt":"2017-11-04T09:57:40Z","receivedAt":"2017-11-04T09:57:49Z","isPatch":true,"sender":{"key":"thomas@xteddy.org","avatar":"https://gravatar.com/avatar/e7256db4738e501e5d2e84f00bb0bd99503165729573848a03330301fc2adc4a?d=mp&s=160"},"body":"On Thu, Nov 02, 2017 at 07:10:17PM -0400, Antoine Beaupré wrote:\n> Actually, is there a standard way to do this in git with Perl\n> extensions? I know about \"option verbosity N\" but how should I translate\n> this into Perl? Carp? Warn? Log::Any? Log4perl?\n\nNo, not really. From a quick glance at some of the existing perl code in git,\na lot of it continues to use \"print STDERR\" -- but then to be fair, a lot of\nthe perl code also reads like it has been written by C programmers...\n\nWhile there's nothing wrong with using \"print STDERR\", it's probably wiser to\ntransition this to using Carp in the long run -- it would decrease the\nround-trip time to debugging should there be a situation where that was\nneeded, and hence I would recommend using \"warn\" for less-severe\nerrors/debugging.\n\n-- Thomas Adam\n"},{"id":"331823","messageId":"20171104105712.m5zhj42nxvkhlc55@laptop.local","threadId":"47063","inReplyTo":"20171029160857.29460-4-anarcat@debian.org","subject":"Re: [PATCH 3/4] remote-mediawiki: show known namespace choices on failure","fromName":"Thomas Adam","fromEmail":"thomas@xteddy.org","sentAt":"2017-11-04T10:57:12Z","receivedAt":"2017-11-04T10:57:20Z","isPatch":true,"sender":{"key":"thomas@xteddy.org","avatar":"https://gravatar.com/avatar/e7256db4738e501e5d2e84f00bb0bd99503165729573848a03330301fc2adc4a?d=mp&s=160"},"body":"On Sun, Oct 29, 2017 at 12:08:56PM -0400, Antoine Beaupré wrote:\n> if we fail to find a requested namespace, we should tell the user\n> which ones we know about, since we already do. this allows users to\n> feetch all namespaces by specifying a dummy namespace, failing, then\n> copying the list of namespaces in the config.\n> \n> eventually, we should have a flag that allows fetching all namespaces\n> automatically.\n> \n> Reviewed-by: Antoine Beaupré \n> Signed-off-by: Antoine Beaupré \n> ---\n> contrib/mw-to-git/git-remote-mediawiki.perl | 4 +++-\n> 1 file changed, 3 insertions(+), 1 deletion(-)\n> \n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> index fc48846a1..07cc74bac 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -1334,7 +1334,9 @@ sub get_mw_namespace_id {\n> \tmy $id;\n> \n> \tif (!defined $ns) {\n> -\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n> +\t\tmy @namespaces = sort keys %namespace_id;\n> +\t\tfor (@namespaces) { s/ /_/g; }\n\nI am sure we can improve upon the need to process @namespaces twice:\n\n my @namespaces = map { s/ /_/g; $_; } sort keys %namespaces_id;\n\n-- Thomas Adam\n"},{"id":"331891","messageId":"xmqqh8u8kzri.fsf@gitster.mtv.corp.google.com","threadId":"47063","inReplyTo":"87375w5qdw.fsf@curie.anarc.at","subject":"Re: [PATCH 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-06T00:38:09Z","receivedAt":"2017-11-06T00:38:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Antoine Beaupré writes:\n\n> On 2017-11-02 10:24:40, Junio C Hamano wrote:\n>> Antoine Beaupré writes:\n>>\n>>> It might still worth fixing this, but I'm not sure what the process is\n>>> here - in the latest \"what's cooking\" Junio said this patchset would be\n>>> merged in \"next\". Should I reroll the patchset to fix this or not?\n>>\n>> The process is for you (the contributor of the topic) to yell at me,\n>> \"don't merge it yet, there still are updates to come\".\n>\n> YELL! \"don't merge it yet, there still are updates to come\". :)\n\nThanks; heard you loud and clear.\n\n>> That message _may_ come to late, in which case we may have to go\n>> incremental, but I usually try to leave at least a few days between\n>> the time I mark a topic as \"will merge\" and the time I actually do\n>> the merge, for this exact reason.\n>\n> Awesome, thanks for the update.\n>\n> i'll roll a v4 with the last tweaks, hopefully that will be the last.\n\nThanks.\n"},{"id":"331939","messageId":"20171106211953.27910-2-anarcat@debian.org","threadId":"47063","inReplyTo":"20171106211953.27910-1-anarcat@debian.org","subject":"[PATCH v4 1/7] remote-mediawiki: add namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-06T21:19:47Z","receivedAt":"2017-11-06T21:20:05Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Kevin \n\nThis introduces a new remote.origin.namespaces argument that is a\nspace-separated list of namespaces. The list of pages extract is then\ntaken from all the specified namespaces.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex e7f857c1a..5ffb57595 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -63,6 +63,10 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.categories\"));\n chomp(@tracked_categories);\n \n+# Just like @tracked_categories, but for MediaWiki namespaces.\n+my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+chomp(@tracked_namespaces);\n+\n # Import media files on pull\n my $import_media = run_git(\"config --get --bool remote.${remotename}.mediaimport\");\n chomp($import_media);\n@@ -256,6 +260,23 @@ sub get_mw_tracked_categories {\n \treturn;\n }\n \n+sub get_mw_tracked_namespaces {\n+ my $pages = shift;\n+ foreach my $local_namespace (@tracked_namespaces) {\n+ my $mw_pages = $mediawiki->list( {\n+ action => 'query',\n+ list => 'allpages',\n+ apnamespace => get_mw_namespace_id($local_namespace),\n+ aplimit => 'max' } )\n+ || die $mediawiki->{error}->{code} . ': '\n+ . $mediawiki->{error}->{details} . \"\\n\";\n+ foreach my $page (@{$mw_pages}) {\n+ $pages->{$page->{title}} = $page;\n+ }\n+ }\n+ return;\n+}\n+\n sub get_mw_all_pages {\n \tmy $pages = shift;\n \t# No user-provided list, get the list of pages from the API.\n@@ -319,6 +340,10 @@ sub get_mw_pages {\n \t\t$user_defined = 1;\n \t\tget_mw_tracked_categories(\\%pages);\n \t}\n+\tif (@tracked_namespaces) {\n+\t\t$user_defined = 1;\n+\t\tget_mw_tracked_namespaces(\\%pages);\n+\t}\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n-- \n2.11.0\n\n"},{"id":"331940","messageId":"20171106211953.27910-4-anarcat@debian.org","threadId":"47063","inReplyTo":"20171106211953.27910-1-anarcat@debian.org","subject":"[PATCH v4 3/7] remote-mediawiki: show known namespace choices on failure","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-06T21:19:49Z","receivedAt":"2017-11-06T21:20:07Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"If we fail to find a requested namespace, we should tell the user\nwhich ones we know about, since those were already fetched. This\nallows users to fetch all namespaces by specifying a dummy namespace,\nfailing, then copying the list of namespaces in the config.\n\nEventually, we should have a flag that allows fetching all namespaces\nautomatically.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex a1d783789..6364d4e91 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -1334,7 +1334,8 @@ sub get_mw_namespace_id {\n \tmy $id;\n \n \tif (!defined $ns) {\n-\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n+\t\tmy @namespaces = map { s/ /_/g; $_; } sort keys %namespaces_id;\n+\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki, known namespaces: @namespaces\\n\";\n \t\t$ns = {is_namespace => 0};\n \t\t$namespace_id{$name} = $ns;\n \t}\n-- \n2.11.0\n\n"},{"id":"331941","messageId":"20171106211953.27910-8-anarcat@debian.org","threadId":"47063","inReplyTo":"20171106211953.27910-1-anarcat@debian.org","subject":"[PATCH v4 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-06T21:19:53Z","receivedAt":"2017-11-06T21:20:10Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Without this, the fetch process seems hanged while we fetch page\nlistings across the namespaces. Obviously, it should be possible to\nsilence this with -q, but that's an issue already present everywhere\nin the code and should be fixed separately:\n\nhttps://github.com/Git-Mediawiki/Git-Mediawiki/issues/30\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 7dccb44e0..fcdc29197 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -279,6 +279,7 @@ sub get_mw_tracked_namespaces {\n aplimit => 'max' } )\n || die $mediawiki->{error}->{code} . ': '\n . $mediawiki->{error}->{details} . \"\\n\";\n+ print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n foreach my $page (@{$mw_pages}) {\n $pages->{$page->{title}} = $page;\n }\n-- \n2.11.0\n\n"},{"id":"331942","messageId":"20171106211953.27910-6-anarcat@debian.org","threadId":"47063","inReplyTo":"20171106211953.27910-1-anarcat@debian.org","subject":"[PATCH v4 5/7] remote-mediawiki: support fetching from (Main) namespace","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-06T21:19:51Z","receivedAt":"2017-11-06T21:20:13Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"When we specify a list of namespaces to fetch from, by default the MW\nAPI will not fetch from the default namespace, refered to as \"(Main)\"\nin the documentation:\n\nhttps://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces\n\nI haven't found a way to address that \"(Main)\" namespace when getting\nthe namespace ids: indeed, when listing namespaces, there is no\n\"canonical\" field for the main namespace, although there is a \"*\"\nfield that is set to \"\" (empty). So in theory, we could specify the\nempty namespace to get the main namespace, but that would make\nspecifying namespaces harder for the user: we would need to teach\nusers about the \"empty\" default namespace. It would also make the code\nmore complicated: we'd need to parse quotes in the configuration.\n\nSo we simply override the query here and allow the user to specify\n\"(Main)\" since that is the publicly documented name.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 7f483180f..7a0824f31 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,7 +264,12 @@ sub get_mw_tracked_categories {\n sub get_mw_tracked_namespaces {\n my $pages = shift;\n foreach my $local_namespace (@tracked_namespaces) {\n- my $namespace_id = get_mw_namespace_id($local_namespace);\n+ my $namespace_id;\n+ if ($local_namespace eq \"(Main)\") {\n+ $namespace_id = 0;\n+ } else {\n+ $namespace_id = get_mw_namespace_id($local_namespace);\n+ }\n # virtual namespaces don't support allpages\n next if !defined($namespace_id) || $namespace_id < 0;\n my $mw_pages = $mediawiki->list( {\n-- \n2.11.0\n\n"},{"id":"331943","messageId":"20171106211953.27910-7-anarcat@debian.org","threadId":"47063","inReplyTo":"20171106211953.27910-1-anarcat@debian.org","subject":"[PATCH v4 6/7] remote-mediawiki: process namespaces in order","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-06T21:19:52Z","receivedAt":"2017-11-06T21:20:20Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Ideally, we'd process them in numeric order since that is more\nlogical, but we can't do that yet since this is where we find the\nnumeric identifiers in the first place. Lexicographic order is a good\ncompromise.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 7a0824f31..7dccb44e0 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -263,7 +263,7 @@ sub get_mw_tracked_categories {\n \n sub get_mw_tracked_namespaces {\n my $pages = shift;\n- foreach my $local_namespace (@tracked_namespaces) {\n+ foreach my $local_namespace (sort @tracked_namespaces) {\n my $namespace_id;\n if ($local_namespace eq \"(Main)\") {\n $namespace_id = 0;\n-- \n2.11.0\n\n"},{"id":"331944","messageId":"20171106211953.27910-5-anarcat@debian.org","threadId":"47063","inReplyTo":"20171106211953.27910-1-anarcat@debian.org","subject":"[PATCH v4 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-06T21:19:50Z","receivedAt":"2017-11-06T21:20:21Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Virtual namespaces do not correspond to pages in the database and are\nautomatically generated by MediaWiki. It makes little sense,\ntherefore, to fetch pages from those namespaces and the MW API doesn't\nsupport listing those pages.\n\nAccording to the documentation, those virtual namespaces are currently\n\"Special\" (-1) and \"Media\" (-2) but we treat all negative namespaces\nas \"virtual\" as a future-proofing mechanism.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 6364d4e91..7f483180f 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,10 +264,13 @@ sub get_mw_tracked_categories {\n sub get_mw_tracked_namespaces {\n my $pages = shift;\n foreach my $local_namespace (@tracked_namespaces) {\n+ my $namespace_id = get_mw_namespace_id($local_namespace);\n+ # virtual namespaces don't support allpages\n+ next if !defined($namespace_id) || $namespace_id < 0;\n my $mw_pages = $mediawiki->list( {\n action => 'query',\n list => 'allpages',\n- apnamespace => get_mw_namespace_id($local_namespace),\n+ apnamespace => $namespace_id,\n aplimit => 'max' } )\n || die $mediawiki->{error}->{code} . ': '\n . $mediawiki->{error}->{details} . \"\\n\";\n-- \n2.11.0\n\n"},{"id":"331945","messageId":"20171106211953.27910-1-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v4 0/7] remote-mediawiki: namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-06T21:19:46Z","receivedAt":"2017-11-06T21:20:23Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Hopefully, the final series. This includes only one more fix, from\nThomas, to remove an extra loop.\n\nThis should, alas, be ready to merge.\n\n"},{"id":"331946","messageId":"20171106211953.27910-3-anarcat@debian.org","threadId":"47063","inReplyTo":"20171106211953.27910-1-anarcat@debian.org","subject":"[PATCH v4 2/7] remote-mediawiki: allow fetching namespaces with spaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-06T21:19:48Z","receivedAt":"2017-11-06T21:20:24Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Ingo Ruhnke \n\nwe still want to use spaces as separators in the config, but we should\nallow the user to specify namespaces with spaces, so we use underscore\nfor this.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 5ffb57595..a1d783789 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -65,6 +65,7 @@ chomp(@tracked_categories);\n \n # Just like @tracked_categories, but for MediaWiki namespaces.\n my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+for (@tracked_namespaces) { s/_/ /g; }\n chomp(@tracked_namespaces);\n \n # Import media files on pull\n-- \n2.11.0\n\n"},{"id":"331979","messageId":"20171107070808.q7zz4i73mkffomcb@laptop.local","threadId":"47063","inReplyTo":"20171106211953.27910-3-anarcat@debian.org","subject":"Re: [PATCH v4 2/7] remote-mediawiki: allow fetching namespaces with spaces","fromName":"Thomas Adam","fromEmail":"thomas@xteddy.org","sentAt":"2017-11-07T07:08:08Z","receivedAt":"2017-11-07T07:08:17Z","isPatch":true,"sender":{"key":"thomas@xteddy.org","avatar":"https://gravatar.com/avatar/e7256db4738e501e5d2e84f00bb0bd99503165729573848a03330301fc2adc4a?d=mp&s=160"},"body":"On Mon, Nov 06, 2017 at 04:19:48PM -0500, Antoine Beaupré wrote:\n> From: Ingo Ruhnke \n> \n> we still want to use spaces as separators in the config, but we should\n> allow the user to specify namespaces with spaces, so we use underscore\n> for this.\n> \n> Reviewed-by: Antoine Beaupré \n> Signed-off-by: Antoine Beaupré \n> ---\n> contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n> 1 file changed, 1 insertion(+)\n> \n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> index 5ffb57595..a1d783789 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -65,6 +65,7 @@ chomp(@tracked_categories);\n> \n> # Just like @tracked_categories, but for MediaWiki namespaces.\n> my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n> +for (@tracked_namespaces) { s/_/ /g; }\n> chomp(@tracked_namespaces);\n\nDepending on the number if namespaces returned, it might be easier to convert\nthis to the following:\n\n my @tracked_namespaces = map {\n \tchomp; s/_/ /g; $_;\n } split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n\nThis would, once again, avoid creating @tracked_namespaces, and iterating over\nit.\n\nNote that this isn't about trying to 'golf' this; it's a performance\nconsideration.\n\nKindly,\nThomas Adam\n"},{"id":"331983","messageId":"20171107104526.ywnapdat57vi66xy@laptop","threadId":"47063","inReplyTo":"20171106211953.27910-4-anarcat@debian.org","subject":"Re: [PATCH v4 3/7] remote-mediawiki: show known namespace choices on failure","fromName":"Thomas Adam","fromEmail":"thomas@xteddy.org","sentAt":"2017-11-07T10:45:27Z","receivedAt":"2017-11-07T10:45:40Z","isPatch":true,"sender":{"key":"thomas@xteddy.org","avatar":"https://gravatar.com/avatar/e7256db4738e501e5d2e84f00bb0bd99503165729573848a03330301fc2adc4a?d=mp&s=160"},"body":"On Mon, Nov 06, 2017 at 04:19:49PM -0500, Antoine Beaupré wrote:\n> If we fail to find a requested namespace, we should tell the user\n> which ones we know about, since those were already fetched. This\n> allows users to fetch all namespaces by specifying a dummy namespace,\n> failing, then copying the list of namespaces in the config.\n> \n> Eventually, we should have a flag that allows fetching all namespaces\n> automatically.\n> \n> Reviewed-by: Antoine Beaupré \n> Signed-off-by: Antoine Beaupré \n> ---\n> contrib/mw-to-git/git-remote-mediawiki.perl | 3 ++-\n> 1 file changed, 2 insertions(+), 1 deletion(-)\n> \n> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n> index a1d783789..6364d4e91 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n> @@ -1334,7 +1334,8 @@ sub get_mw_namespace_id {\n> \tmy $id;\n> \n> \tif (!defined $ns) {\n> -\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n> +\t\tmy @namespaces = map { s/ /_/g; $_; } sort keys %namespaces_id;\n\nOops. This was my typo from my original suggestion. The hash is\n'%namespace_id', not '%namespaces_id'. However, how did this slip through\ntesting? I'm assuming you blindly copied this from my example, which although\nquick to do, is only being caught because of my sharp eyes...\n\n-- Thomas Adam\n"},{"id":"332002","messageId":"87efpa2i14.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"20171107070808.q7zz4i73mkffomcb@laptop.local","subject":"Re: [PATCH v4 2/7] remote-mediawiki: allow fetching namespaces with spaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:03:03Z","receivedAt":"2017-11-07T16:03:13Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-11-07 07:08:08, Thomas Adam wrote:\n> On Mon, Nov 06, 2017 at 04:19:48PM -0500, Antoine Beaupré wrote:\n>> From: Ingo Ruhnke \n>> \n>> we still want to use spaces as separators in the config, but we should\n>> allow the user to specify namespaces with spaces, so we use underscore\n>> for this.\n>> \n>> Reviewed-by: Antoine Beaupré \n>> Signed-off-by: Antoine Beaupré \n>> ---\n>> contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n>> 1 file changed, 1 insertion(+)\n>> \n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> index 5ffb57595..a1d783789 100755\n>> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n>> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> @@ -65,6 +65,7 @@ chomp(@tracked_categories);\n>> \n>> # Just like @tracked_categories, but for MediaWiki namespaces.\n>> my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n>> +for (@tracked_namespaces) { s/_/ /g; }\n>> chomp(@tracked_namespaces);\n>\n> Depending on the number if namespaces returned, it might be easier to convert\n> this to the following:\n>\n> my @tracked_namespaces = map {\n> \tchomp; s/_/ /g; $_;\n> } split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n>\n> This would, once again, avoid creating @tracked_namespaces, and iterating over\n> it.\n>\n> Note that this isn't about trying to 'golf' this; it's a performance\n> consideration.\n\nI'm not sure it's worth it. Mediawiki has only about 10 default\nnamespaces, and the user needs to specify them by hand here. I wouldn't\nbe concerned about the performance.\n\nA.\n\n-- \nEducation is the most powerful weapon which we can use to change the\nworld.\n - Nelson Mandela\n"},{"id":"332003","messageId":"20171107160701.24202-2-anarcat@debian.org","threadId":"47063","inReplyTo":"20171107160701.24202-1-anarcat@debian.org","subject":"[PATCH v5 1/7] remote-mediawiki: add namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:06:55Z","receivedAt":"2017-11-07T16:07:11Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Kevin \n\nThis introduces a new remote.origin.namespaces argument that is a\nspace-separated list of namespaces. The list of pages extract is then\ntaken from all the specified namespaces.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex e7f857c1a..5ffb57595 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -63,6 +63,10 @@ chomp(@tracked_pages);\n my @tracked_categories = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.categories\"));\n chomp(@tracked_categories);\n \n+# Just like @tracked_categories, but for MediaWiki namespaces.\n+my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+chomp(@tracked_namespaces);\n+\n # Import media files on pull\n my $import_media = run_git(\"config --get --bool remote.${remotename}.mediaimport\");\n chomp($import_media);\n@@ -256,6 +260,23 @@ sub get_mw_tracked_categories {\n \treturn;\n }\n \n+sub get_mw_tracked_namespaces {\n+ my $pages = shift;\n+ foreach my $local_namespace (@tracked_namespaces) {\n+ my $mw_pages = $mediawiki->list( {\n+ action => 'query',\n+ list => 'allpages',\n+ apnamespace => get_mw_namespace_id($local_namespace),\n+ aplimit => 'max' } )\n+ || die $mediawiki->{error}->{code} . ': '\n+ . $mediawiki->{error}->{details} . \"\\n\";\n+ foreach my $page (@{$mw_pages}) {\n+ $pages->{$page->{title}} = $page;\n+ }\n+ }\n+ return;\n+}\n+\n sub get_mw_all_pages {\n \tmy $pages = shift;\n \t# No user-provided list, get the list of pages from the API.\n@@ -319,6 +340,10 @@ sub get_mw_pages {\n \t\t$user_defined = 1;\n \t\tget_mw_tracked_categories(\\%pages);\n \t}\n+\tif (@tracked_namespaces) {\n+\t\t$user_defined = 1;\n+\t\tget_mw_tracked_namespaces(\\%pages);\n+\t}\n \tif (!$user_defined) {\n \t\tget_mw_all_pages(\\%pages);\n \t}\n-- \n2.11.0\n\n"},{"id":"332004","messageId":"20171107160701.24202-4-anarcat@debian.org","threadId":"47063","inReplyTo":"20171107160701.24202-1-anarcat@debian.org","subject":"[PATCH v5 3/7] remote-mediawiki: show known namespace choices on failure","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:06:57Z","receivedAt":"2017-11-07T16:07:14Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"If we fail to find a requested namespace, we should tell the user\nwhich ones we know about, since those were already fetched. This\nallows users to fetch all namespaces by specifying a dummy namespace,\nfailing, then copying the list of namespaces in the config.\n\nEventually, we should have a flag that allows fetching all namespaces\nautomatically.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex a1d783789..5e8845893 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -1334,7 +1334,8 @@ sub get_mw_namespace_id {\n \tmy $id;\n \n \tif (!defined $ns) {\n-\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n+\t\tmy @namespaces = map { s/ /_/g; $_; } sort keys %namespace_id;\n+\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki, known namespaces: @namespaces\\n\";\n \t\t$ns = {is_namespace => 0};\n \t\t$namespace_id{$name} = $ns;\n \t}\n-- \n2.11.0\n\n"},{"id":"332005","messageId":"20171107160701.24202-6-anarcat@debian.org","threadId":"47063","inReplyTo":"20171107160701.24202-1-anarcat@debian.org","subject":"[PATCH v5 5/7] remote-mediawiki: support fetching from (Main) namespace","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:06:59Z","receivedAt":"2017-11-07T16:07:16Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"When we specify a list of namespaces to fetch from, by default the MW\nAPI will not fetch from the default namespace, refered to as \"(Main)\"\nin the documentation:\n\nhttps://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces\n\nI haven't found a way to address that \"(Main)\" namespace when getting\nthe namespace ids: indeed, when listing namespaces, there is no\n\"canonical\" field for the main namespace, although there is a \"*\"\nfield that is set to \"\" (empty). So in theory, we could specify the\nempty namespace to get the main namespace, but that would make\nspecifying namespaces harder for the user: we would need to teach\nusers about the \"empty\" default namespace. It would also make the code\nmore complicated: we'd need to parse quotes in the configuration.\n\nSo we simply override the query here and allow the user to specify\n\"(Main)\" since that is the publicly documented name.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 611a04cd7..0e60b85c8 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,7 +264,12 @@ sub get_mw_tracked_categories {\n sub get_mw_tracked_namespaces {\n my $pages = shift;\n foreach my $local_namespace (@tracked_namespaces) {\n- my $namespace_id = get_mw_namespace_id($local_namespace);\n+ my $namespace_id;\n+ if ($local_namespace eq \"(Main)\") {\n+ $namespace_id = 0;\n+ } else {\n+ $namespace_id = get_mw_namespace_id($local_namespace);\n+ }\n # virtual namespaces don't support allpages\n next if !defined($namespace_id) || $namespace_id < 0;\n my $mw_pages = $mediawiki->list( {\n-- \n2.11.0\n\n"},{"id":"332006","messageId":"20171107160701.24202-7-anarcat@debian.org","threadId":"47063","inReplyTo":"20171107160701.24202-1-anarcat@debian.org","subject":"[PATCH v5 6/7] remote-mediawiki: process namespaces in order","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:07:00Z","receivedAt":"2017-11-07T16:07:30Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Ideally, we'd process them in numeric order since that is more\nlogical, but we can't do that yet since this is where we find the\nnumeric identifiers in the first place. Lexicographic order is a good\ncompromise.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 0e60b85c8..c9f46359b 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -263,7 +263,7 @@ sub get_mw_tracked_categories {\n \n sub get_mw_tracked_namespaces {\n my $pages = shift;\n- foreach my $local_namespace (@tracked_namespaces) {\n+ foreach my $local_namespace (sort @tracked_namespaces) {\n my $namespace_id;\n if ($local_namespace eq \"(Main)\") {\n $namespace_id = 0;\n-- \n2.11.0\n\n"},{"id":"332007","messageId":"20171107160701.24202-5-anarcat@debian.org","threadId":"47063","inReplyTo":"20171107160701.24202-1-anarcat@debian.org","subject":"[PATCH v5 4/7] remote-mediawiki: skip virtual namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:06:58Z","receivedAt":"2017-11-07T16:07:38Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Virtual namespaces do not correspond to pages in the database and are\nautomatically generated by MediaWiki. It makes little sense,\ntherefore, to fetch pages from those namespaces and the MW API doesn't\nsupport listing those pages.\n\nAccording to the documentation, those virtual namespaces are currently\n\"Special\" (-1) and \"Media\" (-2) but we treat all negative namespaces\nas \"virtual\" as a future-proofing mechanism.\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 5e8845893..611a04cd7 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -264,10 +264,13 @@ sub get_mw_tracked_categories {\n sub get_mw_tracked_namespaces {\n my $pages = shift;\n foreach my $local_namespace (@tracked_namespaces) {\n+ my $namespace_id = get_mw_namespace_id($local_namespace);\n+ # virtual namespaces don't support allpages\n+ next if !defined($namespace_id) || $namespace_id < 0;\n my $mw_pages = $mediawiki->list( {\n action => 'query',\n list => 'allpages',\n- apnamespace => get_mw_namespace_id($local_namespace),\n+ apnamespace => $namespace_id,\n aplimit => 'max' } )\n || die $mediawiki->{error}->{code} . ': '\n . $mediawiki->{error}->{details} . \"\\n\";\n-- \n2.11.0\n\n"},{"id":"332008","messageId":"20171107160701.24202-8-anarcat@debian.org","threadId":"47063","inReplyTo":"20171107160701.24202-1-anarcat@debian.org","subject":"[PATCH v5 7/7] remote-mediawiki: show progress while fetching namespaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:07:01Z","receivedAt":"2017-11-07T16:07:41Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Without this, the fetch process seems hanged while we fetch page\nlistings across the namespaces. Obviously, it should be possible to\nsilence this with -q, but that's an issue already present everywhere\nin the code and should be fixed separately:\n\nhttps://github.com/Git-Mediawiki/Git-Mediawiki/issues/30\n\nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex c9f46359b..af9cbc9d0 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -279,6 +279,7 @@ sub get_mw_tracked_namespaces {\n aplimit => 'max' } )\n || die $mediawiki->{error}->{code} . ': '\n . $mediawiki->{error}->{details} . \"\\n\";\n+ print {*STDERR} \"$#{$mw_pages} found in namespace $local_namespace ($namespace_id)\\n\";\n foreach my $page (@{$mw_pages}) {\n $pages->{$page->{title}} = $page;\n }\n-- \n2.11.0\n\n"},{"id":"332009","messageId":"20171107160701.24202-1-anarcat@debian.org","threadId":"47063","inReplyTo":"20171102212518.1601-1-anarcat@debian.org","subject":"[PATCH v5 0/7] namespace support","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:06:54Z","receivedAt":"2017-11-07T16:07:44Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"Yet another reroll to fix a typo.\n\n"},{"id":"332010","messageId":"20171107160701.24202-3-anarcat@debian.org","threadId":"47063","inReplyTo":"20171107160701.24202-1-anarcat@debian.org","subject":"[PATCH v5 2/7] remote-mediawiki: allow fetching namespaces with spaces","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:06:56Z","receivedAt":"2017-11-07T16:07:45Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"From: Ingo Ruhnke \n\nwe still want to use spaces as separators in the config, but we should\nallow the user to specify namespaces with spaces, so we use underscore\nfor this.\n\nReviewed-by: Antoine Beaupré \nSigned-off-by: Antoine Beaupré \n---\n contrib/mw-to-git/git-remote-mediawiki.perl | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\nindex 5ffb57595..a1d783789 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki.perl\n+++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n@@ -65,6 +65,7 @@ chomp(@tracked_categories);\n \n # Just like @tracked_categories, but for MediaWiki namespaces.\n my @tracked_namespaces = split(/[ \\n]/, run_git(\"config --get-all remote.${remotename}.namespaces\"));\n+for (@tracked_namespaces) { s/_/ /g; }\n chomp(@tracked_namespaces);\n \n # Import media files on pull\n-- \n2.11.0\n\n"},{"id":"332011","messageId":"87bmke2htf.fsf@curie.anarc.at","threadId":"47063","inReplyTo":"20171107104526.ywnapdat57vi66xy@laptop","subject":"Re: [PATCH v4 3/7] remote-mediawiki: show known namespace choices on failure","fromName":"Antoine Beaupré","fromEmail":"anarcat@debian.org","sentAt":"2017-11-07T16:07:40Z","receivedAt":"2017-11-07T16:07:49Z","isPatch":true,"sender":{"key":"anarcat@debian.org","avatar":"https://avatars.githubusercontent.com/u/796623?v=4"},"body":"On 2017-11-07 10:45:27, Thomas Adam wrote:\n> On Mon, Nov 06, 2017 at 04:19:49PM -0500, Antoine Beaupré wrote:\n>> If we fail to find a requested namespace, we should tell the user\n>> which ones we know about, since those were already fetched. This\n>> allows users to fetch all namespaces by specifying a dummy namespace,\n>> failing, then copying the list of namespaces in the config.\n>> \n>> Eventually, we should have a flag that allows fetching all namespaces\n>> automatically.\n>> \n>> Reviewed-by: Antoine Beaupré \n>> Signed-off-by: Antoine Beaupré \n>> ---\n>> contrib/mw-to-git/git-remote-mediawiki.perl | 3 ++-\n>> 1 file changed, 2 insertions(+), 1 deletion(-)\n>> \n>> diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> index a1d783789..6364d4e91 100755\n>> --- a/contrib/mw-to-git/git-remote-mediawiki.perl\n>> +++ b/contrib/mw-to-git/git-remote-mediawiki.perl\n>> @@ -1334,7 +1334,8 @@ sub get_mw_namespace_id {\n>> \tmy $id;\n>> \n>> \tif (!defined $ns) {\n>> -\t\tprint {*STDERR} \"No such namespace ${name} on MediaWiki.\\n\";\n>> +\t\tmy @namespaces = map { s/ /_/g; $_; } sort keys %namespaces_id;\n>\n> Oops. This was my typo from my original suggestion. The hash is\n> '%namespace_id', not '%namespaces_id'. However, how did this slip through\n> testing? I'm assuming you blindly copied this from my example, which although\n> quick to do, is only being caught because of my sharp eyes...\n\nI must admit I did not test that at all. Honestly, I'm just trying to\nfinalize this so we can move to GitHub and I can move on other\nthings. :)\n\nI rerolled with your fix.\n\nA.\n-- \nIf builders built houses the way programmers built programs,\nThe first woodpecker to come along would destroy civilization.\n - Gerald Weinberg\n"},{"id":"332059","messageId":"xmqqbmkdmskb.fsf@gitster.mtv.corp.google.com","threadId":"47063","inReplyTo":"20171107160701.24202-1-anarcat@debian.org","subject":"Re: [PATCH v5 0/7] namespace support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-08T02:07:32Z","receivedAt":"2017-11-08T02:07:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Antoine Beaupré writes:\n\n> Yet another reroll to fix a typo.\n\nThanks. Will replace. Let's wait for a few more days and then\nmerge it to 'next' and down to 'master'.\n\n"}]}