# [PATCH] git-remote-mediawiki: bugfix for pages w/ >500 revisions 3 messages from 2013-09-22 to 2013-09-23. Participants: Benoit Person, Matthieu Moy, Jonathan Nieder. Thread: https://gitlist.dev/t/34995 ## Benoit Person, 2013-09-22 18:44 Subject: [PATCH] git-remote-mediawiki: bugfix for pages w/ >500 revisions Message-ID: <1379875468-24952-1-git-send-email-benoit.person@gmail.fr> URL: https://gitlist.dev/e/1379875468-24952-1-git-send-email-benoit.person%40gmail.fr ``` Mediawiki introduced a new API for queries w/ more than 500 results in version 1.21. That change triggered an infinite loop while cloning a mediawiki with such a page. Fix that while still preserving the old behavior for old APIs. Signed-off-by: Benoit Person Reported-by: Benjamin Cathey --- Patch tested for all mediawiki versions from 1.19 to 1.21. For now, if the tests suite is run without the fix, the new test introduces an infinite loop. I am not sure if this should be handled ? (a timeout of some kind maybe ?) contrib/mw-to-git/git-remote-mediawiki.perl | 14 ++++++++++++-- contrib/mw-to-git/t/t9365-continuing-queries.sh | 24 ++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 2 deletions(-) create mode 100755 contrib/mw-to-git/t/t9365-continuing-queries.sh diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl index c9a4805..2d7af57 100755 --- a/contrib/mw-to-git/git-remote-mediawiki.perl +++ b/contrib/mw-to-git/git-remote-mediawiki.perl @@ -625,6 +625,9 @@ sub fetch_mw_revisions_for_page { rvstartid => $fetch_from, rvlimit => 500, pageids => $id, + + # let the mediawiki knows that we support the latest API + continue => '', }; my $revnum = 0; @@ -640,8 +643,15 @@ sub fetch_mw_revisions_for_page { push(@page_revs, $page_rev_ids); $revnum++; } - last if (!$result->{'query-continue'}); - $query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid}; + + if ($result->{'query-continue'}) { # For legacy APIs + $query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid}; + } elsif ($result->{continue}) { # For newer APIs + $query->{rvstartid} = $result->{continue}->{rvcontinue}; + $query->{continue} = $result->{continue}->{continue}; + } else { + last; + } } if ($shallow_import && @page_revs) { print {*STDERR} " Found 1 revision (shallow import).\n"; diff --git a/contrib/mw-to-git/t/t9365-continuing-queries.sh b/contrib/mw-to-git/t/t9365-continuing-queries.sh new file mode 100755 index 0000000..6fb5df4 --- /dev/null +++ b/contrib/mw-to-git/t/t9365-continuing-queries.sh @@ -0,0 +1,24 @@ +#!/bin/sh + +test_description='Test the Git Mediawiki remote helper: queries w/ more than 500 results' + +. ./test-gitmw-lib.sh +. ./push-pull-tests.sh +. $TEST_DIRECTORY/test-lib.sh + +test_check_precond + +test_expect_success 'creating page w/ >500 revisions' ' + wiki_reset && + for i in $(seq 1 501) + do + echo "creating revision $i" + wiki_editpage foo "revision $i
" true + done +' + +test_expect_success 'cloning page w/ >500 revisions' ' + git clone mediawiki::'"$WIKI_URL"' mw_dir +' + +test_done -- 1.8.4.GIT ``` ## Matthieu Moy, 2013-09-22 19:27 Subject: Re: [PATCH] git-remote-mediawiki: bugfix for pages w/ >500 revisions Message-ID: URL: https://gitlist.dev/e/vpqsiwwejxz.fsf%40anie.imag.fr In-Reply-To: <1379875468-24952-1-git-send-email-benoit.person@gmail.fr> ``` Benoit Person writes: > Mediawiki introduced a new API for queries w/ more than 500 results in > version 1.21. That change triggered an infinite loop while cloning a > mediawiki with such a page. > > Fix that while still preserving the old behavior for old APIs. That would be nice to explain a bit more here. Where did the infinite loop come from? How does your patch fix it? > For now, if the tests suite is run without the fix, the new test > introduces an infinite loop. I am not sure if this should be handled ? > (a timeout of some kind maybe ?) If the patch fix this, then it's not a really big problem. The test failure is an infinite loop. That would be problematic if ran non-interactively, but I think it's Ok since we only run the testsuite manually. > diff --git a/contrib/mw-to-git/git-remote-mediawiki.perl b/contrib/mw-to-git/git-remote-mediawiki.perl > index c9a4805..2d7af57 100755 > --- a/contrib/mw-to-git/git-remote-mediawiki.perl > +++ b/contrib/mw-to-git/git-remote-mediawiki.perl > @@ -625,6 +625,9 @@ sub fetch_mw_revisions_for_page { > rvstartid => $fetch_from, > rvlimit => 500, > pageids => $id, > + > + # let the mediawiki knows that we support the latest API s/knows/know/ > + continue => '', Indentation with spaces. Please, use tabs. -- Matthieu Moy http://www-verimag.imag.fr/~moy/ ``` ## Jonathan Nieder, 2013-09-23 18:26 Subject: Re: [PATCH] git-remote-mediawiki: bugfix for pages w/ >500 revisions Message-ID: <20130923182601.GC9464@google.com> URL: https://gitlist.dev/e/20130923182601.GC9464%40google.com In-Reply-To: ``` Matthieu Moy wrote: > Benoit Person writes: >> For now, if the tests suite is run without the fix, the new test >> introduces an infinite loop. I am not sure if this should be handled ? >> (a timeout of some kind maybe ?) > > If the patch fix this, then it's not a really big problem. The test > failure is an infinite loop. Yes, I think it's fine. > That would be problematic if ran > non-interactively, but I think it's Ok since we only run the testsuite > manually. Some distros (e.g., Debian) occasionally do run the testsuite automatically, but it is still fine since they have a timeout that varies by platform to detect if the test has stalled. I suppose ideally git's test harness could learn to do the same thing some day, but for now it's easier one level above since an appropriate timeout depends on the speed on the platform, what else is creating load on the test machine, and other factors that are probably not easy for us to guess. (other tweaks snipped) Thanks, both. ```