Re: [RFC/PATCH] Added a remote helper to interact with mediawiki, pull & clone handled
- From
- Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
- Date
- Jun 2, 2011, 20:28 UTC
- Message-ID
- <BANLkTi=eYg3uT1hQZO03i4MLyhRkPzXK6w@mail.gmail.com>
- In-Reply-To
- <20110602170327.GA2928@sigill.intra.peff.net>
2011/6/2 Jeff King <peff@peff.net>:
Show 28 quoted lines
> On Thu, Jun 02, 2011 at 11:28:31AM +0200, Arnaud Lacurie wrote:
>
>> +sub mw_import {
>> [...]
>> + # Get 500 revisions at a time due to the mediawiki api limit
>> + while (1) {
>> + my $result = $mediawiki->api($query);
>> +
>> + # Parse each of those 500 revisions
>> + foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {
>> + my $page_rev_ids;
>> + $page_rev_ids->{pageid} = $page->{pageid};
>> + $page_rev_ids->{revid} = $revision->{revid};
>> + push (@revisions, $page_rev_ids);
>> + $revnum++;
>> + }
>> + last unless $result->{'query-continue'};
>> + $query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};
>> + print "\n";
>> + }
>
> What is this newline at the end here for? With it, my import reliably
> fails with:
>
> fatal: Unsupported command:
> fast-import: dumping crash report to .git/fast_import_crash_6091
>
> Removing it seems to make things work.Yes we actually found it today. It slipped as we've never fetched pages with more than 500 revisions since it got there...
Show 17 quoted lines
>> + # mediawiki revision number in the git note
>> + my $note_comment = encode_utf8("note added by git-mediawiki");
>> + my $note_comment_length = bytes::length($note_comment);
>> + my $note_content = encode_utf8("mediawiki_revision: " . $pagerevids->{revid} . "\n");
>> + my $note_content_length = bytes::length($note_content);
>> +
>> + if ($fetch_from == 1 && $n == 1) {
>> + print "reset refs/notes/commits\n";
>> + }
>> + print "commit refs/notes/commits\n";
>
> Should these go in refs/notes/commits? I don't think we have a "best
> practices" yet for the notes namespaces, as it is still a relatively new
> concept. But I always thought "refs/notes/commits" would be for the
> user's "regular" notes, and that programmatic things would get their own
> notes, like "refs/notes/mediawiki".
>That's a good idea, we didn't think notes could actually not go in refs/notes/commits. This will be perfect to distinguish the user notes from ours.
Show 14 quoted lines
>
>> + } else {
>> + print STDERR "You appear to have cloned an empty mediawiki\n";
>> + #What do we have to do here ? If nothing is done, an error is thrown saying that
>> + #HEAD is refering to unknown object 0000000000000000000
>> + }
>
> Hmm. We do allow cloning empty git repos. It might be nice for there to
> be some way for a remote helper to signal "everything OK, but the result
> is empty". But I think that is probably something that needs to be added
> to the remote-helper protocol, and so is outside the scope of your
> script (maybe it is as simple as interpreting the null sha1 as "empty";
> I dunno).
>Yes, that's a problem we've been running into. We didn't really know how to solve it.
Show 6 quoted lines
> Overall, it's looking pretty good. I like that I can resume a > half-finished import via "git fetch". Though I do have one complaint: > running "git fetch" fetches the metainfo for every revision of every > page, just as it does for an initial clone. Is there something in the > mediawiki API to say "show me revisions since N" (where N would be the > mediawiki revision of the tip of what we imported)?
I am not sure I understand your question. Because actually, we are supporting this, thanks to git notes. Like when you git fetch after a clone, it checks only the last revisions
Thank you very much for your help !
Arnaud Lacurie