Gate between git and mediawiki : remote-helpers

Hi,

I'm a member of the team trying to establish a gate between mediawiki powered wiki and git.

We've tried several things which seems to work. However, it is something like git-svn and would require some commands such as git-mw to work. Is it recommended to use remote-helpers instead of that ?

There is one problem though : nobody wants to git clone the whole Wikipedia for instance. And we should add a functionality which allows the user to clone only a part of a wiki. Are partial clonings doable with remote helpers ?

Thank you for your help,

Best Regards

-- Claire Fousse

This mail referred to that previous one : http://article.gmane.org/gmane.comp.version-control.git/173991

2011/5/22 Claire Fousse <claire.fousse@gmail.com>:

Arnaud Lacurie

Arnaud Lacurie <arnaud.lacurie@gmail.com> writes:

I am not Matthew, but my gut feeling is that it largely depends on what you are interacting with, and how you envision the result to be used in a larger MediaWiki ecosystem.

In the case of SVN interoperability, there is an established community that exchanges their work via:

	svn checkout svn://some.where.xz/project
        svn update
	svn commit

and although people on the git side can already participate with:

	git svn init svn://some.where.xz/project
        git svn rebase
        git svn dcommit

it is understandable that they wish to be able to say:

	git clone svn://some.where.xz/project
        git pull -s rebase
        git push

to make it feel more similar to the native git experience. Now, even if there may be no "svn checkout/update/commit" equivalents in the workflow of established MediaWiki users, it may be nice to be able to work with:

	git clone --vcs=mediawiki http://some.where.xz/wiki/
        git pull
        git push

Then don't let them in your initial version. I do not see any problem in that. People can gain experience with smaller projects, like so:

	git clone --vcs=mediawiki https://git.wiki.kernel.org/
        git pull
        git push

When we need narrow (in the tree dimention) or shallow (in the history dimention) in either native or foreign transports, somebody would eventually add proper support. Currently we do not do "narrow" even for native transport (I have one cooking privately but it is progressing only slowly, and I think there may be others who are interested in it).

I would suggest not to be worried too much about narrow/shallow in your initial round.

Thank you for your advice.

2011/5/22 Junio C Hamano <gitster@pobox.com>:

Is vcs a standard argument (I can't see it anywhere) or do we have to change something in order to implement it?

Additionally, is there an example of protocol which uses remote-helpers? It seems that in the remote-helpers directory, there isn't anything but a git directory. Is there some other protocols than git that already use remote helpers?

Thank you very much.

Arnaud Lacurie <arnaud.lacurie@gmail.com> writes:

I don't know, but anyway, it could be

git clone mediawiki+http://some.where.xz/wiki

Hi, we are in a little bit of trouble trying to understand how remote helpers work and thus how to create our own. Trying to understand this, we created a git that we access through http on a local server on which we push files with

git remote-http http://login@localip/git push +master:master

The problem is we don't really understand which files are involved with remote helpers. We have perl scripts helping us to fetch and push files from mediawiki and we don't know how to implement them as a remote helper. Does anyone have some information regarding this ?

Thanks, Regards

-- Jérémie Nikaes Second year student at Ensimag

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:

I know it doesn't ;-)

As "clone" is a moral equivalent of "init" possibly followed by some auto-configuration followed by "fetch" and "checkout", it is a natural consequence of already supporting the "remove.<name>.vcs" configuration variable, and if we want to avoid the insane svn+http:// syntax, it is a logical thing to add such support.

Heya,

On Mon, May 23, 2011 at 07:14, Junio C Hamano <gitster@pobox.com> wrote:

We currently support either:

git clone mediawiki::http://some.where.xz/wiki

Or say:

git clone svn://example.com/some/svn/repo

Sverre Rabbelier <srabbelier@gmail.com> writes:

Ok, I forgot that we settled on that double-colon prefix.

Thanks for fact-checking.

Thank you for the advice regarding the project. Here is its status :

- We decided with your advice to use remote helpers, using the syntax
mediawiki::http://url.com (thanks to Sverre Rabbelier & Junio Hamano)
- We created a github at the following url
https://github.com/Bibzball/Git-Mediawiki
On it you will find
	* Our source code featuring
		- A start of a remote helper. The functions currently supported are
described on the wiki
		- The import part, for which we would like to thank Jeff King for
his initial perl code that we used and altered to realize the import
function of the remote helper
		- The export part is for now only a quick application of the
mediawiki API to export a file
	* A wiki with the basic documentation on our project
Feel free to consult it and give comments / ask questions

In order to maintain coherence between revisions (mediawiki) and commits (git), we have to store metadata somewhere. Since we want to keep transparency on the mediawiki's side, we want this data to be stored in git. Git notes seem to be a good answer here. They could contain the revision numbers that correspond to the commited files. Also, a commit with t files on git should be t revisions on mediawiki.

We identified a few issues :
* Conflicts :
	- A git user wants to push files
	- He is allowed to do so : no new revisions were made on mediawiki
	- The git user starts to push the files one by one, using the mediawiki API
	- A mediawiki user edits one of the files queued up in the for-push
list and saves his modifications
	-> The git user should be interrupted in his process. If not, the
changes of the mediawiki user will be overridden.
 	We still don't have a solution for this problem. Our guess is that
we should send each file in 2 steps.
a) get the timestamp of the last revision of this file from mediawiki
b) send the file, if allowed to, based on this timestamp

* A tree from mediawiki is necessarily linear. The tree from git can
have multiple branches. Let's have this example :
	- The mediawiki has been cloned on our side
	- We made changes and committed them
	- We want to pull changes from the mediawiki
2 solutions from here :
	a) Pull including merge
		Pros : The tree is saved on our side. If someone wants to clone our
repository, he will keep this structure.
		Cons : Because of mediawiki's trees linearity, keeping branches on
our side forces us to pull again right after the push in order to
maintain coherance between mediawiki revisions and git commits.

	b) Pull -- rebase
		Pros : We are very close to mediawiki's structure, facilitating
synchronisation.
		Cons : Forbids other git users to clone my repository since some of
my branches might die after a rebase. This, in our opinion, kind of
goes against some of git's principles.

To you, which solution do you prefer ? Do you see any alternative ?

* We would like to have the alternative to clone only partially the
mediawiki (for example, only a few pages). Let's illustrate this one :
	- A user clones pages a, b and c
	- Later on, he wants to add another page d to his clone list
	-> The entire history/tree has to be remade as commits have to be
inserted in between others

Thank you for your support on this project. If any information reported was not clear enough, feel free to ask questions. Any contribution / suggestion is welcome.

Regards, The Git-Mediawiki team, Arnaud Lacurie, David Amouyal, Claire Fousse & Jeremie Nikaes.