usage for git-p2p (was: git pack/unpack over bittorrent - works!)

On Sat, Sep 4, 2010 at 2:52 AM, Artur Skawina <art.08.09@gmail.com> wrote:

 i'm _so_ glad you asked :)  please note - for all of these i'm keenly
aware that GPG signing is required (and nicolas has pointed out that
you only need a 20-byte hash to be securely transferred by some OOB
mechanism)

 * distribution of large mailing lists and reduction of load on SMTP
servers by adding in a firebreak layer where the SMTP data goes into a
git repository first, then is extracted on the other side.  users may
choose to download the mailing list using git over a peer-to-peer
network and then use the exact same software that the mailing list
server is running, thus reducing the load on the web front-end.  if
you then choose also to allow users to GPG-sign commits to the same
git repository, you have the possibility of reducing the load on the
SMTP servers *as well*.  _and_ a group of users can operate in
"offline" mode whilst still being able to collaborate (by sharing the
git repo between themselves on an isolated LAN) _and_ when one of them
goes back "online", all their commits sync up, and the rest of the
world becomes aware of that group's offline discussion.

 * distributed wikis.  there are lots of wikis now using git: ikiwiki
is one, and moinmoin can also be configured to use git.  so, wikis
have already solved the "conflict" problem, in particular i know that
joey is a smart bunny and has solved conflict resolution in ikiwiki.
that makes it possible to simply turn an "ordinary" wiki into a
"peer-to-peer" wiki by merely inserting git-p2p underneath.

 * distributed bugtrackers.  the one that i know of which i believe
would work straight away is the ikiwiki "bug" plugin.  the rest i need
to investigate: i know of at least one that's based on git that i
haven't looked at for a loong while (ditrack) - but the principle is
very straightforward: create bugs by UUID, commit, distribute.  if you
want a "central" numbering scheme then all that's needed is for a
"central" service to allocate a "central" number - a symlink to the
UUID would do - and you're done.

 * of course there's source management for other projects other than
linux-2.6 :)

there's plenty more ideas out there, but the primary ones i'm interested in are the ones where "traditional" software development tools are being used which are utterly dependent on the client-server paradigm. for reasons i won't explain as it's not relevant to this technical discussion list, this is making me feel a bit twitchy [free software development being dependent on the client-server paradigm, that is].

l.

On 09/04/10 13:54, Luke Kenneth Casson Leighton wrote:

I'm not sure that git fits well any of the above; it may be convenient to reuse git as it already exists and can be more or less easily plugged in, but it's probably not the best backend choice. The one that might map relatively closely to git is wikis, but there i'm not sure something more scalable wouldn't be better (eg how would wikipedia map onto git..)

The one other use case for git-p2p that i can see is distributing the object store over a LAN. But it's really about a _remote_ object store, not so much a distributed one. IOW, having git-checkout trying to reach "/tmp/.git-unix/G0" and then any remote object store in addition to ".git/objects" would be very useful. But, do you actually want to spread the objects on _every_ developer desktop on the LAN? Wouldn't it much more make sense to have two or three dedicated "object servers", which would give you enough redundancy and, because of the higher chances of finding the required objects in cache, also perform better?

For the case of an ad-hoc group of developers, the protocol described in my previous email would work well, they only need to find one peer and then can move objects back and forth in the swarm.

One of the advantages that git-p2p could bring is better packing -- instead of everyone running git-repack with a limited window, each peer in the swarm could better use its resources by looking much deeper for the perfect delta candidate, but only for a subset of the objects, and then share the results with the rest. As a "git-repack -a -f" can eg shrink the gcc.git repo by ~15% (~100M) the savings could be noticeable.

artur

On Sat, Sep 4, 2010 at 4:08 PM, Artur Skawina <art.08.09@gmail.com> wrote:

 ok - then you should help with the "git hive" effort that casey's
writing, as he is focussing on writing an extension to git which does
exactly that.  the discussion on this thread is focussed on extending
git to work out on the wider internet, not just the case of optimised
transfers on a LAN segment.

l.

Luke Kenneth Casson Leighton wrote:

More accurately, I have entirely punted conflict resolution onto the underlying VCS, such as git, and taken advantage of its generally excellent merging behavior to avoid the most common classes of things that result in conflicts/locking on other wikis (like two users editing different parts of the same page concurrently).

Two branches of a wiki with committers to both can still of course result in conflicts. In this case, it might make sense to avoid conflict resolution during merging, and make the conflicted page be committed containing both versions and some sort of "conflict here!" warning directive. Then the wiki software can provide an web interface for cleaning up the conflict later. A custom merge driver could do something like that, although there's the problem that not all files in the wiki will be wiki pages, or the conflict could involve conflicting renames of pages. (Another approach would be to use the "ours" merge strategy.)