http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

On Wed, Sep 22, 2010 at 11:20 AM, Luke Kenneth Casson Leighton <luke.leighton@gmail.com> wrote:

 i trust that the above subject-line comment, or more specifically its
constituent links, begins to hint at why i believe it is important
that git-over-p2p be given a higher priority than it is at present.
you can easily join the dots to see where this government insanity is
going: it won't be long before this insanity is utilised to take
control of free software web sites just because they happen to be
hosting tools which MIGHT be utilised "for copyright infringment".
it's bad enough that the DMCA exists and can be used to intimidate
sourceforget into removing rtmpdump - now imagine if someone decided
to take control of sourceforget ok actually yeah maybe that's not so
great a loss after all but it's the PRINCIPLE damnit :)

 l.

On Sat, Nov 27, 2010 at 18:33, Luke Kenneth Casson Leighton <luke.leighton@gmail.com> wrote:

We give "priority" to the stuff people submit patches for. There isn't a git-over-p2p because nobody is working on it, would you like to work on it?

I'm also not convinced that this is actually needed. It's trivial to set up a p2p-like network by just emulating a darknet by manually adding remotes & fetching/pushing.

That's not viable for the stuff that usually gets distributed on p2p networks, but it sure is viable when you have people working on the same codebase. Which would be the use case for a git-over-p2p client, right?

On Sat, Nov 27, 2010 at 6:36 PM, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

 i was / have been.  unfortunately it's unfunded work, and, thanks to
receiving less money than i require for living and for feeding my
family i now face a court hearing on 16th december (3 weeks time)
which has the aim and ultimate goal of making us homeless.  i
therefore respectfull request that you please don't try to make me
feel like i _should_ be working on this in order to "earn your
respect" ( that goes for you too, johnathon ).

 ah.. but that's the single-use case for a single shared repository.
git "as-is" the fetch/push over http is limited to a single
repository.  i'm thinking more in terms of a global network, with
searches for checksummed objects "accidentally" potentially coming
from wildly different repositories.

 the work i did a few months ago showed that just "dropping" git on
top of e.g. the bittorrent protocol as-is, files/dirs mapped to git
objects, wouldn't cut the mustard, because the bloody bittorrent
protocol "chunks" system treats the entire fileset as an ordered but
contiguous "stream" of data (metadata in the actual .torrent file says
how long each of the files are and in what order they are, and this is
utilised at the receiving end to reconstruct the files after the
datablocks have been received.  it means that if you want just one
single file, even of 2 bytes in length, you might have to pull 2
blocks of e.g. 256k and grab the byte and the end of the 1st and the
byte at the beginning of the 2nd! yukk...)

but i believe that a system where each git object was given its own .torrent file _would_ work, and you utilise DHT or other mechanism (perhaps even a torrent containing the metadata!) to get the structural information. the success of this model depends on everyone being connected to everyone (a global network), in order to be able to do a DHT search for the node(s) that are seeding that git object's torrent file.

 the "global network" concept has the advantage of being far more
robust in the face of DNS take-overs, as you actually just need to
know _one_ DNS name out of potentially an unlimited number of DNS
names (or even an IP address or set of IP addresses) and you're joined
to other peers.

 codebase-in-a-git, wiki-in-a-git, website-in-a-git,
backups-of-filesystem-in-a-git (joey hess, one of the most amazing
debian developers around, has done all these things :)

yes it would... but i'm thinking well beyond that :)

at first glance the concept of git-over-darknet for example looks great. however it doesn't satisfy what i believe to be one of the key strategic requirements to be "many-pushers to many-pullers real-time resilient". lose access to the darknet, or even if the git server dies, what do you do?? if it's a darknet, you've got serious problems: you can't find out who it was!

git-over-p2p on the other hand _would_ be [m-p-t-m-p-r-t-r], because once the git-objects have been pulled by one client, they can be pulled from that client by other clients too. it's this feature of p2p distribution that is so distinct from the way in which git is normally described as "a distributed revision control system", and makes it a bugger to explain what the fuss is all about over this git-p2p stuff :)

 l.

Luke Kenneth Casson Leighton wrote:

Just for clarity: I never meant to imply such a thing. Just that messages like the one you just sent have quite an off-topic feeling and that this mailing list is better for design advice, usage advice, bug reporting, patch review, and similar things. Just my own opinion.

Kind regards, Jonathan

On Sat, Nov 27, 2010 at 20:06, Luke Kenneth Casson Leighton <luke.leighton@gmail.com> wrote:

I don't mean to discourage you. But as Jonathan points out this list is much better suited for submitting patches and discussing technical details.

I'd be happy to review your patches, I'm very interested in getting something like this myself. If you've been working on something I'd love to see it.

And I hope you'll do all-right in your personal life.

Right, but is there an actual use case for people who are developing code to use something like git over p2p? Maybe I'm just being unimaginative, but I can't see a case where people are working on the same project and can't find a way to push/pull from each other using the existing methods. Especially since it's easy to sign up for free Git hosting and use something like Tor to pull/push from there. Or to set up your own git HTTP server on a Tor *.onion server.

But of course using the Git data format for non-source-code would be very interesting, and much better for something like git-p2p.

Anyway, sorry about the brief reply. I'm not really knowledgable about this, and didn't fully grok/read your reply (going out).

On Sat, 2010-11-27 at 21:19 +0100, Ævar Arnfjörð Bjarmason wrote:

To me, the use-case wouldn't be because I /can't/ use existing methods,
it's because I /don't want to/ use existing methods :)
p2p tends to imply:
 - Resume-able downloads
 - Downloads from multiple sources at the same time
 - Transparently using an alternative source if one becomes unavailable
 - Load-balancing
 - referencing/linking based on "what it is" instead of "where it is"

The reason this stuff isn't a big itch for anybody is that these problems tend to come about when cloning, which doesn't really happen that often. When there's a new release, fetching the latest release-tag /might/ bring about the same needs, though in my experience "there's a new release" generally means nothing to me, since I check if there's a new release by running "git remote update" and looking for new tags, merging with my local patches if I see a relevant one.

I'm not convinced that any of this is something which is git's job, so much as it sounds like something which could benefit from another transport layer that git can optionally tie into. Sure, these could be developed together (p2p with git in mind), but adding a giant p2p section to the git codebase sounds like bloat.

I want my version control software to use p2p concepts for efficiency. I don't want my version control software to be a p2p client any more than I want my text-editor to be a mail client.

(In case it's not clear, I am in favour of p2p ("gittorrent")-like functionality being in git)

On Mon, Nov 29, 2010 at 5:12 PM, Will Palmer <wmpalmer@gmail.com> wrote:

I would add "automatically" and it's a real case for me. I work on the same project on different machines (each of them has different resources that I need from time to time). So I clone to all the machines. If I make a fix in one machine, other machines should automatically fetch it. It's like a private p2p network with a single user. If I add another machine to the network, all nodes should be aware of it, without me doing "git remote add" on each node.

Keep in mind that adding p2p concepts to something doesn't make it more efficient, in fact in most cases it makes it dramatically *LESS* efficient.

git-torrent like concepts have come up in the past, and I keep pointing out how and why they likely won't be useful. The biggest reason: there is no advantage for a client to stay in the cloud once they have their data. You can force this, sure, but clones are seldom and rare to begin with (as you mentioned) so there won't be a very large cloud to pull from to start with. With that in mind, I really see no advantage to p2p inside of the git core at all. It adds a lot of complexity for little gain.

Now you do mention things that would be useful:

- Ability to resume a clone that you only have a partial download for
(maybe just pack files?)
- Ability to include something like a 'meta-link' like list of
repositories to check for data (inferred from the multiple download
locations)

There are things we can learn from p2p, but I don't think adding it to git is actually useful.

Just my $0.02 though.

- John 'Warthog9' Hawley

On Mon, 2010-11-29 at 11:27 -0800, J.H. wrote:

As a simple use-case: Everyone comes into an office in the morning and runs "git remote update". This potentially causes a lot of traffic between the office and their offsite central repository. If this were a p2p scenario, the transfer from the offsite could potentially happen only once.

That counts as "more efficient", to me.

This is true of bittorrent as well: People stay in the cloud for altruistic reasons.

You're thinking p2p in terms of "every peer serves data, and keeps serving data. The network is more robust over time". I'm thinking p2p in terms of "Every peer has the ability to serve data. Adding a server is as trivial as adding a client."

The greatest advantage I can think of is "no existing server needs to agree to the addition of a new server", or at least "the addition of an existing server is accepted by convention, no questions asked"

I agree that git itself is not a good place to explore p2p concepts. I assume it would be much more useful to develop an independent p2p layer and allow git to somehow use that. And while there won't be "a large cloud", it's almost a guarantee that there would be more than the /one/ server currently available during clones or long fetches.

The biggest hurdle, I assume, would be "get git to talk to more than one source of data at once", even if one needs to set those up manually. If I understand correctly, packfiles are generated in a way which would not necessarily be consistent between clones.

--- Will Palmer <wmpalmer@gmail.com> wrote:

The same savings can be achieved by:

- Hosting the repository locally in the office;
- Automatically updating the 'official' location whenever something is
  pushed to the local repository;
- Having all developers use the office repository as their remote.

Takes about five minutes to set up.

(That does not invalidate your overall arguments, of course, but I don't have time to address them in much detail.)

-Jan