git peer-to-peer project: info needed

hi folks,

[please could you kindly cc on responses because i am subscribed with "no mail" set]

i need some guidance on what i should be doing, to add peer-to-peer networking to "git fetch". i can take care of the peer-to-peer networking side: the bit i'm unsure about is what sequence of events are required to happen. i'm presently looking at the use of walker_fetch in builtin-http-fetch.c which gives me some idea. however as i aim to implement this first in python not c, i need to be using git via command-line only.

perhaps... does anyone know of an implementation of "git fetch" in say... shell-script (or perhaps in python) or just absolutely any language _other_ than c?

i believe... i believe i may need the git rev-list and git pack-objects commands, would that be right? rev-list gives the list of commit revisions; pack-objects, if i can merely transfer the .pack file and .index file over the peer-to-peer network, the job's half done, am i right?

(the only missing step is to have a way to find the list of objects at the remote end - i know how to do that)

following that, i can do "git unpack-objects" on the .pack file(s)

and... i've just done a test on that (manually, by running git-pack-objects --thin --all --stdout and then git-unpack-objects) ... um.... :) i'm missing the "tag updating", aren't i? the objects are now in the .git repo but they're "hanging about", as can be seen with git fsck:

$ git fsck dangling commit af87b49b9fbcae28ae19b86ca04af5bd4a9f6778

so... my next step iis... ermm... update the head ref? err? :) i see that gitpython is simply writing [the above ref] into .git/ref/heads/master or whereever is specified (must begin with refs) which is kinda cheating but perfectly reasonable... err.. but wait... i'm into what "git-pull.sh" is doing at this point, aren't i?

soo... i kinda don't have to worry about that bit, am i right? as long as i get to complete the "git-unpack-objects" stage, the rest can be handled by git-pull.sh, am i right?

much obliged some answers so that i can get this done, as free software, and provide people with a peer-to-peer distributed version of git.

l.

Hi Luke,

Luke Kenneth Casson Leighton wrote:

Not sure if this is the best answer or not, but you might enjoy looking at the git-remote-helpers(7) feature and git-remote-testgit.py example.

Also, in case it is useful: the old scripted "git fetch" is available as contrib/examples/git-fetch.sh.

Hope that helps, Jonathan

On Sun, Aug 29, 2010 at 11:06 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

       git-remote-helpers - Helper programs to interact with remote
       repositories

okaay, i see where you're going with that one.  thanks.

 have to find that.

 ahh.  ah _ha_.  yes.  excellent.

 btw, gitpython interestingly has a 4-line method for working out the
list of commits required to be obtained in order for the local copy to
catch up:

        repo_refs = self.git.rev_list(ref, '--').strip().splitlines()
        other_repo_refs = other_repo.git.rev_list(other_ref,
'--').strip().splitlines()
        diff_refs = list(set(other_repo_refs) - set(repo_refs))
        return map(lambda ref: Commit.find_all(other_repo, ref,
max_count=1)[0], diff_refs)

it's going to be a leeetle more tricky than that, over a p2p network, unless i cheat, and place the other_repo's rev-list into a meta-file which is also distributed over the p2p network.... yuk to that, on the basis that long-living repos that's going to be a biiig file (would that actually be the case? what's the largest "git rev-list --all" that anyone's ever seen? yikes - on git itself it's 860k!) hmmm.. git rev {commit1}...{commit2} might give me what i need (and be equivalent to the above 4 python lines but it can be done by the remote end - that's still potentially a damn big amount of info. well, i'm sure i'll cope :)

i just want to get something working, damnit. there simply aren't enough free software client-server-paradigm-independent development tools around, and that's a situation that's causing me some concern.

l.

Luke Kenneth Casson Leighton <luke.leighton@gmail.com> writes:

There have already been some attempts at a "gittorrent" mechanism. Google will tell you more about that.

On Mon, Aug 30, 2010 at 7:56 AM, Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> wrote:

 i know.

 it's best to assume that i've been following those, given that i
wrote the advogato and slashdot articles which brought sam and jonas'
efforts to peoples' attention, but for the sake of brevity in
contacting the git list i didn't want to mention that, so i apologise
for not mentioning that i'm aware of gittorrent.

 sam has already ruled out the bittorrent protocol as a means to
create "mirrorsync".  mirrorsync is, as it stands, a lower-level
protocol requiring the addition of DHT and other peer-to-peer
infrastucture (NAT-busting), and sam is designing mirrorsync to be
part of git-daemon (i.e. it requires an HTTP port).

i believe that the use of HTTP is a mistake, and i believe that a proper peer to peer git distribution protocol _requires_ bittorrent-like features, in order to have a chance of success (i.e. be "simple" enough to use i.e. _not_ require knowledge of firewall configuration etc.)

 so whilst this is all way outside of the scope of the git mailing
list, i'm describing the rough plan here in case anyone's interested:
the rough plan is to create a VFS layer into which i can then work
"pack objects" into quotes torrents quotes, named by filename after
the SHA1 hash.  the bittorrent protocol is perfectly capable of
supporting multiple files; thus it should not be too hard a job to rip
out the hard-coded filesystem access in the bittornado source code -
os.listdir, open(fname, "r"/"w"), osstat etc - and then redirect the
file/directory operations onto underlying git operations.

l.

On Sun, Aug 29, 2010 at 10:56:27PM +0100, Luke Kenneth Casson Leighton wrote:

I have a peer-to-peer git patchset that I'm already working on, which is nearly in a showable state. I've wanted to sand off a few more edges before a public review but I'll happily share it.

--CJD

On Mon, Aug 30, 2010 at 01:39:07PM -0400, Casey Dahlin wrote:

On further review you seem to be talking more about making fetch itself peer-to-peer, where my patchset is more of a peer-to-peer branch-sharing mechanism. Complementary but not necessarily conflicting. Still I'll hurry up and get it online :)

--CJD

On Mon, Aug 30, 2010 at 6:45 PM, Casey Dahlin <cdahlin@redhat.com> wrote:

 only on the basis that that's what i believed would be a good start,
lacking any experience or advice :)

 what's the difference?  i did look at making "blobs / trees"
available but i figured that that would just get you a snapshot (not
helpful).  i'm afraid i don't actually know the difference between
fetch sharing and branch sharing, as my knowledge of git internals is
a bit "swiss cheese".

 commented already.  was a bit of a bit message in the inbox, with an
inline thingy rather than attachment, but i'll cope.

l.