{"thread":{"id":"24905","subject":"git peer-to-peer project: info needed","startedAt":"2010-08-29T21:56:27Z","lastAt":"2010-08-30T21:22:03Z","messageCount":8,"participants":["Luke Kenneth Casson Leighton","Jonathan Nieder","Matthieu Moy","Casey Dahlin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"149244","messageId":"AANLkTi=xLJ4w1D4=p40f8nUQfLu5hpEfNORnjroffM1v@mail.gmail.com","threadId":"24905","inReplyTo":null,"subject":"git peer-to-peer project: info needed","fromName":"Luke Kenneth Casson Leighton","fromEmail":"luke.leighton@gmail.com","sentAt":"2010-08-29T21:56:27Z","receivedAt":"2010-08-29T21:56:27Z","isPatch":false,"sender":{"key":"luke.leighton@gmail.com","avatar":null},"body":"hi folks,\n\n[please could you kindly cc on responses because i am subscribed with\n\"no mail\" set]\n\ni need some guidance on what i should be doing, to add peer-to-peer\nnetworking to \"git fetch\". i can take care of the peer-to-peer\nnetworking side: the bit i'm unsure about is what sequence of events\nare required to happen. i'm presently looking at the use of\nwalker_fetch in builtin-http-fetch.c which gives me some idea.\nhowever as i aim to implement this first in python not c, i need to be\nusing git via command-line only.\n\nperhaps... does anyone know of an implementation of \"git fetch\" in\nsay... shell-script (or perhaps in python) or just absolutely any\nlanguage _other_ than c?\n\ni believe... i believe i may need the git rev-list and git\npack-objects commands, would that be right? rev-list gives the list\nof commit revisions; pack-objects, if i can merely transfer the .pack\nfile and .index file over the peer-to-peer network, the job's half\ndone, am i right?\n\n(the only missing step is to have a way to find the list of objects at\nthe remote end - i know how to do that)\n\nfollowing that, i can do \"git unpack-objects\" on the .pack file(s)\n\nand... i've just done a test on that (manually, by running\ngit-pack-objects --thin --all --stdout and then git-unpack-objects)\n... um.... :) i'm missing the \"tag updating\", aren't i? the objects\nare now in the .git repo but they're \"hanging about\", as can be seen\nwith git fsck:\n\n$ git fsck\ndangling commit af87b49b9fbcae28ae19b86ca04af5bd4a9f6778\n\nso... my next step iis... ermm... update the head ref? err? :) i see\nthat gitpython is simply writing [the above ref] into\n.git/ref/heads/master or whereever is specified (must begin with refs)\nwhich is kinda cheating but perfectly reasonable... err.. but wait...\ni'm into what \"git-pull.sh\" is doing at this point, aren't i?\n\nsoo... i kinda don't have to worry about that bit, am i right? as\nlong as i get to complete the \"git-unpack-objects\" stage, the rest can\nbe handled by git-pull.sh, am i right?\n\nmuch obliged some answers so that i can get this done, as free\nsoftware, and provide people with a peer-to-peer distributed version\nof git.\n\nl.\n"},{"id":"149246","messageId":"20100829220654.GH1890@burratino","threadId":"24905","inReplyTo":"AANLkTi=xLJ4w1D4=p40f8nUQfLu5hpEfNORnjroffM1v@mail.gmail.com","subject":"Re: git peer-to-peer project: info needed","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-29T22:06:54Z","receivedAt":"2010-08-29T22:06:54Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Luke,\n\nLuke Kenneth Casson Leighton wrote:\n\n> i need some guidance on what i should be doing, to add peer-to-peer\n> networking to \"git fetch\". i can take care of the peer-to-peer\n> networking side: the bit i'm unsure about is what sequence of events\n> are required to happen.\n\nNot sure if this is the best answer or not, but you might enjoy\nlooking at the git-remote-helpers(7) feature and git-remote-testgit.py\nexample.\n\nAlso, in case it is useful: the old scripted \"git fetch\" is available\nas contrib/examples/git-fetch.sh.\n\nHope that helps,\nJonathan\n"},{"id":"149252","messageId":"AANLkTi=vcXX8n3NzGF51c=wPhgqushpwM4R7hR=O6QuU@mail.gmail.com","threadId":"24905","inReplyTo":"20100829220654.GH1890@burratino","subject":"Re: git peer-to-peer project: info needed","fromName":"Luke Kenneth Casson Leighton","fromEmail":"luke.leighton@gmail.com","sentAt":"2010-08-29T22:57:47Z","receivedAt":"2010-08-29T22:57:47Z","isPatch":false,"sender":{"key":"luke.leighton@gmail.com","avatar":null},"body":"On Sun, Aug 29, 2010 at 11:06 PM, Jonathan Nieder wrote:\n> Hi Luke,\n>\n> Luke Kenneth Casson Leighton wrote:\n>\n>> i need some guidance on what i should be doing, to add peer-to-peer\n>> networking to \"git fetch\".  i can take care of the peer-to-peer\n>> networking side: the bit i'm unsure about is what sequence of events\n>> are required to happen.\n>\n> Not sure if this is the best answer or not, but you might enjoy\n> looking at the git-remote-helpers(7) feature\n\n git-remote-helpers - Helper programs to interact with remote\n repositories\n\nokaay, i see where you're going with that one. thanks.\n\n> and git-remote-testgit.py\n> example.\n\n have to find that.\n\n> Also, in case it is useful: the old scripted \"git fetch\" is available\n> as contrib/examples/git-fetch.sh.\n\n ahh. ah _ha_. yes. excellent.\n\n btw, gitpython interestingly has a 4-line method for working out the\nlist of commits required to be obtained in order for the local copy to\ncatch up:\n\n repo_refs = self.git.rev_list(ref, '--').strip().splitlines()\n other_repo_refs = other_repo.git.rev_list(other_ref,\n'--').strip().splitlines()\n diff_refs = list(set(other_repo_refs) - set(repo_refs))\n return map(lambda ref: Commit.find_all(other_repo, ref,\nmax_count=1)[0], diff_refs)\n\nit's going to be a leeetle more tricky than that, over a p2p network,\nunless i cheat, and place the other_repo's rev-list into a meta-file\nwhich is also distributed over the p2p network.... yuk to that, on the\nbasis that long-living repos that's going to be a biiig file (would\nthat actually be the case? what's the largest \"git rev-list --all\"\nthat anyone's ever seen? yikes - on git itself it's 860k!) hmmm..\ngit rev {commit1}...{commit2} might give me what i need (and be\nequivalent to the above 4 python lines but it can be done by the\nremote end - that's still potentially a damn big amount of info.\nwell, i'm sure i'll cope :)\n\ni just want to get something working, damnit. there simply aren't\nenough free software client-server-paradigm-independent development\ntools around, and that's a situation that's causing me some concern.\n\nl.\n"},{"id":"149285","messageId":"vpq39twpp0e.fsf@bauges.imag.fr","threadId":"24905","inReplyTo":"AANLkTi=xLJ4w1D4=p40f8nUQfLu5hpEfNORnjroffM1v@mail.gmail.com","subject":"Re: git peer-to-peer project: info needed","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2010-08-30T06:56:33Z","receivedAt":"2010-08-30T06:56:33Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Luke Kenneth Casson Leighton writes:\n\n> hi folks,\n>\n> [please could you kindly cc on responses because i am subscribed with\n> \"no mail\" set]\n>\n> i need some guidance on what i should be doing, to add peer-to-peer\n> networking to \"git fetch\".\n\nThere have already been some attempts at a \"gittorrent\" mechanism.\nGoogle will tell you more about that.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"149302","messageId":"AANLkTi=Fr02G6u0tEhJvaZNhG=WGdQeJacH7XuJXkgaP@mail.gmail.com","threadId":"24905","inReplyTo":"vpq39twpp0e.fsf@bauges.imag.fr","subject":"Re: git peer-to-peer project: info needed","fromName":"Luke Kenneth Casson Leighton","fromEmail":"luke.leighton@gmail.com","sentAt":"2010-08-30T11:31:11Z","receivedAt":"2010-08-30T11:31:11Z","isPatch":false,"sender":{"key":"luke.leighton@gmail.com","avatar":null},"body":"On Mon, Aug 30, 2010 at 7:56 AM, Matthieu Moy\n wrote:\n> Luke Kenneth Casson Leighton writes:\n>\n>> hi folks,\n>>\n>> [please could you kindly cc on responses because i am subscribed with\n>> \"no mail\" set]\n>>\n>> i need some guidance on what i should be doing, to add peer-to-peer\n>> networking to \"git fetch\".\n>\n> There have already been some attempts at a \"gittorrent\" mechanism.\n> Google will tell you more about that.\n\n i know.\n\n it's best to assume that i've been following those, given that i\nwrote the advogato and slashdot articles which brought sam and jonas'\nefforts to peoples' attention, but for the sake of brevity in\ncontacting the git list i didn't want to mention that, so i apologise\nfor not mentioning that i'm aware of gittorrent.\n\n sam has already ruled out the bittorrent protocol as a means to\ncreate \"mirrorsync\". mirrorsync is, as it stands, a lower-level\nprotocol requiring the addition of DHT and other peer-to-peer\ninfrastucture (NAT-busting), and sam is designing mirrorsync to be\npart of git-daemon (i.e. it requires an HTTP port).\n\ni believe that the use of HTTP is a mistake, and i believe that a\nproper peer to peer git distribution protocol _requires_\nbittorrent-like features, in order to have a chance of success (i.e.\nbe \"simple\" enough to use i.e. _not_ require knowledge of firewall\nconfiguration etc.)\n\n so whilst this is all way outside of the scope of the git mailing\nlist, i'm describing the rough plan here in case anyone's interested:\nthe rough plan is to create a VFS layer into which i can then work\n\"pack objects\" into quotes torrents quotes, named by filename after\nthe SHA1 hash. the bittorrent protocol is perfectly capable of\nsupporting multiple files; thus it should not be too hard a job to rip\nout the hard-coded filesystem access in the bittornado source code -\nos.listdir, open(fname, \"r\"/\"w\"), osstat etc - and then redirect the\nfile/directory operations onto underlying git operations.\n\nl.\n"},{"id":"149351","messageId":"20100830173905.GA9955@fearengine.rdu.redhat.com","threadId":"24905","inReplyTo":"AANLkTi=xLJ4w1D4=p40f8nUQfLu5hpEfNORnjroffM1v@mail.gmail.com","subject":"Re: git peer-to-peer project: info needed","fromName":"Casey Dahlin","fromEmail":"cdahlin@redhat.com","sentAt":"2010-08-30T17:39:07Z","receivedAt":"2010-08-30T17:39:07Z","isPatch":false,"sender":{"key":"cdahlin@redhat.com","avatar":null},"body":"On Sun, Aug 29, 2010 at 10:56:27PM +0100, Luke Kenneth Casson Leighton wrote:\n> hi folks,\n> \n> [please could you kindly cc on responses because i am subscribed with\n> \"no mail\" set]\n> \n> i need some guidance on what i should be doing, to add peer-to-peer\n> networking to \"git fetch\". i can take care of the peer-to-peer\n> networking side: the bit i'm unsure about is what sequence of events\n> are required to happen. i'm presently looking at the use of\n> walker_fetch in builtin-http-fetch.c which gives me some idea.\n> however as i aim to implement this first in python not c, i need to be\n> using git via command-line only.\n> \n\nI have a peer-to-peer git patchset that I'm already working on, which is\nnearly in a showable state. I've wanted to sand off a few more edges\nbefore a public review but I'll happily share it.\n\n--CJD\n"},{"id":"149353","messageId":"20100830174506.GB9955@fearengine.rdu.redhat.com","threadId":"24905","inReplyTo":"20100830173905.GA9955@fearengine.rdu.redhat.com","subject":"Re: git peer-to-peer project: info needed","fromName":"Casey Dahlin","fromEmail":"cdahlin@redhat.com","sentAt":"2010-08-30T17:45:07Z","receivedAt":"2010-08-30T17:45:07Z","isPatch":false,"sender":{"key":"cdahlin@redhat.com","avatar":null},"body":"On Mon, Aug 30, 2010 at 01:39:07PM -0400, Casey Dahlin wrote:\n> On Sun, Aug 29, 2010 at 10:56:27PM +0100, Luke Kenneth Casson Leighton wrote:\n> > hi folks,\n> > \n> > [please could you kindly cc on responses because i am subscribed with\n> > \"no mail\" set]\n> > \n> > i need some guidance on what i should be doing, to add peer-to-peer\n> > networking to \"git fetch\". i can take care of the peer-to-peer\n> > networking side: the bit i'm unsure about is what sequence of events\n> > are required to happen. i'm presently looking at the use of\n> > walker_fetch in builtin-http-fetch.c which gives me some idea.\n> > however as i aim to implement this first in python not c, i need to be\n> > using git via command-line only.\n> > \n> \n> I have a peer-to-peer git patchset that I'm already working on, which is\n> nearly in a showable state. I've wanted to sand off a few more edges\n> before a public review but I'll happily share it.\n> \n> --CJD\n\nOn further review you seem to be talking more about making fetch itself\npeer-to-peer, where my patchset is more of a peer-to-peer branch-sharing\nmechanism. Complementary but not necessarily conflicting. Still I'll\nhurry up and get it online :)\n\n--CJD\n"},{"id":"149363","messageId":"AANLkTinjow-aRmNB7rSgq17ww00csN82tWBM+6Nv350k@mail.gmail.com","threadId":"24905","inReplyTo":"20100830174506.GB9955@fearengine.rdu.redhat.com","subject":"Re: git peer-to-peer project: info needed","fromName":"Luke Kenneth Casson Leighton","fromEmail":"luke.leighton@gmail.com","sentAt":"2010-08-30T21:22:03Z","receivedAt":"2010-08-30T21:22:03Z","isPatch":false,"sender":{"key":"luke.leighton@gmail.com","avatar":null},"body":"On Mon, Aug 30, 2010 at 6:45 PM, Casey Dahlin wrote:\n\n> On further review you seem to be talking more about making fetch itself\n> peer-to-peer,\n\n only on the basis that that's what i believed would be a good start,\nlacking any experience or advice :)\n\n> where my patchset is more of a peer-to-peer branch-sharing\n> mechanism.\n\n what's the difference? i did look at making \"blobs / trees\"\navailable but i figured that that would just get you a snapshot (not\nhelpful). i'm afraid i don't actually know the difference between\nfetch sharing and branch sharing, as my knowledge of git internals is\na bit \"swiss cheese\".\n\n\n> Complementary but not necessarily conflicting. Still I'll\n> hurry up and get it online :)\n\n commented already. was a bit of a bit message in the inbox, with an\ninline thingy rather than attachment, but i'll cope.\n\nl.\n"}]}