Re: Resumable clone/Gittorrent (again)

From: Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date: Jan 9, 2011, 17:48 UTC
Message-ID: <AANLkTimkDYCL7+N-Rno1-0p3Gy6o0wYrnuStV_n5k4Hk@mail.gmail.com>
In-Reply-To: <AANLkTinwb8orMBjcQjK0ogXd6rMEtRwT8SV41k8D3AXL@mail.gmail.com>

On Sun, Jan 9, 2011 at 8:55 PM, Luke Kenneth Casson Leighton <luke.leighton@gmail.com> wrote:

>  you still have to come up with a mapping from "chains" to "pieces".
> in the bittorrent protocol the mapping is done *entirely* implicitly
> and algorithmically.

Given a commit SHA-1, the mapping can be done algorithmically because the graph from the commit tip is fixed. Perhaps not mapping all at once, but as you have more pieces in the graph, you can map more.

Show 6 quoted lines

> the "meta" info in the .torrent contains
> filenames and file lengths.  stack the files one after the other in a
> big long data block, get a chopper and just go "whack, whack, whack"
> at regular piece-long points, that's your "pieces".  so, reassembly is
> a complete bitch, and picking just _one_ file to download rather than
> the whole lot becomes a total pain.

Well, there won't be .torrent files. Torrent files serve as checksums for file pieces (let's forget the tracker part). We do sha-1 checksum on every objects in git. The object graph without real content _is_ "info" part in .torrent files. Instead of passing around torrent files, I only need to pass around the sha-1 of the commit tip(s). That should be enough for any peer to discover the rest.

Reassembling, in its simplest way, is to just dump loose objects to $GIT_DIR/objects. But it's been six years since git's birth now, I'll pack them instead.

Show 11 quoted lines

>  that's hard work and you're now into "unproven" territory.  the
> successful R&D proof-of-concept code that i wrote i _deliberately_
> stayed away from "adapting" a proven bittorrent protocol, and as a
> result managed to get that proof-of-concept up and running within ...
> i think it was... 3 days.  most of the time was spent arseing about
> adding in a VFS layer into bittornado, in order to libratise it.
>
> i mention that just to give you something to think about.  if you're
> up to the challenge of writing your own p2p protocol, however, GREAT!
> you'll become a world expert on _both_ peer-to-peer protocols _and_
> git :)

Maybe I have gone insane ;) But I have another aim for this work: to adjust narrow clone area (pretty much path-based clones). So while it may not become real torrent for git (i.e p2p exchanging, depends on my needs), restartable clone from multiple sources is still worth it.

-- 
Duy