Re: git pack/unpack over bittorrent - works!

From: Luke Kenneth Casson Leighton <luke.leighton@gmail.com>
Date: Sep 2, 2010, 18:07 UTC
Message-ID: <AANLkTi=kO9USQYoTLQZyCRrjCHWRtPtd4S5EuFk4-gPv@mail.gmail.com>
In-Reply-To: <20100902155810.GB14508@sigill.intra.peff.net>

On Thu, Sep 2, 2010 at 4:58 PM, Jeff King <peff@peff.net> wrote:

Show 17 quoted lines

> pack-objects will reuse previously found deltas. So the deltas you have
> in your existing packs matter. The deltas you have in your existing
> packs depend on many things. At least:
>
>  1. Options you used when packing (e.g., --depth and --window).
>
>  2. Probably exactly _when_ you packed. You could find a good delta
>     from A to B. Later, object C comes into existence, and would
>     provide a better delta base for B. I don't think we will ever try A
>     against C, unless --no-reuse-delta is set.
>
>     You have a different pack than somebody who packed after A, B, and
>     C all existed.
>
>     In practice, this tends not to happen much because the best deltas
>     are usually going backwards in time to a previous version. But it
>     can happen.

 jeff, thanks for explaining (and to nicolas, i see, since beginning this)

 mrhmfffh.  just been reading Documentation/technical/pack-heuristics.txt.

 so... options include:

 * writing an alternative "canonical" pack-object algorithm.  i'm
inclined to select "git format-patch"! :)  but that would be the lazy
way....

 * taking the seeder's pack-objects as the "canonical" ones,
regardless.  cacheing of the results would, sadly, be virtually
unavoidable, given the situation (multi-threading etc.)

 * throw away bittorrent entirely as a transport mechanism.

 * force-feed one peer to be "the" provider of a particular given
pack.  doesn't matter whom you contact to _obtain_ a pack from, as
long as you solely and exclusively get the pack from that particular
peer.

 * slight improvement / variation on the above: if two peers just
coincidentally happen to create or have the same pack (as can be shown
by having the same SHA-1, and/or by having the same data in their
cache) then ta-daaa, you have a file-sharing network for that
particular pack.

i think.... i think i miiight be able, hmmm... i believe it would be possible to implement this last option by creating separate .torrents for packs (one each!). by splitting things down, so that pack objects are named as {ref}-{objref}-{SHA-1}.torrent and by providing a "top level" torrent which contains the refs/heads/* and the associate rev-list(s)... each set of rev-lists would have the SHA-1 of the pack-object that happened to be created (and shared) at that particular time, from that particular client: you then genuinely don't give a stuff about who has what, it's all the same, and...

hmmm, i feel a modification to / deviation from the bittorrent protocol coming on :)

l.