Re: git pack/unpack over bittorrent - works!

From: Theodore Tso <tytso@mit.edu>
Date: Sep 3, 2010, 10:37 UTC
Message-ID: <B757A854-C7BF-4CBF-9132-91D205344606@mit.edu>
In-Reply-To: <alpine.LFD.2.00.1009021624170.19366@xanadu.home>

On Sep 2, 2010, at 7:09 PM, Nicolas Pitre wrote:

> The BitTorrent model is simply unappropriate for Git.  It doesn't fit to 
> the Git model at all as BitTorrent works on stable and static data, and 
> requires a lot of people wanting that same data.

I wonder if this would work.   Assume for the moment that once every N days (where N might be one month), the "Gittorrent master" generates a "canonical pack construction file" that contains information about the objects, their delta pairing, the version of diff and zlib required, etc., and the SHA1 of the expected resulting encoding of this pack file.   This would contain the canonical pack for the entire repository, and for the Linux kernel, it might be released on www.kernel.org, and contain all of the objects from the beginning of time to the latest commit in Linus Torvalds' repository.

Although everybody's git repository will have different packs that they use and store natively, by following the "canonical pack construction file" they will be able to build up a canonical encoding for the pack that can then be used to more quickly allow newbie developers who are pulling the full clone of Linus's tree for the first time to download using a peer2peer Bittorrent style download.   So people who are willing to participate as part of the peer2peer network can download the instructions for how to make the canonical pack once a month, and use it to create the canonical pack.  If the "Gittorrent master" has spent a lot of time to carefully compute the most efficient set of delta pairings, they will get the slight benefit of a more efficient pack which they could use instead of th
 eir local one without having to use large values of --window and --depth to "git repack".

This allows the peer2peer download to be used where it most matters --- for the bulk download for people who are cloning from the "canonical repository" for the first time.  After that, they will no doubt find it far more efficient to download incremental uploads using the git protocol.

-- Ted