Re: Resumable clone/Gittorrent (again) - stable packs?

From: Jeff King <peff@peff.net>
Date: Jan 7, 2011, 05:22 UTC
Message-ID: <20110107052207.GA23128@sigill.intra.peff.net>
In-Reply-To: <alpine.LFD.2.00.1101062221480.22191@xanadu.home>

On Thu, Jan 06, 2011 at 11:33:51PM -0500, Nicolas Pitre wrote:

Show 16 quoted lines

> Here's what I suggest:
> 
> 	cd my_project
> 	BUNDLENAME=my_project_$(date "+%s").gitbundle
> 	git bundle create $BUNDLENAME --all
> 	maketorrent-console your_favorite_tracker $BUNDLENAME
> 
> Then start seeding that bundle, and upload $BUNDLENAME.torrent as 
> bundle.torrent inside my_project.git on your server.
> 
> Now... Git clients could be improved to first check for the availability 
> of the file "bundle.torrent" on the remote side, either directly in 
> my_project.git, or through some Git protocol extension.  Or even better, 
> the torrent hash could be stored in a Git ref, such as 
> refs/bittorrent/bundle and the client could use that to retrieve the 
> bundle.torrent file through some other means.

I really like the simplicity of this idea. It could even be generalized to handle more traditional mirrors, too. Just slice up the refs/mirrors namespace to provide different methods of fetching some initial set of objects. For example, upload-pack might advertise (in addition to the usual refs):

  refs/mirrors/bundle/torrent
  refs/mirrors/bundle/http
  refs/mirrors/fetch/git
  refs/mirrors/fetch/http

and the client can decide its preferred way of getting data: a bundle by http or by torrent, or connecting directly to some other git repository by git protocol or http. It would fetch the appropriate ref, which would contain a blob in some method-specific format. For torrent, it would be a torrent file. For the others, probably a newline-delimited set of URLs. You could also provide a torrent-magnet ref if you didn't even want to distribute the torrent file.

And no matter what the method used, at the end you have some set of refs and objects, and you can re-try your (now much smaller fetch). And there are a few obvious optimizations:

  1. When you get the initial set of refs from the master, remember
     them. If the mirror actually satisfies everything you were going to
     fetch, then you don't even have to reconnect for the final fetch.

  2. You can optionally cache the mirror list, and go straight to a
     mirror for future fetches instead of checking the master. This is
     only a reasonable thing to do if the mirrors are kept up to date,
     and provide good incremental access (i.e., only actual git-protocol
     mirrors, not torrent or http file).

-Peff