Git transfer protocols (was: Re: Git-commits mailing list feed)

From: Mike Taht <mike.taht@timesys.com>
Date: Apr 23, 2005, 21:28 UTC
Message-ID: <426ABE1B.7000905@timesys.com>
In-Reply-To: <20050423204957.GA16751@delft.aura.cs.cmu.edu>

Jan Harkes wrote:

Show 7 quoted lines

> rsync works fine for now, but people are already looking at implementing
> smarter (more efficient) ways to synchronize git repositories by
> grabbing missing commits, and from there fetching any missing tree and
> file blobs. However there is no such linkage to discover missing tag
> objects, only a full rsync would be able to get them and for that it has
> to send the name of every object in the repository to the other side to
> check for any missing ones.

I think that one reason why rsync is inefficient for git is that it appears to need an acknowledgement after every file. (at least, that's what what the rhythm of the packets looked like when I sniffed it earlier, I don't know anything else about it) For a series of very small files this interacts badly with tcp's flow control mechanisms. Perhaps rsync could be modified for a "sliding file acknowledgement window".

Most "swarming protocols" (e.g BitTorrent, eDonkey) work well for one big file shared among multiple hosts, but poorly for lots of small files.

*Nothing* out there matches the simplicity of git's sha1 filename length... but

Something like robcast or fcast/flute might be of interest:

http://www.inrialpes.fr/planete/people/roca/mcl/mcl_in_short.html

Or one of the multicast netnews experiments:

"mcntp" http://mcntp.sourceforge.net/ "newscaster" http://www.dmn.tzi.org/en/newscaster.html

lastly, Monotone has it's own "netsync" protocol (via http://www.venge.net/monotone/faq.html)

"[netsync] is a bi-directional pipelined protocol for synchronizing collections using a tree of hashed indices. It allows any copy of monotone to function as either a client or a server, and rapidly synchronize or half-synchronize (push / pull) their database with another user. It is somewhat similar in flavor to rsync or Unison, in that it quickly and idempotently synchronizes information across the network without needing to store any local state; however, it is much more efficient than these protocols."

-- 
Mike Taht