Re: Resumable git clone?

> On Wed, Mar 02, 2016 at 03:22:17PM +0700, Duy Nguyen wrote:
>
>>> As a simple proposal, the server could send the list of hashes (in
>>> approximately the same order it would send the pack), the client could
>>> send back a bitmap where '0' means "send it" and '1' means "got that one
>>> already", and the client could compress that bitmap.  That gives you the
>>> RLE and similar without having to write it yourself.  That might not be
>>> optimal, but it would likely set a high bar with minimal effort.
>>
>> We have an implementation of EWAH bitmap compression, so compressing
>> is not a problem.
>>
>> But I still don't see why it's more efficient to have the server send
>> the hash list to the client. Assume you need to transfer N objects.
>> That direction makes you always send N hashes. But if the client sends
>> the list of already fetched objects, M, then M <= N. And we won't need
>> to send the bitmap. What did I miss?
>
> Right, I don't see what the point is in compressing the bitmap. The sha1
> list for a clone of linux.git is 87 megabytes. The return bitmap, even
> naively, is 500K. Unless you are trying to optimize for wildly
> asymmetric links.
>
> If the client just naively sends "here's what I have", then we know it
> can never be _more_ than 87 megabytes. And as a bonus, the longer the
> list is, the more we are saving (so at the moment you are sending 82MB,
> it's really worth it, because you do have 95% of the pack, which is
> worth amortizing).
>
> I'm still a little dubious that anything involving "send all the hashes"
> is going to be useful in practice, especially for something like the
> kernel (where you have tons of huge small objects that delta well). It
> would work better when you have gigantic objects that don't delta (so
> the cost of a sha1 versus the object size is way better), but then I
> think we'd do better to transfer all of the normal-sized bits up front,
> and then allow fetching the large stuff separately.
>
> -Peff
>