Re: Resumable git clone?

From: Duy Nguyen <pclouds@gmail.com>
Date: Mar 2, 2016, 07:37 UTC
Message-ID: <CACsJy8DcNrOmrKKPibV6GuSqspovBmHzUv_mRB6fZyLjw5wWzQ@mail.gmail.com>
In-Reply-To: <xmqq8u215r25.fsf@gitster.mtv.corp.google.com>

On Wed, Mar 2, 2016 at 1:31 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 26 quoted lines

> Al Viro <viro@ZenIV.linux.org.uk> writes:
>
>> FWIW, I wasn't proposing to recreate the remaining bits of that _pack_;
>> just do the normal pull with one addition: start with sending the list
>> of sha1 of objects you are about to send and let the recepient reply
>> with "I already have <set of sha1>, don't bother with those".  And exclude
>> those from the transfer.
>
> I did a quick-and-dirty unscientific experiment.
>
> I had a clone of Linus's repository that was about a week old, whose
> tip was at 4de8ebef (Merge tag 'trace-fixes-v4.5-rc5' of
> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace,
> 2016-02-22).  To bring it up to date (i.e. a pull about a week's
> worth of progress) to f691b77b (Merge branch 'for-linus' of
> git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs, 2016-03-01):
>
>     $ git rev-list --objects 4de8ebef..f691b77b1fc | wc -l
>     1396
>     $ git rev-parse 4de8ebef..f691b77b1fc |
>       git pack-objects --revs --delta-base-offset --stdout |
>       wc -c
>     2444127
>
> So in order to salvage some transfer out of 2.4MB, the hypothetical
> Al protocol would first have the upload-pack give 20*1396 = 28kB

It could be 10*1396 or less. If the server calculates the shortest unambiguous SHA-1 length (quite cheap on fully packed repo) and sends it to the client, the client can just sends short SHA-1 instead. It's racy though because objects are being added to the server and abbrev length may go up. But we can check ambiguity for all SHA-1 sent by client and ask for resend for ambiguous ones.

On my linux-2.6.git, 10 letters (so 5 bytes) are needed for unambiguous short SHA-1. But we can even go optimistic and ask the client for shorter SHA-1 with hope that resend won't be many.

> object names to fetch-pack; no matter how fetch-pack encodes its
> preference, its answer would be less than 28kB.  We would likely to
> design this part of the new protocol in line with the existing part
> and use textual object names, so let's round them up to 100kB.

-- 
Duy