Re: resume downloads

From: Junio C Hamano <gitster@pobox.com>
Date: May 10, 2015, 22:19 UTC
Message-ID: <CAPc5daVwxEniz-s-6dcowQkE-bK50wJ4MOCWGkAM=u02BGtN+w@mail.gmail.com>
In-Reply-To: <CACnwZYeW3+LQZYV4D_+vWggeh_yoWa-C3KdssHO21sRbsEKrMg@mail.gmail.com>

The current thinking is to model this after the "repo" tool. Prepare a reasonably up-to-date bundle file on the server side, add a protocol capability to advertise the URL to download that bundle from upload-pack, and have "git clone" to pay attention to it.

Then, a "git clone" could become:

 - If the capability advertises such a prebuilt bundle, spawn "curl"
   or "wget" internally to fetch it. This can be resumed when the
   connection goes down and will grab majority of the data necessary.

 - Extract the bundle into temporary area inside .git/refs/ to help
   the next step.

 - Internally do a "git fetch" to the original server. Thanks to the
   bundle transfer that has already happened, this step will become
   a small incremental update.

 - Then prune away the temporary .git/refs/ refs that were in the
   bundle, as these are not the up-to-date refs that exist on the
   server side.

A few points that need to be considered by whoever is doing this are:

 - Where to download the bundle, so that after killing "git clone"
   that is still in the bundle-download phase, the next invocation
   of "git clone" can notice and resume the bundle-download;

 - What kind of transfer protocols do we want to support? Is http
   and https from CDN sufficient? In other words, what exactly
   should the new capability say to point at the prebuilt bundle?

These (and probably there are several others) are not something that "repo" does not have to worry about, but would become issues when we try to fold this into "git clone".

On Sun, May 10, 2015 at 2:55 PM, Thiago Farina <tfransosi@gmail.com> wrote:

Show 14 quoted lines

> Hi,
>
> Is there links to discussion on this? I mean, is resume downloads a
> feature that is still being considered?
>
> Being able to download huge repos like WebKit, Linux, LibreOffice in
> small parts seems like a good feature to me.
>
> --
> Thiago Farina
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html