Stalled git cloning and possible solutions

> Quite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans what 
> been downloaded, and process needs re-start.
>
> Is there a way to recover or continue from already downloaded files during 
> cloning ?

> V.Krishn wrote:
> > Quite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans
> > what been downloaded, and process needs re-start.
> > 
> > Is there a way to recover or continue from already downloaded files
> > during cloning ?
> 
> No, sadly.  The pack sent for a clone is generated dynamically, so
> there's no easy way to support the equivalent of an HTTP Range request
> to resume.  Someone might implement an appropriate protocol extension
> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,
> but for now it doesn't exist.

> 
> What you *can* do today is create a bundle from the large repo
> somewhere with a reliable connection and then grab that using a
> resumable transport such as HTTP.  A kind person made a service to do
> that.
> 
>   http://thread.gmane.org/gmane.comp.version-control.git/181380

> On Friday, August 30, 2013 02:40:34 AM you wrote:
>> V.Krishn wrote:
>> > Quite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans
>> > what been downloaded, and process needs re-start.
>> > 
>> > Is there a way to recover or continue from already downloaded files
>> > during cloning ?
>> 
>> No, sadly.  The pack sent for a clone is generated dynamically, so
>> there's no easy way to support the equivalent of an HTTP Range request
>> to resume.  Someone might implement an appropriate protocol extension
>> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,
>> but for now it doesn't exist.
>
> This is what I tried but then realized something more is needed:
>
> During stalled clone avoid  Ctrl+c. 
> 1. Copy the content .i.e .git folder some other place.
> 2. cd <new dir>
> 3. git config fetch.unpackLimit 999999
> 4. git config transfer.unpackLimit 999999

> "V.Krishn" <vkrishn4@gmail.com> writes:
> > On Friday, August 30, 2013 02:40:34 AM you wrote:
> >> V.Krishn wrote:
> >> > Quite sometimes when cloning a large repo stalls, hitting Ctrl+c
> >> > cleans what been downloaded, and process needs re-start.
> >> > 
> >> > Is there a way to recover or continue from already downloaded files
> >> > during cloning ?
> >> 
> >> No, sadly.  The pack sent for a clone is generated dynamically, so
> >> there's no easy way to support the equivalent of an HTTP Range request
> >> to resume.  Someone might implement an appropriate protocol extension
> >> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,
> >> but for now it doesn't exist.
> > 
> > This is what I tried but then realized something more is needed:
> > 
> > During stalled clone avoid  Ctrl+c.
> > 1. Copy the content .i.e .git folder some other place.
> > 2. cd <new dir>
> > 3. git config fetch.unpackLimit 999999
> > 4. git config transfer.unpackLimit 999999
> 
> These two steps will not help, as negotiation between the sender and
> the receiver is based on the commits that are known to be complete,
> and an earlier failed "fetch" will not (and should not) update refs
> on the receiver's side.
> 
> >> What you *can* do today is create a bundle from the large repo
> >> somewhere with a reliable connection and then grab that using a
> >> resumable transport such as HTTP.
> 
> Yes.
> 
> Another possibility is, if the project being cloned has a tag (or a
> branch) that points at a commit back when it was smaller, do this
> 
> 	git init x &&
>         cd x &&
>         git fetch $that_repository
> $that_tag:refs/tags/back_then_i_was_small
> 
> to prime the object store of a temporary repository 'x' with a
> hopefully smaller transfer, and then use it as a "--reference"
> repository to the real clone.

> "V.Krishn" <vkrishn4@gmail.com> writes:
> > On Friday, August 30, 2013 02:40:34 AM you wrote:
> >> V.Krishn wrote:
> >> > Quite sometimes when cloning a large repo stalls, hitting Ctrl+c
> >> > cleans what been downloaded, and process needs re-start.
> >> > 
> >> > Is there a way to recover or continue from already downloaded files
> >> > during cloning ?
> >> 
> >> No, sadly.  The pack sent for a clone is generated dynamically, so
> >> there's no easy way to support the equivalent of an HTTP Range request
> >> to resume.  Someone might implement an appropriate protocol extension
> >> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,
> >> but for now it doesn't exist.
> > 
> > This is what I tried but then realized something more is needed:
> > 
> > During stalled clone avoid  Ctrl+c.
> > 1. Copy the content .i.e .git folder some other place.
> > 2. cd <new dir>
> > 3. git config fetch.unpackLimit 999999
> > 4. git config transfer.unpackLimit 999999
> 
> These two steps will not help, as negotiation between the sender and
> the receiver is based on the commits that are known to be complete,
> and an earlier failed "fetch" will not (and should not) update refs
> on the receiver's side.
> 
> >> What you *can* do today is create a bundle from the large repo
> >> somewhere with a reliable connection and then grab that using a
> >> resumable transport such as HTTP.
> 
> Yes.
> 
> Another possibility is, if the project being cloned has a tag (or a
> branch) that points at a commit back when it was smaller, do this
> 
> 	git init x &&
>         cd x &&
>         git fetch $that_repository
> $that_tag:refs/tags/back_then_i_was_small
> 
> to prime the object store of a temporary repository 'x' with a
> hopefully smaller transfer, and then use it as a "--reference"
> repository to the real clone.

> V.Krishn wrote:
>
>> Quite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans what
>> been downloaded, and process needs re-start.
>>
>> Is there a way to recover or continue from already downloaded files during
>> cloning ?
>
> No, sadly.  The pack sent for a clone is generated dynamically, so
> there's no easy way to support the equivalent of an HTTP Range request
> to resume.  Someone might implement an appropriate protocol extension
> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,
> but for now it doesn't exist.

> OK how about a new capability "resume" to upload-pack. fetch-pack can
> then send capability "resume[=<SHA-1>,<skip>]" to upload-pack. The
> first time it sends "resume" without parameters, and upload-pack will
> send back an SHA-1 to identify the pack being transferred together
> with a full pack as usual. When early disconnection happens, it sends
> the received SHA-1 and the received pack's size so far. It either
> receives the remaining part, or a full pack.
>
> When upload-pack gets "resume", it calculates a checksum of all input
> that may impact pack generation. If the checksum matches the SHA-1
> from fetch-pack, it'll continue to generate the pack as usual, but
> will skip sending the first <skip> bytes (maybe with a fake header so
> that fetch-pack realizes this is a partial pack). If the checksum does
> not match, it sends full pack again. I count on index-pack to spot
> corrupt resumed pack due to bugs.
>
> The input to calculate SHA-1 checksum includes:
>
>  - the result SHA-1 list from rev-list
>  - git version string
>  - .git/shallow
>  - replace object database
>  - pack.* config
>  - maybe some other variables (I haven't checked pack-objects)

> Another Git implementation can generate this SHA-1 in a totally
> different way and may even cache the generated pack.
>
> If at resume time, the load balancer directs the request to another
> upload-pack that generates this SHA-1 differently, ok this won't work
> (i.e. full pack is returned). In a busy repository, some refs may have
> moved so rev-list result at the resume time won't match any more, but
> we can deal with that later by relaxing to allow "want " lines with
> SHA-1 that are reachable from current refs, not just one of the refs
> (pack v4 or reachability bitmaps help).
> --
> Duy