In my country government make connections unstable on purpose. Please add resume capability for commands like git clone
threads / discuss / 58575
[feature request] resume capability for users in enemies of Internet countries
Subject: [feature request] resume capability for users in enemies of Internet countries
Re: [feature request] resume capability for users in enemies of Internet countries
On 10/7/22 01:01, m wrote:
> In my country government make connections unstable on purpose. Please add resume capability for commands like git clone >
Bandwidth issue?
-- An old man doll... just what I always wanted! - Clara
Re: [feature request] resume capability for users in enemies of Internet countries
On Fri, Oct 07, 2022 at 08:44:09AM +0700, Bagas Sanjaya wrote:
Show 6 quoted lines
> On 10/7/22 01:01, m wrote: > > In my country government make connections unstable on purpose. Please add resume capability for commands like git clone > > > > Bandwidth issue? >
Bandwidth is one thing but the other thing is that git network operations require that the whole operation succeeds in one go.
If your connectivity is bad to the point that the TCP connection breaks you have downloaded a bunch of data that is AFAIK just thrown away when you retry.
It is difficult to know if that data would be useful in the future, and you cannot meaningfully 'resume' because the remote state might have changed in the meantine as well.
Further, this whole fetch operation is using a heuristic to fetch some data in the hope that it will be enough to reconstruct the history that is requested, and this has been wrong in some cases, too. Not very precise and reproducible hence hard to 'resume' as well.
Let's say that the git networking has been developed at and tuned for the 'first world' Internet, and may be problematic to use in net-wise backwater areas. And it would require non-trivial effort to change.
Thanks
Michal
Re: [feature request] resume capability for users in enemies of Internet countries
Michal Suchánek <msuchanek@suse.de> wrote:
Show 26 quoted lines
> On Fri, Oct 07, 2022 at 08:44:09AM +0700, Bagas Sanjaya wrote: > > On 10/7/22 01:01, m wrote: > > > In my country government make connections unstable on purpose. Please add resume capability for commands like git clone > > > > > > > Bandwidth issue? > > > Bandwidth is one thing but the other thing is that git network > operations require that the whole operation succeeds in one go. > > If your connectivity is bad to the point that the TCP connection breaks > you have downloaded a bunch of data that is AFAIK just thrown away when > you retry. > > It is difficult to know if that data would be useful in the future, and > you cannot meaningfully 'resume' because the remote state might have > changed in the meantine as well. > > Further, this whole fetch operation is using a heuristic to fetch some > data in the hope that it will be enough to reconstruct the history that > is requested, and this has been wrong in some cases, too. Not very > precise and reproducible hence hard to 'resume' as well. > > Let's say that the git networking has been developed at and tuned for > the 'first world' Internet, and may be problematic to use in net-wise > backwater areas. And it would require non-trivial effort to change.
Increased adoption of bundles would help, since `wget -c' and such would work nicely, but that puts the burden on hosts for extra storage.
Perhaps GIT_SMART_HTTP=0 and having dumb clones not throwaway incomplete xfers would be more transparent to hosters, but dumb HTTP tends to be slow even on good connections.
Re: [feature request] resume capability for users in enemies of Internet countries
On 07/10/2022 08:26, Eric Wong wrote:
Show 32 quoted lines
> Michal Suchánek <msuchanek@suse.de> wrote: >> On Fri, Oct 07, 2022 at 08:44:09AM +0700, Bagas Sanjaya wrote: >>> On 10/7/22 01:01, m wrote: >>>> In my country government make connections unstable on purpose. Please add resume capability for commands like git clone >>>> >>> Bandwidth issue? >>> >> Bandwidth is one thing but the other thing is that git network >> operations require that the whole operation succeeds in one go. >> >> If your connectivity is bad to the point that the TCP connection breaks >> you have downloaded a bunch of data that is AFAIK just thrown away when >> you retry. >> >> It is difficult to know if that data would be useful in the future, and >> you cannot meaningfully 'resume' because the remote state might have >> changed in the meantine as well. >> >> Further, this whole fetch operation is using a heuristic to fetch some >> data in the hope that it will be enough to reconstruct the history that >> is requested, and this has been wrong in some cases, too. Not very >> precise and reproducible hence hard to 'resume' as well. >> >> Let's say that the git networking has been developed at and tuned for >> the 'first world' Internet, and may be problematic to use in net-wise >> backwater areas. And it would require non-trivial effort to change. > Increased adoption of bundles would help, since `wget -c' and such > would work nicely, but that puts the burden on hosts for extra storage. > > Perhaps GIT_SMART_HTTP=0 and having dumb clones not throwaway > incomplete xfers would be more transparent to hosters, but dumb > HTTP tends to be slow even on good connections.
There is work going on by Stollee (cc'd) on Bundle-URIs that look to split up repository serving into bite sized chunk, so may be worth looking at. https://lore.kernel.org/git/pull.1248.v4.git.1660050761.gitgitgadget@gmail.com/ : [PATCH v4 0/2] bundle URIs: design doc
Also discussed at the recent Contributor's Summit https://lore.kernel.org/git/YzXvwv%2FzK5AjhVvV@nand.local/ : [TOPIC 1/8] Bundle URIs
-- Philip
Re: [feature request] resume capability for users in enemies of Internet countries
On 10/7/22 6:03 AM, Philip Oakley wrote:
> On 07/10/2022 08:26, Eric Wong wrote:
Show 13 quoted lines
>> Perhaps GIT_SMART_HTTP=0 and having dumb clones not throwaway >> incomplete xfers would be more transparent to hosters, but dumb >> HTTP tends to be slow even on good connections. > > There is work going on by Stollee (cc'd) on Bundle-URIs that look to > split up repository serving into bite sized chunk, so may be worth > looking at. > https://lore.kernel.org/git/pull.1248.v4.git.1660050761.gitgitgadget@gmail.com/ > : [PATCH v4 0/2] bundle URIs: design doc > > Also discussed at the recent Contributor's Summit > https://lore.kernel.org/git/YzXvwv%2FzK5AjhVvV@nand.local/ : [TOPIC 1/8] > Bundle URIs
Thanks, Philip.
The bundle URIs feature should make it possible to help here.
One is that we could break the clone into smaller pieces, but I think that most bundle providers will still have at least one bundle with "most" of the repo. The real payoff is that we should be able to have resumable downloads for those bundles (pending some implementation details, but I'm waiting for the rest of the series to land first).
Of course, this all depends on the host providing bundles and advertising them, but we will get to that when we can.
Thanks, -Stolee