threads / discuss / 41137

Initial git clone behaviour

Subject: Initial git clone behaviour

## tl;dr

3 messages between Jan 6, 2016 and Jan 6, 2016.

replies: 2people: 2as markdown or json

Eric Curtin· Jan 6, 2016, 22:26 UTC · lore

Hi Guys,

I am not a git developer or a git expert but just a change I would love to see.

When I clone a really large repository (like the linux kernel for example) initially on a brand new machine, it can take quite a while before I can start working with the code.

Often I do a standard git clone:

git clone (name of repo)

Followed by a depth=1 clone in parallel, so I can get building and working with the code asap:

git clone --depth=1 (name of repo)

Could we change the default behavior of git so that we initially get all the current files quickly so that we can start working them and then getting the rest of the data? At least a user could get to work quicker this way. Any disadvantages of this approach? Maybe I am not the first to suggest something like this.

Junio C Hamano· Jan 6, 2016, 23:14 UTC · re: Eric Curtin · lore

Re: Initial git clone behaviour

On Wed, Jan 6, 2016 at 2:26 PM, Eric Curtin <ericcurtin17@gmail.com> wrote:

Show 14 quoted lines

>
> Often I do a standard git clone:
>
> git clone (name of repo)
>
> Followed by a depth=1 clone in parallel, so I can get building and
> working with the code asap:
>
> git clone --depth=1 (name of repo)
>
> Could we change the default behavior of git so that we initially get
> all the current files quickly so that we can start working them and
> then getting the rest of the data? At least a user could get to work
> quicker this way. Any disadvantages of this approach?

It would put more burden on a shared and limited resource (i.e. the server side).

For example, I just tried a depth=1 clone of Linus's repository from

  git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

which transferred ~150MB pack data to check out 52k files in 90 seconds.

On the other hand, a full clone transferred ~980MB pack data and it took 170 seconds to complete. You can already see that a full clone is highly optimized--it does not take even twice the time of getting the most recent checkout to grab 10 years worth of development (562k of commits).

This efficiency comes from some tradeoffs, and one of them is that not all the data necessary to check out the latest tree contents can be stored near the beginning of the pack data. So "we'll checkout the tip while the remainder of the data is still incoming" would not be a workable, unless you are willing to destroy the full-clone performance.

Eric Curtin· Jan 6, 2016, 23:24 UTC · re: Junio C Hamano · lore

Re: Initial git clone behaviour

On 6 January 2016 at 23:14, Junio C Hamano <gitster@pobox.com> wrote:

Show 35 quoted lines

> On Wed, Jan 6, 2016 at 2:26 PM, Eric Curtin <ericcurtin17@gmail.com> wrote:
>>
>> Often I do a standard git clone:
>>
>> git clone (name of repo)
>>
>> Followed by a depth=1 clone in parallel, so I can get building and
>> working with the code asap:
>>
>> git clone --depth=1 (name of repo)
>>
>> Could we change the default behavior of git so that we initially get
>> all the current files quickly so that we can start working them and
>> then getting the rest of the data? At least a user could get to work
>> quicker this way. Any disadvantages of this approach?
>
> It would put more burden on a shared and limited resource (i.e.
> the server side).
>
> For example, I just tried a depth=1 clone of Linus's repository from
>
>   git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>
> which transferred ~150MB pack data to check out 52k files in 90 seconds.
>
> On the other hand, a full clone transferred ~980MB pack data and it took
> 170 seconds to complete. You can already see that a full clone is highly
> optimized--it does not take even twice the time of getting the most recent
> checkout to grab 10 years worth of development (562k of commits).
>
> This efficiency comes from some tradeoffs, and one of them is that not
> all the data necessary to check out the latest tree contents can be stored
> near the beginning of the pack data. So "we'll checkout the tip while the
> remainder of the data is still incoming" would not be a workable, unless
> you are willing to destroy the full-clone performance.

Ok, my internet connection at home is pretty terrible then! I don't get nowhere near these timings. It takes over an hour to do a full clone from my house. And approx 30 mins for the depth=1 (approx, did not time it).

That all makes sense I guess, probably not a good idea to regress the full time performance for the sake of this use case. Was just a query really!