Re: faster git clone

From: Emily Shaffer <emilyshaffer@google.com>
Date: Jan 22, 2021, 19:53 UTC
Message-ID: <CAJoAoZkrYYz=1wKDtUKdewPGX9wr2Zwhhyq9kd5C2_KDn9UJ=w@mail.gmail.com>
In-Reply-To: <20210122030103.GA73465@gmail.com>

On Thu, Jan 21, 2021 at 7:01 PM William Chen <williamchen32335@gmail.com> wrote:

>
> Dear Emily,
>
> I see your excellent contribution to git clone. I hope that you are well.

Hi William, this is a question much better directed at the Git list as a whole.

Show 11 quoted lines

>
> When I try to clone a repo of a large size from github, it is slow.
>
> $ git clone https://github.com/git/git
> ...
> remote: Enumerating objects: 56, done.
> remote: Counting objects: 100% (56/56), done.
> remote: Compressing objects: 100% (25/25), done.
> Receiving objects:  23% (70386/299751), 33.00 MiB | 450.00 KiB/s
>
> The following aria2c command, which can use multiple downloading threads, is much faster. Would you please let me know whether there is a way to speed up git clone (maybe by using parallelization)?

In general, it would be more compelling to see actual numbers than "much faster", e.g. the outputs of `time git clone https://github.com/git/git` and `time aria2c https://github.com/git/git/archive/master.zip` - or even an estimation from you, like, "I think clone takes a minute or two but aria does the same thing in only a couple of seconds". "Much faster" means something different to everyone :)

Show 8 quoted lines

>
> Your help is much appreciated! I look forward to hearing from you. Thanks.
>
> $ aria2c https://github.com/git/git/archive/master.zip
>
> 01/21 20:16:04 [NOTICE] Downloading 1 item(s)
>
> 01/21 20:16:04 [NOTICE] CUID#7 - Redirecting to https://codeload.github.com/git/git/zip/master

Right here it looks like your zip download redirects to a CDN or something, which is probably better optimized for serving archives than the Git server itself, so I would guess that has something to do with it too.

Show 10 quoted lines

> [#59b6a2 8.2MiB/0B CN:1 DL:3.8MiB]
> 01/21 20:16:08 [NOTICE] Download complete: /private/tmp/git-master.zip
>
> Download Results:
> gid   |stat|avg speed  |path/URI
> ======+====+===========+=======================================================
> 59b6a2|OK  |   2.9MiB/s|/private/tmp/git-master.zip
>
> Status Legend:
> (OK):download completed.

There are others on the list who are better able to explain this than me. But I'd guess the upshot is that 'git clone https://github.com/git/git' is asking a Git server, which is good at Git repo management (e.g. accepting pushes, generating packfiles to send you a specific object or branch, etc) - but when you ask for "git/git/archive/master.zip" you're getting the result of some work the Git server already did a while ago to zip up the current 'master' into an archive and give it to some other server.

We've done some other work[1] around enabling use of CDNs and prebuilt chunks lately, but again, there are others on the list better able to explain than me.

[1]: https://github.com/git/git/blob/master/Documentation/technical/packfile-uri.txt