{"thread":{"id":"55034","subject":"Re: faster git clone","startedAt":"2021-01-22T22:24:56Z","lastAt":"2021-01-23T00:43:05Z","messageCount":2,"participants":["Emily Shaffer","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"415010","messageId":"CAJoAoZkrYYz=1wKDtUKdewPGX9wr2Zwhhyq9kd5C2_KDn9UJ=w@mail.gmail.com","threadId":"55034","inReplyTo":"20210122030103.GA73465@gmail.com","subject":"Re: faster git clone","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-01-22T19:53:21Z","receivedAt":"2021-01-22T22:24:56Z","isPatch":false,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Jan 21, 2021 at 7:01 PM William Chen <williamchen32335@gmail.com> wrote:\n>\n> Dear Emily,\n>\n> I see your excellent contribution to git clone. I hope that you are well.\n\nHi William, this is a question much better directed at the Git list as a whole.\n\n>\n> When I try to clone a repo of a large size from github, it is slow.\n>\n> $ git clone https://github.com/git/git\n> ...\n> remote: Enumerating objects: 56, done.\n> remote: Counting objects: 100% (56/56), done.\n> remote: Compressing objects: 100% (25/25), done.\n> Receiving objects:  23% (70386/299751), 33.00 MiB | 450.00 KiB/s\n>\n> The following aria2c command, which can use multiple downloading threads, is much faster. Would you please let me know whether there is a way to speed up git clone (maybe by using parallelization)?\n\nIn general, it would be more compelling to see actual numbers than\n\"much faster\", e.g. the outputs of `time git clone\nhttps://github.com/git/git` and `time aria2c\nhttps://github.com/git/git/archive/master.zip` - or even an estimation\nfrom you, like, \"I think clone takes a minute or two but aria does the\nsame thing in only a couple of seconds\". \"Much faster\" means something\ndifferent to everyone :)\n\n>\n> Your help is much appreciated! I look forward to hearing from you. Thanks.\n>\n> $ aria2c https://github.com/git/git/archive/master.zip\n>\n> 01/21 20:16:04 [NOTICE] Downloading 1 item(s)\n>\n> 01/21 20:16:04 [NOTICE] CUID#7 - Redirecting to https://codeload.github.com/git/git/zip/master\n\nRight here it looks like your zip download redirects to a CDN or\nsomething, which is probably better optimized for serving archives\nthan the Git server itself, so I would guess that has something to do\nwith it too.\n\n> [#59b6a2 8.2MiB/0B CN:1 DL:3.8MiB]\n> 01/21 20:16:08 [NOTICE] Download complete: /private/tmp/git-master.zip\n>\n> Download Results:\n> gid   |stat|avg speed  |path/URI\n> ======+====+===========+=======================================================\n> 59b6a2|OK  |   2.9MiB/s|/private/tmp/git-master.zip\n>\n> Status Legend:\n> (OK):download completed.\n\nThere are others on the list who are better able to explain this than\nme. But I'd guess the upshot is that 'git clone\nhttps://github.com/git/git' is asking a Git server, which is good at\nGit repo management (e.g. accepting pushes, generating packfiles to\nsend you a specific object or branch, etc) - but when you ask for\n\"git/git/archive/master.zip\" you're getting the result of some work\nthe Git server already did a while ago to zip up the current 'master'\ninto an archive and give it to some other server.\n\nWe've done some other work[1] around enabling use of CDNs and prebuilt\nchunks lately, but again, there are others on the list better able to\nexplain than me.\n\n[1]: https://github.com/git/git/blob/master/Documentation/technical/packfile-uri.txt\n"},{"id":"415028","messageId":"YAtwuSoEE68TV4LN@camp.crustytoothpaste.net","threadId":"55034","inReplyTo":"CAJoAoZkrYYz=1wKDtUKdewPGX9wr2Zwhhyq9kd5C2_KDn9UJ=w@mail.gmail.com","subject":"Re: faster git clone","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-01-23T00:41:29Z","receivedAt":"2021-01-23T00:43:05Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-01-22 at 19:53:21, Emily Shaffer wrote:\n> On Thu, Jan 21, 2021 at 7:01 PM William Chen <williamchen32335@gmail.com> wrote:\n> > When I try to clone a repo of a large size from github, it is slow.\n> >\n> > $ git clone https://github.com/git/git\n> > ...\n> > remote: Enumerating objects: 56, done.\n> > remote: Counting objects: 100% (56/56), done.\n> > remote: Compressing objects: 100% (25/25), done.\n> > Receiving objects:  23% (70386/299751), 33.00 MiB | 450.00 KiB/s\n> >\n> > The following aria2c command, which can use multiple downloading threads, is much faster. Would you please let me know whether there is a way to speed up git clone (maybe by using parallelization)?\n> \n> In general, it would be more compelling to see actual numbers than\n> \"much faster\", e.g. the outputs of `time git clone\n> https://github.com/git/git` and `time aria2c\n> https://github.com/git/git/archive/master.zip` - or even an estimation\n> from you, like, \"I think clone takes a minute or two but aria does the\n> same thing in only a couple of seconds\". \"Much faster\" means something\n> different to everyone :)\n\nWhen Git shows the download speed, I believe it shows the speed at the\ngiven interval.  So it may be that at the given moment, performance\nvaries.  Part of that is server side, since Git performs compression of\ndata, and part of that is client side.  For instance, if you're using\nSHA-1-DC (which you should, for security), there is a theoretical\nperformance limit for clones of about 50 MiB/s on the client side which\nwould be improved if you were cloning a SHA-256 repository[0].\n\nIf this is a fresh clone, it is probably already cached on the server\nside, since frequently requested pack files are cached at GitHub\n(although I'm not clear on whether a given request is cached can be\ndetermined), so it could be that it's just pushing data over the pipe as\nfast as your system can process it.\n\n> > Your help is much appreciated! I look forward to hearing from you. Thanks.\n> >\n> > $ aria2c https://github.com/git/git/archive/master.zip\n> >\n> > 01/21 20:16:04 [NOTICE] Downloading 1 item(s)\n> >\n> > 01/21 20:16:04 [NOTICE] CUID#7 - Redirecting to https://codeload.github.com/git/git/zip/master\n> \n> Right here it looks like your zip download redirects to a CDN or\n> something, which is probably better optimized for serving archives\n> than the Git server itself, so I would guess that has something to do\n> with it too.\n\nThis is indeed backed by a CDN which may be much closer to you\nphysically.  Without seeing the full request, it's hard for me to say\nwhere your request was served from (CDN or not).  I should point out\nthat in this case you're cloning a single revision (so much less data),\nthe data is usually cached, and your end of the system is not performing\nany decompression or hash verification, so it may appear faster when\nyou're not performing equivalent work.\n\n(I would kindly ask that you not try to download every revision in\nhistory for a comparison, because that would be a clearly excessive and\nabusive level of usage.)\n\nI should also point out that you can't use multiple download threads to\ndownload from these endpoints because they don't, in general, handle\nRange requests.  (Basically, they do if they're already cached, but not\nif they're not.)\n\n> > [#59b6a2 8.2MiB/0B CN:1 DL:3.8MiB]\n> > 01/21 20:16:08 [NOTICE] Download complete: /private/tmp/git-master.zip\n> >\n> > Download Results:\n> > gid   |stat|avg speed  |path/URI\n> > ======+====+===========+=======================================================\n> > 59b6a2|OK  |   2.9MiB/s|/private/tmp/git-master.zip\n> >\n> > Status Legend:\n> > (OK):download completed.\n> \n> There are others on the list who are better able to explain this than\n> me. But I'd guess the upshot is that 'git clone\n> https://github.com/git/git' is asking a Git server, which is good at\n> Git repo management (e.g. accepting pushes, generating packfiles to\n> send you a specific object or branch, etc) - but when you ask for\n> \"git/git/archive/master.zip\" you're getting the result of some work\n> the Git server already did a while ago to zip up the current 'master'\n> into an archive and give it to some other server.\n\nIt's impossible for me to say definitively what the performance problem\nis in this case, but I don't think it's intrinsically Git if you're\nseeing less than a 50 MiB/s speed.  Git can and does process data at\nthat speed on the local system (and at 15 MiB/s on my local network over\nWi-Fi), so I'd guess that it's either a limitation in network\nperformance based on two different serving locations or perhaps a\ntemporarily overloaded server combined with the packfile not being\ncached.\n\n[0] I personally think 50 MiB/s is a very reasonable transfer speed, but\nsome people disagree.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"}]}