git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Multi-threaded 'git clone'

From
DLDavid Lang <david@lang.hm>
Date
Feb 16, 2015, 13:31 UTC
Message-ID
<alpine.DEB.2.02.1502160521030.23770@nftneq.ynat.uz>
In-Reply-To
<CACSCj9yoso1oLHzySx1F3O+DFAPiz-XEz1YNCEUMu1pj7KmX7w@mail.gmail.com>
On Mon, 16 Feb 2015, Koosha Khajehmoogahi wrote:
Show 6 quoted lines
> Cloning huge repositories like Linux kernel takes considerable amount
> of time. Is it possible to incorporate a multi-threaded simultaneous
> connections functionality for cloning? To what extent do we need to
> change the architecture of the current code and how large would be the
> scope of the work? That just seems an interesting idea to me and would
> liked to share it with the community.

They key question is what is it that takes the time in clonding and can that be multi-threaded.

If it's the netwrok traffic that takes the most time, where is the bottleneck?

Is it in the server software assembling what will be sent? Is it in the receiving software processing it? If so, multiple threads could help.

Is it in network bandwidth? If so doing multiple connections won't help much. TCP connections favour a few connections passing a lot of data rather than many connections passing a little. The one place where multiple connections can help is when you have non-congestion induced packet loss as a lost packet on a connection will cause the throughput of that connection to drop (if the drop is due to congestion, this is TCP working as designed, throttling back to match the available bandwidth). This can be a significant effect if you have a very high bandwidth, high latency connection (think multiple Gb on international connections), but for lower bandwidth connections it's much less of a factor. You can look at projects like bbcp

I think it's an interesting question to look at, but before you start looking at changing the architecture of the current code, I would suggest doing a bit more analisys of the problem to see if the bottleneck is really where you think it is.

First measure, then optimize :-)
David Lang
Previous: Koosha KhajehmoogahiNext: Jeff King
Message 2 of 9 in “Multi-threaded 'git clone'”
  1. Koosha KhajehmoogahiFeb 16, 2015
  2. David LangFeb 16, 2015
  3. Jeff KingFeb 16, 2015
  4. David LangFeb 16, 2015
  5. Jeff KingFeb 16, 2015
  6. Junio C HamanoFeb 16, 2015
  7. Shawn PearceFeb 17, 2015
  8. Duy NguyenFeb 16, 2015
  9. Jeff KingFeb 17, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.