git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCHv4 06/14] run-command: add an asynchronous parallel child processor

From
Junio C Hamano <gitster@pobox.com>
Date
Sep 23, 2015, 19:34 UTC
Message-ID
<xmqq7fnhx74i.fsf@gitster.mtv.corp.google.com>
In-Reply-To
<xmqqlhbxxbai.fsf@gitster.mtv.corp.google.com>
Junio C Hamano <gitster@pobox.com> writes:
> You are running "git fetch" that are is a lot more heavy-weight.
> Because once each of them started fully they will be network bound,
> it is likely that you would want to run more processes than you have
> core.

I thought the conclusion would be obvious, but just in case the readers need the flow of thought completed, from the above it follows that on a N-core box (say 8-core) you may want to run the fetch with -j16 (or more). If we start everything at once, the time before one process starts to produce first meaningful response (I am not counting the "starting command for submodule-$i" message as "meaningful") would take twice as long under such condition even if you have infinite network bandwidth and talking to an infinitely fast server.

I agree that a full slow-start ramping-up is not necessary in order to hide the start-up lag. All you need to do is to start just one and make sure it becomes the foreground, and give it a time alone to make enough progress to produce an early output without getting slowed down by system activities caused by the other background processes in the group that start all at the same time. And once the foreground process starts showing its output, you can unleash the remainder of the herd to thunder and nobody would notice as they run all in the background. Again, I am assuming something like "git fetch" that takes reasonably large amount of resource and time to start-up and tear-down.

Previous: Junio C HamanoNext: Stefan Beller
Message 11 of 39 in “[PATCHv4 00/14] fetch submodules in parallel and a preview on parallel "submodule update"”
  1. Stefan BellerSep 23, 2015
  2. 01/14 submodule: Send "Fetching submodule <foo>" to standard errorStefan Beller, Sep 23, 2015
  3. 02/14 xread: poll on non blocking fdsStefan Beller, Sep 23, 2015
  4. 03/14 xread_nonblock: add functionality to read from fds without blockingStefan Beller, Sep 23, 2015
  5. 04/14 strbuf: add strbuf_read_once to read without blockingStefan Beller, Sep 23, 2015
  6. 05/14 run-command: factor out return value computationStefan Beller, Sep 23, 2015
  7. 06/14 run-command: add an asynchronous parallel child processorStefan Beller, Sep 23, 2015
  8. Junio C HamanoSep 23, 2015
  9. Stefan BellerSep 23, 2015
  10. Junio C HamanoSep 23, 2015
  11. Junio C HamanoSep 23, 2015
  12. Stefan BellerSep 23, 2015
  13. Junio C HamanoSep 23, 2015
  14. Junio C HamanoSep 23, 2015
  15. Junio C HamanoSep 23, 2015
  16. Junio C HamanoSep 23, 2015
  17. [PATCHv5] Another squash on run-command: add an asynchronous parallel child processorStefan Beller, Sep 23, 2015
  18. Junio C HamanoSep 24, 2015
  19. 0/2 Another squash on run-command: add an asynchronous parallel child processorStefan Beller, Sep 24, 2015
  20. 2/2 SQUASH for "fetch_populated_submodules: use new parallel job processing"Stefan Beller, Sep 24, 2015
  21. 1/2 SQUASH???Stefan Beller, Sep 24, 2015
  22. Junio C HamanoSep 25, 2015
  23. Junio C HamanoSep 25, 2015
  24. Stefan BellerSep 25, 2015
  25. Junio C HamanoSep 25, 2015
  26. Junio C HamanoSep 25, 2015
  27. Stefan BellerSep 25, 2015
  28. Junio C HamanoSep 25, 2015
  29. Stefan BellerSep 25, 2015
  30. Junio C HamanoSep 25, 2015
  31. 07/14 fetch_populated_submodules: use new parallel job processingStefan Beller, Sep 23, 2015
  32. 08/14 submodules: allow parallel fetching, add tests and documentationStefan Beller, Sep 23, 2015
  33. 09/14 submodule-config: Untangle logic in parse_configStefan Beller, Sep 23, 2015
  34. 10/14 submodule config: keep update strategy aroundStefan Beller, Sep 23, 2015
  35. 11/14 git submodule update: cmd_update_recursiveStefan Beller, Sep 23, 2015
  36. 12/14 git submodule update: cmd_update_cloneStefan Beller, Sep 23, 2015
  37. Junio C HamanoSep 23, 2015
  38. 13/14 git submodule update: cmd_update_fetchStefan Beller, Sep 23, 2015
  39. 14/14 Rewrite submodule update in CStefan Beller, Sep 23, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.