git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCHv5] Another squash on run-command: add an asynchronous parallel child processor

From
Stefan Beller <sbeller@google.com>
Date
Sep 23, 2015, 23:41 UTC
Message-ID
<1443051711-15322-1-git-send-email-sbeller@google.com>
In-Reply-To
<xmqqpp19xbpz.fsf@gitster.mtv.corp.google.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---

I agree the commit 7087c5f3 (SQUASH??? on top of run-command: add an asynchronous parallel child processor) makes sense; I arrived at the same patch after adding in the feedback.

However I have the difference below, which I think could also be squashed in. when we operate under the assumption that the CPUs are the bottleneck in the case of parallelism.

 run-command.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/run-command.c b/run-command.c
index b6d8b39..e9722ff 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1107,8 +1107,13 @@ static void pp_collect_finished(struct parallel_processes *pp)
 	}
 }
 
-
-#define SPAWN_CAP (pp.max_processes + 1) /* spawn as many as possible */
+/*
+ * Throttle spawning children when starting the asynchronous processing,
+ * such that the first process to produce output is not slowed down to
+ * enable the fastest early feedback to the user.
+ */
+#define SPAWN_CAP (pp.max_processes + 1 < online_cpus() - 1 ? \
+		   pp.max_processes + 1 : online_cpus() - 1)
 
 int run_processes_parallel(int n, void *data,
 			   get_next_task_fn get_next_task,

-- 
2.5.0.273.g6fa2560.dirty
Previous: Junio C HamanoNext: Junio C Hamano
Message 17 of 39 in “[PATCHv4 00/14] fetch submodules in parallel and a preview on parallel "submodule update"”
  1. Stefan BellerSep 23, 2015
  2. 01/14 submodule: Send "Fetching submodule <foo>" to standard errorStefan Beller, Sep 23, 2015
  3. 02/14 xread: poll on non blocking fdsStefan Beller, Sep 23, 2015
  4. 03/14 xread_nonblock: add functionality to read from fds without blockingStefan Beller, Sep 23, 2015
  5. 04/14 strbuf: add strbuf_read_once to read without blockingStefan Beller, Sep 23, 2015
  6. 05/14 run-command: factor out return value computationStefan Beller, Sep 23, 2015
  7. 06/14 run-command: add an asynchronous parallel child processorStefan Beller, Sep 23, 2015
  8. Junio C HamanoSep 23, 2015
  9. Stefan BellerSep 23, 2015
  10. Junio C HamanoSep 23, 2015
  11. Junio C HamanoSep 23, 2015
  12. Stefan BellerSep 23, 2015
  13. Junio C HamanoSep 23, 2015
  14. Junio C HamanoSep 23, 2015
  15. Junio C HamanoSep 23, 2015
  16. Junio C HamanoSep 23, 2015
  17. [PATCHv5] Another squash on run-command: add an asynchronous parallel child processorStefan Beller, Sep 23, 2015
  18. Junio C HamanoSep 24, 2015
  19. 0/2 Another squash on run-command: add an asynchronous parallel child processorStefan Beller, Sep 24, 2015
  20. 2/2 SQUASH for "fetch_populated_submodules: use new parallel job processing"Stefan Beller, Sep 24, 2015
  21. 1/2 SQUASH???Stefan Beller, Sep 24, 2015
  22. Junio C HamanoSep 25, 2015
  23. Junio C HamanoSep 25, 2015
  24. Stefan BellerSep 25, 2015
  25. Junio C HamanoSep 25, 2015
  26. Junio C HamanoSep 25, 2015
  27. Stefan BellerSep 25, 2015
  28. Junio C HamanoSep 25, 2015
  29. Stefan BellerSep 25, 2015
  30. Junio C HamanoSep 25, 2015
  31. 07/14 fetch_populated_submodules: use new parallel job processingStefan Beller, Sep 23, 2015
  32. 08/14 submodules: allow parallel fetching, add tests and documentationStefan Beller, Sep 23, 2015
  33. 09/14 submodule-config: Untangle logic in parse_configStefan Beller, Sep 23, 2015
  34. 10/14 submodule config: keep update strategy aroundStefan Beller, Sep 23, 2015
  35. 11/14 git submodule update: cmd_update_recursiveStefan Beller, Sep 23, 2015
  36. 12/14 git submodule update: cmd_update_cloneStefan Beller, Sep 23, 2015
  37. Junio C HamanoSep 23, 2015
  38. 13/14 git submodule update: cmd_update_fetchStefan Beller, Sep 23, 2015
  39. 14/14 Rewrite submodule update in CStefan Beller, Sep 23, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.