git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCHv6 0/8] fetch submodules in parallel

From
Junio C Hamano <gitster@pobox.com>
Date
Oct 2, 2015, 18:48 UTC
Message-ID
<xmqqeghd5cp9.fsf@gitster.mtv.corp.google.com>
In-Reply-To
<1443664456-1307-1-git-send-email-sbeller@google.com>
Stefan Beller <sbeller@google.com> writes:
> * renamed return_value_fn to task_finished_fn
It made interdiff noisier but I think it gives us a good end result.
Show 5 quoted lines
> * the main loop of the parallel processing was first adapted to Junios suggestion,
>   but Jonathan pointed out more improvements.  We can get rid of `no_more_task`
>   completely as `if (!pp->nr_processes)` as the exit condition is sufficient.
>   (pp->nr_processes is modified only when starting or reaping a child, so we will
>   capture the whole output of each subprocess even in case of a quick shutdown)

Interesting. The original motivation for "no-more-task" check was that even when we are no longer running anything (i.e. everybody finished) we may get a new task from next_task(), and the condition to "break" out of the loop could be placed anywhere in that loop (e.g. after we wait and cull the finished tasks, or even in the outermost while(1) condition).

But you can take advantage of the specific placement of the check; it is after the part that spawns new tasks and before the part that culls the existing tasks, so not having any running task at that point is sufficient condition.

Will replace what was queued.
Thanks.
Previous: Stefan Beller
Message 12 of 12 in “[PATCHv6 0/8] fetch submodules in parallel”
  1. Stefan BellerOct 1, 2015
  2. 1/8 submodule.c: write "Fetching submodule <foo>" to stderrStefan Beller, Oct 1, 2015
  3. 2/8 xread: poll on non blocking fdsStefan Beller, Oct 1, 2015
  4. 3/8 xread_nonblock: add functionality to read from fds without blockingStefan Beller, Oct 1, 2015
  5. 4/8 strbuf: add strbuf_read_once to read without blockingStefan Beller, Oct 1, 2015
  6. 5/8 sigchain: add command to pop all common signalsStefan Beller, Oct 1, 2015
  7. 6/8 run-command: add an asynchronous parallel child processorStefan Beller, Oct 1, 2015
  8. 7/8 fetch_populated_submodules: use new parallel job processingStefan Beller, Oct 1, 2015
  9. 8/8 submodules: allow parallel fetching, add tests and documentationStefan Beller, Oct 1, 2015
  10. Ramsay JonesOct 1, 2015
  11. Stefan BellerOct 1, 2015
  12. Junio C HamanoOct 2, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.