[PATCHv3 00/13] fetch submodules in parallel and a preview on parallel "submodule update"

This build on top of origin/sb/submodule-helper

Patches 1-8 are parallelising fetching submodules. Thanks Eric, Jeff and Junio for the discussion and review! Based on your feedback:

Patch 2 is optional and may be spun out as its own mini series. Instead of spinning we can poll until data are ready on the fd.

Patch 3,4 have been revamped to include your suggestions, having xread_nonblock and strbuf_read_once as their own functions not to be integrated into other existing functions.

Patch 5 was fixed to return 0 or -1 in the error case. Patch 6 has major syntactically changes such as no slots any more, but in_use flags. The return code was flipped (1 means there is a child to be executed, while 0 means we're done).

Patch 7,8 convert git fetch --recuse-submodules to use the fruit of patch 6.

Patches 9 an onwards are RFC, so I could spin them out as their own series as they convert parts of "submodule update" to C using the new asynchronous run command code. Patch 9 adds the update strategy to the submodule cache Patches 10,11,12 are just moving code around to make cmd_update leaner. Patch 13 shows how I plan to parallelize updating the submodules for now.

Any feedback welcome, Thanks, Stefan

Jonathan Nieder (1):
  Sending "Fetching submodule <foo>" output to stderr

Stefan Beller (13):
  xread: poll on non blocking fds
  xread_nonblock: add functionality to read from fds nonblockingly
  strbuf: add strbuf_read_once to read without blocking
  run-command: factor out return value computation
  run-command: add an asynchronous parallel child processor
  fetch_populated_submodules: use new parallel job processing
  submodules: allow parallel fetching, add tests and documentation
  submodule config: keep update strategy around
  git submodule update: pass --prefix only with a non empty prefix
  git submodule update: cmd_update_recursive
  git submodule update: cmd_update_clone
  git submodule update: cmd_update_fetch
  Rewrite submodule update in C

 Documentation/fetch-options.txt |   7 +
 builtin/fetch.c                 |   6 +-
 builtin/pull.c                  |   6 +
 builtin/submodule--helper.c     | 247 +++++++++++++++++++++++++++++
 git-compat-util.h               |   1 +
 git-submodule.sh                | 339 ++++++++++++++--------------------------
 run-command.c                   | 329 +++++++++++++++++++++++++++++++++++---
 run-command.h                   |  38 +++++
 strbuf.c                        |  11 ++
 strbuf.h                        |   6 +
 submodule-config.c              |  16 ++
 submodule-config.h              |   1 +
 submodule.c                     | 119 +++++++++-----
 submodule.h                     |   2 +-
 t/t0061-run-command.sh          |  20 +++
 t/t5526-fetch-submodules.sh     |  70 ++++++---
 test-run-command.c              |  24 +++
 wrapper.c                       |  41 ++++-
 18 files changed, 972 insertions(+), 311 deletions(-)