[RFC/PATCH 0/4] parallel fetch for submodules

When I was looking at the branches of Jens for work done on submodules not yet upstream I found a commit "WIP threaded submodule fetching[1], and I was side tracked wanting to present a different approach to that.

The first patch is a bit unrelated as it relates to the rewrite of git-submodule.sh but also has code in submodule.c and the following patches modify code just around that, so I did not remove that patch from this series. It is the same I sent yesterday.

The next patch 2/4 presents a framework for parallel threaded work. It allows to setup a worker pool of <n> threads and then have a queue of tasks which are worked on by the threads. The patch is a the one which I'd request most comments on as I think that can be reused in a variety of situations (parallel checkout of files, parallel fetch of different remotes, or such).

I consider the third patch farely boring as it adds argv_array_copy, so I would not expect much discussion there.

The last patch 4/4 presents the new workdispatcher from 2/4 in use with just one unsolved problem of how to handle the output of the parallel commands to stdout and stderr. It may be useful to put handling of parallel outputs into the work dispatcher.

[1] https://github.com/jlehmann/git-submod-enhancements/commit/47597753206d40e234a47392e258065c9489e2b3

This series applies on top of origin/sb/submodule-helper (d2c6c09ac819, submodule: implement `module_list` as a builtin helper) and can also be found at https://github.com/stefanbeller/git/tree/parallel-submodule-fetch

Stefan Beller (4):
  submodule: implement `module_name` as a builtin helper
  Add a workdispatcher to get work done in parallel
  argv_array: add argv_array_clone to clone an existing argv array
  submodule: add infrastructure to fetch submodules in parallel

 Makefile                    |   1 +
 argv-array.c                |  13 ++++
 argv-array.h                |   1 +
 builtin/fetch.c             |   3 +-
 builtin/submodule--helper.c |  23 ++++++
 git-submodule.sh            |  32 ++------
 submodule.c                 |  92 ++++++++++++++++------
 submodule.h                 |   3 +-
 workdispatcher.c            | 184 ++++++++++++++++++++++++++++++++++++++++++++
 workdispatcher.h            |  29 +++++++
 10 files changed, 332 insertions(+), 49 deletions(-)
 create mode 100644 workdispatcher.c
 create mode 100644 workdispatcher.h