[RFC PATCH 0/5] Progressing with `git submodule foreach_parallel`

This series applies on top of 10d0bef9981c8045b8c9aac1ca7b8761896bed0c in the branch origin/sb/submodule-helper

The first patch is a resend of module_clone, which now contains all memory leak fixes.

The second patch adding the new task queue is rewritten to not use semaphores any more, but only mutexes and condition variables as that's what we use at other places in the code base. So it is better to align to these places, it makes the threading pool also easier to understand internally. Originally there were 2 semaphores to control the queue length (0 < queue length < number of threads), such that we don't overallocate memory, but rather slow down the thread adding new tasks to the queue. This has been removed and the queue may grow infinitely large.

The third patch adding `git submodule foreach_parallel` learned a totally different behavior how to interact with the output channels (way better for human consumption as you have early feedback and feedback all the time, ok-ish for machine consumption, but far from perfect).

The fourth patch rewriting index-pack was made smaller and doesn't rewrite the workflow any more, but only plugs in the new threading API instead of constructing the pthreads themselves. I started benchmarking that patch, but I seem to get only bogus results, so I cannot tell if it still makes it 2 % slower as Jeff claimed.

The fifth patch refactors pack-objects to be *way* more understandable IMHO! It also switches to the new threading pool, but the refactor may be suitable also if we decide to not use the threading API there. The reason for it being more readable is the movement of the rebalancing into the threads themselves, which then turns out to require a lot less of locking and communication variables.

Any feedback is welcome! Thanks, Stefan

Stefan Beller (5):
  submodule: implement `module_clone` as a builtin helper
  thread-utils: add a threaded task queue
  submodule: helper to run foreach in parallel
  index-pack: Use the new worker pool
  pack-objects: Use new worker pool

 builtin/index-pack.c        |  23 ++--
 builtin/pack-objects.c      | 175 ++++++++----------------
 builtin/submodule--helper.c | 318 +++++++++++++++++++++++++++++++++++++++++++-
 git-submodule.sh            |  91 ++-----------
 run-command.c               |  29 ++--
 thread-utils.c              | 227 +++++++++++++++++++++++++++++++
 thread-utils.h              |  35 +++++
 7 files changed, 677 insertions(+), 221 deletions(-)