[PATCH 0/4] index-pack improvements

The first two patches are posted already, which help keep memory usage down in the present of large blobs. The forth patch attempts to resolve deltas in parallel.

$ time ./git index-pack --threads=1 --verify -v XXXX Indexing objects: 100% (165375/165375), done. Resolving deltas: 100% (124749/124749), done.

real 1m15.470s user 1m14.899s sys 0m0.552s

$ time ./git index-pack --threads=2 --verify -v XXXX Indexing objects: 100% (165375/165375), done. Resolving deltas: 100% (124749/124749), done.

real 0m41.339s user 1m15.116s sys 0m0.680s

$ time ./git index-pack --threads=3 --verify -v XXXX Indexing objects: 100% (165375/165375), done. Resolving deltas: 100% (124749/124749), done.

real 0m37.008s user 1m35.742s sys 0m0.803s

$ time ./git index-pack --verify -v XXXX # four core machine Indexing objects: 100% (165375/165375), done. Resolving deltas: 100% (124749/124749), done.

real 0m33.701s user 1m51.316s sys 0m0.768s

$ time ./git index-pack --threads=8 --verify -v XXXX Indexing objects: 100% (165375/165375), done. Resolving deltas: 100% (124749/124749), done.

real 0m33.638s user 1m51.783s sys 0m0.773s

So there's improvement from user persepective, but overhead is too high (user time from 1m14 to 1m51). Making threaded_second_pass to process 512 consecutive objects each iteration to reduce contention on work_mutex does not help. Any ideas?

Nguyễn Thái Ngọc Duy (4):
  index-pack --verify: skip sha-1 collision test
  index-pack: reduce memory usage when the pack has large blobs
  index-pack: move second pass code into separate function
  index-pack: support multithreaded delta resolving

 Documentation/config.txt         |    4 +
 Documentation/git-index-pack.txt |   10 ++
 Makefile                         |    2 +-
 builtin/index-pack.c             |  298 +++++++++++++++++++++++++++++++-------
 4 files changed, 260 insertions(+), 54 deletions(-)