[PATCH v2 0/2] read-cache: call verify_hdr() in a background thread

From: Jeff Hostetler <jeffhost@microsoft.com>

Version 2 of this patch series simplifies this to just turn off the hash verification. Independent comments from Linus and Peff suggested that we could just turn this off and not worry about it. So I've updated this patch to do that. I added a global variable to allow the original code path to be used. I also added a t/helper command to demonstrate the differences.

On the Linux repo, the effect is rather trivial:

    $ ~/work/gfw/t/helper/test-skip-verify-index -c 3
    0.029884 0 [cache_nr 57994]
    0.031035 0 [cache_nr 57994]
    0.024308 0 [cache_nr 57994]
    0.028409 0 avg
    0.018359 1 [cache_nr 57994]
    0.017025 1 [cache_nr 57994]
    0.011087 1 [cache_nr 57994]
    0.015490 1 avg

On my Windows source tree (450MB index), I'm seeing a savings of 0.6 seconds -- read_index() went from 1.2 to 0.6 seconds.

=====================

This patch contains a performance optimization to run verify_hdr() in a background thread while the foreground thread parses the index. This allows do_read_index() to complete faster.

This idea was recently discussed on the mailing list in: https://public-inbox.org/git/85221b97-759f-b7a9-1256-21515d163cbf@jeffhostetler.com/ during a discussion on sha1dc.

Our testing on Windows showed that verifying the SHA1 hash on the index file takes approximately the same amount of time as parsing the file and building the array of cache_entries. (It could also be that having 2 threads better ammortizes the page faults of reading from the mmap'd file.)

An earlier version of this change has been in use in GfW since December: https://github.com/git-for-windows/git/pull/978

Jeff Hostetler (2):
  read-cache: skip index SHA verification
  skip_verify_index: helper test

 Makefile                          |  1 +
 cache.h                           |  5 +++
 read-cache.c                      |  6 ++++
 t/helper/.gitignore               |  1 +
 t/helper/test-skip-verify-index.c | 73 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 86 insertions(+)
 create mode 100644 t/helper/test-skip-verify-index.c