[PATCH 00/19] index-helper, watchman

This is a rebase and extension of Duy's work on git index-helper and watchman support.

Patches 1-14 and are more-or-less just Duy's patches, rebased. I fixed up a race condition with signal handling, but otherwise didn't change much.

Patch 15 preserves the untracked cache and watchman extensions across checkouts.

Patches 16-18 are new index-helper patches that I wrote to improve the general usability of index-helper.

And patch 19 is for discussion only -- if the overall concept is correct, I'll rewrite it into the places where it should go. I didn't do it yet in part because I am not sure folks will agree with it and in part because I didn't want to mess too much with Duy's code before re-sending it.

I haven't used this code much. I just wrote patches 16-18 today, in fact. It's got fewer tests than I would like, in part because it is somewhat difficult to test since it involves two separate persistent daemons (index-helper, watchman), and the interaction between them.

At Twitter, we're still using something based on my earlier watchman patches[1]. We would like to switch to this (especially if this is the version that mainstream git is going towards). But we have other local patches[2], and I haven't fully finished migrating them to 2.8. My informal testing shows that with the untracked cache and watchman index extensions, and git index-helper[3], performance is slightly better than my earlier code. That's not a surprise, since the index-helper eliminates index verification time.

[1] https://github.com/dturner-tw/git/tree/dturner/watchman

[2] e.g. something like this: https://github.com/dturner-tw/git/tree/dturner/journal

[3] In both cases, I'm using a version which has the SSE ref-parsing and hashmap code; these were rejected due to complexity but they provide a benefit for us, so we still use them.

David Turner (5):
  unpack-trees: preserve index extensions
  index-helper: rewrite pidfile after daemonizing
  index-helper: process management
  index-helper: autorun
  hack: watchman/untracked cache mashup

Nguyễn Thái Ngọc Duy (14):
  trace.c: add GIT_TRACE_PACK_STATS for pack usage statistics
  read-cache.c: fix constness of verify_hdr()
  read-cache: allow to keep mmap'd memory after reading
  index-helper: new daemon for caching index and related stuff
  trace.c: add GIT_TRACE_INDEX_STATS for index statistics
  index-helper: add --strict
  daemonize(): set a flag before exiting the main process
  index-helper: add --detach
  index-helper: add Windows support
  read-cache: add watchman 'WAMA' extension
  Add watchman support to reduce index refresh cost
  read-cache: allow index-helper to prepare shm before git reads it
  index-helper: use watchman to avoid refreshing index with lstat()
  update-index: enable/disable watchman support

 .gitignore                         |   1 +
 Documentation/git-index-helper.txt |  56 +++++
 Documentation/git.txt              |   4 +
 Makefile                           |  21 ++
 builtin/gc.c                       |   2 +-
 builtin/update-index.c             |  11 +
 cache.h                            |  20 +-
 config.c                           |   5 +
 config.mak.uname                   |   3 +
 configure.ac                       |   8 +
 daemon.c                           |   2 +-
 dir.c                              |  23 +-
 dir.h                              |   6 +
 environment.c                      |   3 +
 git-compat-util.h                  |   1 +
 git.c                              |  37 +++-
 index-helper.c                     | 437 ++++++++++++++++++++++++++++++++++++
 read-cache.c                       | 441 +++++++++++++++++++++++++++++++++++--
 setup.c                            |   4 +-
 sha1_file.c                        |  24 ++
 shm.c                              | 163 ++++++++++++++
 shm.h                              |  23 ++
 t/t7063-status-untracked-cache.sh  |  23 ++
 t/t7900-index-helper.sh            |  23 ++
 t/test-lib-functions.sh            |   4 +
 trace.c                            |  16 ++
 trace.h                            |   1 +
 unpack-trees.c                     |   1 +
 watchman-support.c                 | 134 +++++++++++
 watchman-support.h                 |   7 +
 30 files changed, 1481 insertions(+), 23 deletions(-)
 create mode 100644 Documentation/git-index-helper.txt
 create mode 100644 index-helper.c
 create mode 100644 shm.c
 create mode 100644 shm.h
 create mode 100755 t/t7900-index-helper.sh
 create mode 100644 watchman-support.c
 create mode 100644 watchman-support.h