Watchman support for git

The most sigificant patch uses Facebook's watchman daemon[1] to monitor the repository work tree for changes. This makes allows git status to avoid traversing the entire work tree to find changes.

This change requires libwatchman[2], a client library that I wrote for watchman.

While making the watchman change, I also made a change to the index format (contributed here in a separate patch). Index integrity checking uses the same SHA1 algorithm as the rest of git; this is actually relatively slow. It's not a huge part of run-time, but since I wanted to do the same checking for the Watchman filesystem cache (which is about twice as large as the index), I decided to optimize it anyway. I switched to VMAC. VMAC is supposed to be a MAC, but there's no reason it can't be used with a fixed key as a simple integrity check. VMAC is roughly five times faster than SHA1 on my machine; This adds up to a 5% overal speed improvement on git status (depending on the structure of your repo, and about 15% on git diff --cached with no cached changes).

The index format change might be less important with the split index; I haven't investigated that since at the time I wrote these patches, it didn't exist.

Some numbers follow. They are on my laptop, which has 4x i5-2520M processors, 8GB of RAM, and a solid state disk. They're all tested with a hot cache.

Test repository 1: Linux

Linux is about 45k files in 3k directories. The average length of a filename is about 32 bytes.

Git status timing: no watchman: 125ms watchman: 90ms

Test repository 2: Superscience

My second test repository (which is a semi-synthetic repo generated from various Twitter internal repos) is somewhat larger than this, and gets a correspondingly larger improvement. It is about 65k files in 20k directories; the average length of a filename is 67 bytes.

Git status timing: no watchman, index version 4: 370 ms no watchman, index version 5: 365 ms watchman, index version 4: 170 ms watchman, index version 5: 165 ms

[1] https://github.com/facebook/watchman [2] https://github.com/twitter/libwatchman