[PATCH v4 0/1] unpack-trees: skip stat on fsmonitor-valid files

The index might be aware that a file hasn't modified via fsmonitor, but unpack-trees did not pay attention to it and checked via ie_match_stat which can be inefficient on certain filesystems. This significantly slows down commands that run oneway_merge, like checkout and reset --hard.

This patch makes oneway_merge check whether a file is considered unchanged through fsmonitor and skips ie_match_stat on it. unpack-trees also now correctly copies over fsmonitor validity state from the source index. Finally, for correctness, we force a refresh of fsmonitor state in tweak_fsmonitor.

After this change, commands like stash (that use reset --hard internally) go from 8s or more to ~2s on a 250k file repository on a mac.

Changes since the last version are:

 * The sanity checks around accessing the fsmonitor_dirty bitmap have been
   moved to another patch, which is in message id [1]
 * Unintended indentation changes in fsmonitor have been removed
 * A comment explaining what untracked->use_fsmonitor means has been
   re-added (it was dropped in the previous version)
 * A few "helped-by" entries have been added to the patch

1: (xmqqzhh0d0ma.fsf@gitster-ct.c.googlers.com)

Helped-by: Junio C Hamano gitster@pobox.com [gitster@pobox.com]Helped-by:
Kevin Willford Kevin.Willford@microsoft.com [Kevin.Willford@microsoft.com]
Signed-off-by: Utsav Shah utsav@dropbox.com [utsav@dropbox.com]

Utsav Shah (1):
  unpack-trees: skip stat on fsmonitor-valid files

 fsmonitor.c                 | 23 +++++++++++++++++------
 t/t7519-status-fsmonitor.sh |  9 +++++++--
 unpack-trees.c              |  6 +++++-
 3 files changed, 29 insertions(+), 9 deletions(-)

base-commit: 566a1439f6f56c2171b8853ddbca0ad3f5098770
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-424%2FUtsav2%2Fskip-lstat-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-424/Utsav2/skip-lstat-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/424

Range-diff vs v3:

 1:  4bea7075cf ! 1:  ea7880f2d0 unpack-trees: skip stat on fsmonitor-valid files
     @@ -17,52 +17,29 @@
          internally) go from 8s or more to ~2s on a 250k file repository on a
          mac.
      
     +    Changes since the last version are:
     +    * The sanity checks around accessing the fsmonitor_dirty bitmap have
     +    been moved to another patch, which is in message id [1]
     +    * Unintended indentation changes in fsmonitor have been removed
     +    * A comment explaining what untracked->use_fsmonitor means has been
     +    re-added (it was dropped in the previous version)
     +    * A few "helped-by" entries have been added to the patch
     +
     +    [1]: (xmqqzhh0d0ma.fsf@gitster-ct.c.googlers.com)
     +
     +    Helped-by: Junio C Hamano <gitster@pobox.com>
     +    Helped-by: Kevin Willford <Kevin.Willford@microsoft.com>
          Signed-off-by: Utsav Shah <utsav@dropbox.com>
      
       diff --git a/fsmonitor.c b/fsmonitor.c
       --- a/fsmonitor.c
       +++ b/fsmonitor.c
      @@
     - 
     - 	if (pos >= istate->cache_nr)
     - 		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
     --		    (uintmax_t)pos, istate->cache_nr);
     -+			(uintmax_t)pos, istate->cache_nr);
     - 
     - 	ce = istate->cache[pos];
     - 	ce->ce_flags &= ~CE_FSMONITOR_VALID;
     -@@
     - 	}
     - 	istate->fsmonitor_dirty = fsmonitor_dirty;
     - 
     --	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     --		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     --		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
     -+	if (!istate->split_index && istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+			(uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     -+
     - 
     - 	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
     - 	return 0;
     -@@
     - 	uint32_t ewah_size = 0;
     - 	int fixup = 0;
     - 
     --	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     --		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     --		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
     -+	if (!istate->split_index && istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+			(uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     - 
     - 	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
     - 	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
     -@@
       		}
       		if (bol < query_result.len)
       			fsmonitor_refresh_callback(istate, buf + bol);
      +
     ++		/* Now mark the untracked cache for fsmonitor usage */
      +		if (istate->untracked)
      +			istate->untracked->use_fsmonitor = 1;
       	} else {
     @@ -88,11 +65,7 @@
       		if (istate->untracked)
       			istate->untracked->use_fsmonitor = 0;
      @@
     - 			/* Mark all previously saved entries as dirty */
     - 			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     - 				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     --				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
     -+					(uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
     + 				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
       			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
       
      -			/* Now mark the untracked cache for fsmonitor usage */