[RFC HACK] refresh_index: lstat() in inode order

From: Thomas Rast <trast@inf.ethz.ch>
Date: Mar 6, 2012, 08:27 UTC
Message-ID: <871up6cewp.fsf@thomas.inf.ethz.ch>

Hi,

This is prompted by a recent lkml discussion[1] which is also backed up by old stuff threads like[2] where Theodore Ts'o writes about spd_readdir.

As explained at the links, ext3&4 seem to show suboptimal performance if you do the common pattern of lstat()'ing everything returned by readdir() and the corresponding inodes must be fetched from disk. They suggest (and spd_readdir does just this, in an LD_PRELOADable format) sorting the entries by inode.

Let me emphasise: inodes fetched from disk. This does not matter at all for the cached case.

Probably this also applies to us in the search for untracked files, but right now I don't care about that too much because I have git-status configured not to show them.

However, a similar trick could be applied to lstat() across the index, which *is* a big part of the work required to accurately display repository status in git-status and git-diff.

The dirty hack below uses the inode field in the index to sort the index entries prior to the real work of refresh_index. I have been able to measure a significant (in the statistical sense) speedup using measurements like

  ( for i in $(seq 1 30); do
      sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'
      /usr/bin/time -f "time %U %S %E" ~/dev/git/git-status 2>&1 | grep time
    done ) | tee results-patched

In numbers, across 30 trials for unpatched and patched git, my time needed to get a cache-cold 'git status' went from 4.24s to 3.97s on average, at p=0.006.

Note that this only has an effect if your directory has the inodes all jumbled. I tried doing bigger trials, but I do not have a realistic work repository bigger than git.git. You can look at the lstat() order you are currently getting with

  strace -e lstat -v git status 2>&1 | egrep -o 'st_ino=[0-9]+'

With the patch it should be in good sorted order. Note, however, that at the very end it also runs lstat() on .git/refs/*; those will still be out of order.

So I'd be interested to hear success (or non-success) stories from people who are actively working with larger repos, perhaps linux-2.6 or your favourite corporate repo. The lkml posts also seem to say that it only matters on ext3&4, not on btrfs, but perhaps other FSes also suffer in this area.

I'd also be interested to hear from the refresh_index experts whether my change works in principle, or is already flawed in some major way. You will note I sort by (inode,stage) to handle the search forward for the highest stage entry, but maybe this is not the only twist. (As written it is also not thread-safe.)

I did run the test suite and it passes, so it can't be *that* bad.

Footnotes: 
[1]  https://lkml.org/lkml/2012/2/29/210

[2]  http://lkml.indiana.edu/hypermail/linux/kernel/0802.2/1076.html

---- 8< ----

diff --git i/read-cache.c w/read-cache.c
index 274e54b..b0d1942 100644
--- i/read-cache.c
+++ w/read-cache.c
@@ -1092,10 +1092,28 @@ static void show_file(const char * fmt, const char * name, int in_porcelain,
 	printf(fmt, name);
 }
 
+static struct index_state *istate_for_cmp;
+
+int cmp_by_inode(const void *a, const void *b)
+{
+	struct cache_entry *ca, *cb;
+
+	ca = istate_for_cmp->cache[*(const int *)a];
+	cb = istate_for_cmp->cache[*(const int *)b];
+
+	if (ca->ce_ino < cb->ce_ino)
+		return -1;
+	if (ca->ce_ino > cb->ce_ino)
+		return 1;
+	if (ce_stage(ca) < ce_stage(cb))
+		return -1;
+	return 1;
+}
+
 int refresh_index(struct index_state *istate, unsigned int flags, const char **pathspec,
 		  char *seen, const char *header_msg)
 {
-	int i;
+	int i, j;
 	int has_errors = 0;
 	int really = (flags & REFRESH_REALLY) != 0;
 	int allow_unmerged = (flags & REFRESH_UNMERGED) != 0;
@@ -1110,18 +1128,28 @@ int refresh_index(struct index_state *istate, unsigned int flags, const char **p
 	const char *typechange_fmt;
 	const char *added_fmt;
 	const char *unmerged_fmt;
+	int *by_inode_idx;
 
 	modified_fmt = (in_porcelain ? "M\t%s\n" : "%s: needs update\n");
 	deleted_fmt = (in_porcelain ? "D\t%s\n" : "%s: needs update\n");
 	typechange_fmt = (in_porcelain ? "T\t%s\n" : "%s needs update\n");
 	added_fmt = (in_porcelain ? "A\t%s\n" : "%s needs update\n");
 	unmerged_fmt = (in_porcelain ? "U\t%s\n" : "%s: needs merge\n");
-	for (i = 0; i < istate->cache_nr; i++) {
+
+	by_inode_idx = xmalloc(istate->cache_nr * sizeof(int));
+	for (i = 0; i < istate->cache_nr; i++)
+		by_inode_idx[i] = i;
+	istate_for_cmp = istate;
+	qsort(by_inode_idx, istate->cache_nr, sizeof(int), cmp_by_inode);
+
+	for (j = 0; j < istate->cache_nr; j++) {
 		struct cache_entry *ce, *new;
 		int cache_errno = 0;
 		int changed = 0;
 		int filtered = 0;
 
+		i = by_inode_idx[j];
+
 		ce = istate->cache[i];
 		if (ignore_submodules && S_ISGITLINK(ce->ce_mode))
 			continue;

-- 
Thomas Rast
trast@{inf,student}.ethz.ch