Re: [PATCH] git gc: Speed it up by 18% via faster hash comparisons

From: Ingo Molnar <mingo@elte.hu>
Date: Apr 28, 2011, 10:19 UTC
Message-ID: <20110428101902.GA17257@elte.hu>
In-Reply-To: <BANLkTim+Kk_ah_4+pQKCi8bXtA8thRVRjQ@mail.gmail.com>

* Erik Faye-Lund <kusmabite@gmail.com> wrote:

Show 17 quoted lines

> 2011/4/28 Ingo Molnar <mingo@elte.hu>:
> >
> > * Erik Faye-Lund <kusmabite@gmail.com> wrote:
> >
> >> > Secondly, the combined speedup of the cached case with my two patches
> >> > appears to be more than 30% on my testbox so it's a very nifty win from two
> >> > relatively simple changes.
> >>
> >> That speed-up was on ONE test vector, no? There are a lot of other uses of
> >> hash-comparisons in Git, did you measure those?
> >
> > I picked this hash function because it showed up in the profile (see the
> > profile i posted). There's one other hash that mattered as well in the profile,
> > see the lookup_object() patch i sent yesterday.
> 
> My point was that the 30% improvement was in "git gc", which is not
> the only important use-case. How does this affect other git commands?

In a followup mail i measured git fsck, which showed a speedup too. (despite being mostly dependent on external libraries to do most of the processing)

If you'd like to see other things tested please suggest a testcase that you think uses these hashes extensively, i don't really know what the slowest (and affected) Git commands are - git gc is the one *i* notice as being pretty slow (for good reasons).

Show 9 quoted lines

> >> from the exception handler, others doesn't. So this patch is pretty much
> >> guaranteed to cause a crash in some setups.
> >
> > If unsigned char arrays are allocated unaligned then that's another bug i
> > suspect that should be fixed.
> 
> We can't. The compiler decides the alignment of variables on the stack. Some 
> compilers / compiler-setting pairs might align char-arrays, while others 
> might not.

Even if that were true it can be solved: you'd need to declare the sha1 not as a char array but as a u32 * array or so. We do have control over the alignment of data structures, obviously.

Show 9 quoted lines

> > Unaligned access on x86 is not free either - there's cycle penalties.
> >
> > Alas, i have not seen these sha1 hash buffers being allocated unaligned (in 
> > my very limited testing). In which spots are they allocated unaligned?
> 
> Like I said above, it can happen when allocated on the stack. But it can also 
> happen in malloc'ed structs, or in global variables. An array is aligned to 
> the size of it's base member type. But malloc does worst-case-allignment, 
> because it happens at run-time without type-information.

Well, should we ready be ready to throw up our hands as if we didnt have control over the alignment of objects and have to accept suboptimal code as a result? We do have control over that.

In any case, i'll retract the null case as it really isnt called that often in the tests i've done - updated patch below - it simply falls back on to hashcmp.

Thanks,

	Ingo

Signed-off-by: Ingo Molnar <mingo@elte.hu>

diff --git a/cache.h b/cache.h
index 2674f4c..39fa9cd 100644
--- a/cache.h
+++ b/cache.h
@@ -675,14 +675,24 @@ extern char *sha1_pack_name(const unsigned char *sha1);
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 extern const char *find_unique_abbrev(const unsigned char *sha1, int);
 extern const unsigned char null_sha1[20];
-static inline int is_null_sha1(const unsigned char *sha1)
+
+static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2)
 {
-	return !memcmp(sha1, null_sha1, 20);
+	int i;
+
+	for (i = 0; i < 20; i++, sha1++, sha2++) {
+		if (*sha1 != *sha2)
+			return *sha1 - *sha2;
+	}
+
+	return 0;
 }
-static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2)
+
+static inline int is_null_sha1(const unsigned char *sha1)
 {
-	return memcmp(sha1, sha2, 20);
+	return !hashcmp(sha1, null_sha1);
 }
+
 static inline void hashcpy(unsigned char *sha_dst, const unsigned char *sha_src)
 {
 	memcpy(sha_dst, sha_src, 20);