threads / patch / 31069

v3, 2 partsblock-sha1: avoid pointer conversion that violates alignment constraints

Subject: [PATCH lt/block-sha1 0/2 v3] block-sha1: avoid pointer conversion that violates alignment constraints

## tl;dr

9 messages between Jul 22, 2012 and Jul 23, 2012. Diffs are folded; open one to read it.

replies: 8people: 2as markdown or json

Jonathan Nieder· Jul 22, 2012, 23:35 UTC · lore
Hi Junio,

This is a series of two patches: the first avoids alignment faults that were making git either slow on Alpha machines or crashy, depending on the machine's configuration, and the second patch is a cosmetic nit noticed while reviewing the first.

Patches are based against
  30ae47b4 remove ARM and Mozilla SHA1 implementations, 2009-08-17

(aka the tip of the lt/block-sha1 branch) for no particular reason. They should work fine against a more modern codebase if you prefer that.

No changes since v2[1] except to commit messages.

I think this should be ready for application. Thanks to Michael and Linus for shaping the original patch into something sane.

Thoughts? Jonathan

[1] http://thread.gmane.org/gmane.comp.version-control.git/201434/focus=201456
Jonathan Nieder (2):
  block-sha1: avoid pointer conversion that violates alignment
    constraints
  block-sha1: put expanded macro parameters in parentheses
 block-sha1/sha1.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
Jonathan Nieder· Jul 22, 2012, 23:39 UTC · re: Jonathan Nieder · lore

[PATCH 1/2] block-sha1: avoid pointer conversion that violates alignment constraints

With 660231aa (block-sha1: support for architectures with memory alignment restrictions, 2009-08-12), blk_SHA1_Update was modified to access 32-bit chunks of memory one byte at a time on arches that prefer that:

	#define get_be32(p)    ( \
		(*((unsigned char *)(p) + 0) << 24) | \
		(*((unsigned char *)(p) + 1) << 16) | \
		(*((unsigned char *)(p) + 2) <<  8) | \
		(*((unsigned char *)(p) + 3) <<  0) )
The code previously accessed these values by just using htonl(*p).

Unfortunately, Michael noticed on an Alpha machine that git was using plain 32-bit reads anyway. As soon as we convert a pointer to int *, the compiler can assume that the object pointed to is correctly aligned as an int (C99 section 6.3.2.3 "pointer conversions" paragraph 7), and gcc takes full advantage by using a single 32-bit load, resulting in a whole bunch of unaligned access traps.

So we need to obey the alignment constraints even when only dealing with pointers instead of actual values. Do so by changing the type of 'data' to void *. This patch renames 'data' to 'block' at the same time to make sure all references are updated to reflect the new type.

Reported-tested-and-explained-by: Michael Cree <mcree@orcon.net.nz>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
---
Changes since v2:
 - fixed explanation of how the alignment assumption shows up in the
   assembler[1]
 - add Linus's ack
[1] http://thread.gmane.org/gmane.comp.version-control.git/201434/focus=201484
 block-sha1/sha1.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
Show changes to block-sha1/sha1.c +3 −3
diff --git a/block-sha1/sha1.c b/block-sha1/sha1.c
index d8934757..10fd94d1 100644
--- a/block-sha1/sha1.c
+++ b/block-sha1/sha1.c
@@ -100,7 +100,7 @@
  * Where do we get the source from? The first 16 iterations get it from
  * the input data, the next mix it from the 512-bit array.
  */
-#define SHA_SRC(t) get_be32(data + t)
+#define SHA_SRC(t) get_be32((unsigned char *) block + t*4)
 #define SHA_MIX(t) SHA_ROL(W(t+13) ^ W(t+8) ^ W(t+2) ^ W(t), 1)
 
 #define SHA_ROUND(t, input, fn, constant, A, B, C, D, E) do { \
@@ -114,7 +114,7 @@
 #define T_40_59(t, A, B, C, D, E) SHA_ROUND(t, SHA_MIX, ((B&C)+(D&(B^C))) , 0x8f1bbcdc, A, B, C, D, E )
 #define T_60_79(t, A, B, C, D, E) SHA_ROUND(t, SHA_MIX, (B^C^D) ,  0xca62c1d6, A, B, C, D, E )
 
-static void blk_SHA1_Block(blk_SHA_CTX *ctx, const unsigned int *data)
+static void blk_SHA1_Block(blk_SHA_CTX *ctx, const void *block)
 {
 	unsigned int A,B,C,D,E;
 	unsigned int array[16];
@@ -125,7 +125,7 @@ static void blk_SHA1_Block(blk_SHA_CTX *ctx, const unsigned int *data)
 	D = ctx->H[3];
 	E = ctx->H[4];
 
-	/* Round 1 - iterations 0-16 take their input from 'data' */
+	/* Round 1 - iterations 0-16 take their input from 'block' */
 	T_0_15( 0, A, B, C, D, E);
 	T_0_15( 1, E, A, B, C, D);
 	T_0_15( 2, D, E, A, B, C);
-- 
1.7.10.4
Jonathan Nieder· Jul 22, 2012, 23:40 UTC · re: Jonathan Nieder · lore

[PATCH 2/2] block-sha1: put expanded macro parameters in parentheses

't' is currently always a numeric constant, but it can't hurt to prepare for the day that it becomes useful for a caller to pass in a more complex expression.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Clarified subject line.  No other change.
Thanks for reading.
 block-sha1/sha1.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Show changes to block-sha1/sha1.c +2 −2
diff --git a/block-sha1/sha1.c b/block-sha1/sha1.c
index 10fd94d1..6f885c43 100644
--- a/block-sha1/sha1.c
+++ b/block-sha1/sha1.c
@@ -100,8 +100,8 @@
  * Where do we get the source from? The first 16 iterations get it from
  * the input data, the next mix it from the 512-bit array.
  */
-#define SHA_SRC(t) get_be32((unsigned char *) block + t*4)
-#define SHA_MIX(t) SHA_ROL(W(t+13) ^ W(t+8) ^ W(t+2) ^ W(t), 1)
+#define SHA_SRC(t) get_be32((unsigned char *) block + (t)*4)
+#define SHA_MIX(t) SHA_ROL(W((t)+13) ^ W((t)+8) ^ W((t)+2) ^ W(t), 1);
 
 #define SHA_ROUND(t, input, fn, constant, A, B, C, D, E) do { \
 	unsigned int TEMP = input(t); setW(t, TEMP); \
-- 
1.7.10.4
Junio C Hamano· Jul 23, 2012, 04:28 UTC · re: Jonathan Nieder · lore

Re: [PATCH lt/block-sha1 0/2 v3] block-sha1: avoid pointer conversion that violates alignment constraints

Jonathan Nieder <jrnieder@gmail.com> writes:
Show 7 quoted lines
> This is a series of two patches: the first avoids alignment faults
> that were making git either slow on Alpha machines or crashy,
> depending on the machine's configuration, and the second patch is a
> cosmetic nit noticed while reviewing the first.
> ...
> Thoughts?
> Jonathan
Thanks.

Did somebody actually compiled Git for Alpha, and even more surprisingly on a big-endian variant of one?

Jonathan Nieder· Jul 23, 2012, 04:51 UTC · re: Junio C Hamano · lore

Re: [PATCH lt/block-sha1 0/2 v3] block-sha1: avoid pointer conversion that violates alignment constraints

Junio C Hamano wrote:
> Thanks.
>
> Did somebody actually compiled Git for Alpha, and even more
> surprisingly on a big-endian variant of one?
Logs from building for Alpha and running the test suite are here:
 http://buildd.debian-ports.org/status/logs.php?pkg=git&arch=alpha
The big-endian part was just my idiocy, sorry.
Junio C Hamano· Jul 23, 2012, 05:10 UTC · re: Jonathan Nieder · lore

Re: [PATCH lt/block-sha1 0/2 v3] block-sha1: avoid pointer conversion that violates alignment constraints

Jonathan Nieder <jrnieder@gmail.com> writes:
Show 12 quoted lines
> Junio C Hamano wrote:
>
>> Thanks.
>>
>> Did somebody actually compiled Git for Alpha, and even more
>> surprisingly on a big-endian variant of one?
>
> Logs from building for Alpha and running the test suite are here:
>
>  http://buildd.debian-ports.org/status/logs.php?pkg=git&arch=alpha
>
> The big-endian part was just my idiocy, sorry.
Hrm, do we want an update log message for 1/2 then?
Jonathan Nieder· Jul 23, 2012, 05:28 UTC · re: Junio C Hamano · lore

Re: [PATCH lt/block-sha1 0/2 v3] block-sha1: avoid pointer conversion that violates alignment constraints

Junio C Hamano wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>> The big-endian part was just my idiocy, sorry.
>
> Hrm, do we want an update log message for 1/2 then?
Hm, I thought all the crazy had been eliminated already.
*looks again*

I guess "using a single 32-bit load" makes it sound like it's using a big-endian load instead of a load followed by twiddling in registers.

Simplest fix would be to drop the phrase "by using a single 32-bit load", leaving "... and gcc takes full advantage, resulting in a whole bunch of unaligned access traps." Would that work for you, or should I resend?

Thanks, Jonathan

Junio C Hamano· Jul 23, 2012, 05:42 UTC · re: Jonathan Nieder · lore

Re: [PATCH lt/block-sha1 0/2 v3] block-sha1: avoid pointer conversion that violates alignment constraints

Jonathan Nieder <jrnieder@gmail.com> writes:
Show 17 quoted lines
> Junio C Hamano wrote:
>> Jonathan Nieder <jrnieder@gmail.com> writes:
>
>>> The big-endian part was just my idiocy, sorry.
>>
>> Hrm, do we want an update log message for 1/2 then?
>
> Hm, I thought all the crazy had been eliminated already.
>
> *looks again*
>
> I guess "using a single 32-bit load" makes it sound like it's using a
> big-endian load instead of a load followed by twiddling in registers.
>
> Simplest fix would be to drop the phrase "by using a single 32-bit
> load", leaving "... and gcc takes full advantage, resulting in a whole
> bunch of unaligned access traps."

I was just being stupid, misread the original code and did not realize that the generic code quoted in your log message:

	#define get_be32(p) ( \
        	... byte-at-a-time implementation )

is not limited to big endian boxes [*1*]. So I think the message is fine as-is.

[Footnote]

*1* In other words, the ntohl(*(uint *)(p)) is used only on selected little endian boxes, but that does not mean the generic code was big-endian only. Little endian boxes that are not listed in #if block do use the generic byte-at-a-time macro.

Jonathan Nieder· Jul 23, 2012, 06:29 UTC · re: Junio C Hamano · lore

[PATCH/RFC 3/1] Makefile: BLK_SHA1 does not require fast htonl() and unaligned loads

block-sha1/ is fast on most known platforms. Clarify the Makefile to be less misleading about that.

Early versions of block-sha1/ explicitly relied on fast htonl() and fast 32-bit loads with arbitrary alignment. Now it uses those on some arches but the default behavior is byte-at-a-time access for the sake of arches like ARM, Alpha, and their kin and it is still pretty fast on these arches (fast enough to supersede the mozilla SHA1 implementation and the hand-written ARM assembler implementation that were bundled before).

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Junio C Hamano wrote:
> *1* In other words, the ntohl(*(uint *)(p)) is used only on selected
> little endian boxes, but that does not mean the generic code was
> big-endian only.  Little endian boxes that are not listed in #if
> block do use the generic byte-at-a-time macro.
Oh, that reminds me.  Here's a third patch.
 Makefile |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
Show changes to Makefile +2 −3
diff --git a/Makefile b/Makefile
index 134606b9..eadcc70a 100644
--- a/Makefile
+++ b/Makefile
@@ -84,9 +84,8 @@ all::
 # specify your own (or DarwinPort's) include directories and
 # library directories by defining CFLAGS and LDFLAGS appropriately.
 #
-# Define BLK_SHA1 environment variable if you want the C version
-# of the SHA1 that assumes you can do unaligned 32-bit loads and
-# have a fast htonl() function.
+# Define BLK_SHA1 environment variable to make use of the bundled
+# optimized C SHA1 routine.
 #
 # Define PPC_SHA1 environment variable when running make to make use of
 # a bundled SHA1 routine optimized for PowerPC.
-- 
1.7.10.4

← back to recent threads