# [PATCH v2 1/3] sha1dc: correct endian detection for Solaris (and others?) 9 messages from 2017-06-27 to 2017-06-28. Participants: Ævar Arnfjörð Bjarmason, Junio C Hamano. Thread: https://gitlist.dev/t/46273 ## Ævar Arnfjörð Bjarmason, 2017-06-27 20:33 Subject: [PATCH v2 1/3] sha1dc: correct endian detection for Solaris (and others?) Message-ID: <20170627203348.19827-2-avarab@gmail.com> URL: https://gitlist.dev/e/20170627203348.19827-2-avarab%40gmail.com In-Reply-To: <20170627203348.19827-1-avarab@gmail.com> ``` Update sha1dc from my PR #36[1] which'll hopefully be integrated by upstream soon. This solves the Big Endian detection on Solaris reported against v2.13.2[2], hopefully without any regressions. See commit a0103914c2 ("sha1dc: update from upstream", 2017-05-20) and 6b851e536b ("sha1dc: update from upstream", 2017-06-06) for previous attempts in the 2.13 series to address various compile-time feature detection in this library. 1. https://github.com/cr-marcstevens/sha1collisiondetection/pull/36 https://github.com/avar/sha1collisiondetection/commit/56ab30c4c998e1e7f3075705087a2f0c4c4202d7 2. (https://public-inbox.org/git/CAKKM46tHq13XiW5C8sux3=PZ1VHSu_npG8ExfWwcPD7rkZkyRQ@mail.gmail.com/) Signed-off-by: Ævar Arnfjörð Bjarmason --- sha1dc/sha1.c | 73 +++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 54 insertions(+), 19 deletions(-) diff --git a/sha1dc/sha1.c b/sha1dc/sha1.c index facea1bb56..1e7e30763e 100644 --- a/sha1dc/sha1.c +++ b/sha1dc/sha1.c @@ -23,6 +23,13 @@ #include "sha1.h" #include "ubc_check.h" +#if (defined(__amd64__) || defined(__amd64) || defined(__x86_64__) || defined(__x86_64) || \ + defined(i386) || defined(__i386) || defined(__i386__) || defined(__i486__) || \ + defined(__i586__) || defined(__i686__) || defined(_M_IX86) || defined(__X86__) || \ + defined(_X86_) || defined(__THW_INTEL__) || defined(__I86__) || defined(__INTEL__) || \ + defined(__386) || defined(_M_X64) || defined(_M_AMD64)) +#define SHA1DC_ON_INTEL_LIKE_PROCESSOR +#endif /* Because Little-Endian architectures are most common, @@ -32,28 +39,63 @@ If you are compiling on a big endian platform and your compiler does not define one of these, you will have to add whatever macros your tool chain defines to indicate Big-Endianness. */ -#ifdef SHA1DC_BIGENDIAN -#undef SHA1DC_BIGENDIAN + +#if defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) +/* + * Should detect Big Endian under GCC since at least 4.6.0 (gcc svn + * rev #165881). See + * https://gcc.gnu.org/onlinedocs/cpp/Common-Predefined-Macros.html + * + * This also works under clang since 3.2, it copied the GCC-ism. See + * clang.git's 3b198a97d2 ("Preprocessor: add __BYTE_ORDER__ + * predefined macro", 2012-07-27) + */ +#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define SHA1DC_BIGENDIAN #endif -#if (defined(_BYTE_ORDER) || defined(__BYTE_ORDER) || defined(__BYTE_ORDER__)) +#else /* Not under GCC-alike */ -#if ((defined(_BYTE_ORDER) && (_BYTE_ORDER == _BIG_ENDIAN)) || \ - (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \ - (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) ) +#if defined(__BYTE_ORDER) && defined(__BIG_ENDIAN) +/* + * Should detect Big Endian under glibc.git since 14245eb70e ("entered + * into RCS", 1992-11-25). Defined in which will have been + * brought in by standard headers. See glibc.git and + * https://sourceforge.net/p/predef/wiki/Endianness/ + */ +#if __BYTE_ORDER == __BIG_ENDIAN #define SHA1DC_BIGENDIAN #endif -#else +#else /* Not under GCC-alike or glibc */ -#if (defined(_BIG_ENDIAN) || defined(__BIG_ENDIAN) || defined(__BIG_ENDIAN__) || \ - defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \ +#if (defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \ defined(__MIPSEB__) || defined(__MIPSEB) || defined(_MIPSEB) || \ defined(__sparc)) +/* + * Should define Big Endian for a whitelist of known processors. See + * https://sourceforge.net/p/predef/wiki/Endianness/ and + * http://www.oracle.com/technetwork/server-storage/solaris/portingtosolaris-138514.html + */ #define SHA1DC_BIGENDIAN -#endif -#endif +#else /* Not under GCC-alike or glibc or */ + +#if defined(SHA1DC_ON_INTEL_LIKE_PROCESSOR) +/* + * As a last resort before we do anything else we're not 100% sure + * about below, we blacklist specific processors here. We could add + * more, see e.g. https://wiki.debian.org/ArchitectureSpecificsMemo + */ +#else /* Not under GCC-alike or glibc or or */ + +/* We do nothing more here for now */ +/*#error "Uncomment this to see if you fall through all the detection"*/ + +#endif /* !SHA1DC_ON_INTEL_LIKE_PROCESSOR */ +#endif /* Big Endian under whitelist of processors */ +#endif /* Big Endian under glibc */ +#endif /* Big Endian under GCC-alike */ #if (defined(SHA1DC_FORCE_LITTLEENDIAN) && defined(SHA1DC_BIGENDIAN)) #undef SHA1DC_BIGENDIAN @@ -63,15 +105,8 @@ #endif /*ENDIANNESS SELECTION*/ -#if (defined SHA1DC_FORCE_UNALIGNED_ACCESS || \ - defined(__amd64__) || defined(__amd64) || defined(__x86_64__) || defined(__x86_64) || \ - defined(i386) || defined(__i386) || defined(__i386__) || defined(__i486__) || \ - defined(__i586__) || defined(__i686__) || defined(_M_IX86) || defined(__X86__) || \ - defined(_X86_) || defined(__THW_INTEL__) || defined(__I86__) || defined(__INTEL__) || \ - defined(__386) || defined(_M_X64) || defined(_M_AMD64)) - +#if defined(SHA1DC_FORCE_UNALIGNED_ACCESS) || defined(SHA1DC_ON_INTEL_LIKE_PROCESSOR) #define SHA1DC_ALLOW_UNALIGNED_ACCESS - #endif /*UNALIGNMENT DETECTION*/ -- 2.13.1.611.g7e3b11ae1 ``` ## Ævar Arnfjörð Bjarmason, 2017-06-27 20:33 Subject: [PATCH v2 2/3] sha1dc: optionally use sha1collisiondetection as a submodule Message-ID: <20170627203348.19827-3-avarab@gmail.com> URL: https://gitlist.dev/e/20170627203348.19827-3-avarab%40gmail.com In-Reply-To: <20170627203348.19827-1-avarab@gmail.com> ``` Add an option to use the sha1collisiondetection library from the submodule in sha1collisiondetection/ instead of in the copy in the sha1dc/ directory. This allows us to try out the submodule in sha1collisiondetection without breaking the build for anyone who's not expecting them as we work out any kinks. This uses my own fork which integrates PR #36. See the preceding commit ("sha1dc: update from my PR #36", 2017-06-27) for details. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- .gitmodules | 4 ++++ Makefile | 12 ++++++++++++ hash.h | 4 ++++ sha1collisiondetection | 1 + 4 files changed, 21 insertions(+) create mode 100644 .gitmodules create mode 160000 sha1collisiondetection diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000000..2fea9996e9 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,4 @@ +[submodule "sha1collisiondetection"] + path = sha1collisiondetection + url = https://github.com/avar/sha1collisiondetection.git + branch = bigend-detect-solaris-again diff --git a/Makefile b/Makefile index b94cd5633c..f0cac1f246 100644 --- a/Makefile +++ b/Makefile @@ -162,6 +162,12 @@ all:: # algorithm. This is slower, but may detect attempted collision attacks. # Takes priority over other *_SHA1 knobs. # +# Define DC_SHA1_SUBMODULE in addition to DC_SHA1 to use the +# sha1collisiondetection shipped as a submodule instead of the +# non-submodule copy in sha1dc/. This is an experimental option used +# by the git project to migrate to using sha1collisiondetection as a +# submodule. +# # Define OPENSSL_SHA1 environment variable when running make to link # with the SHA1 routine from openssl library. # @@ -1448,8 +1454,14 @@ ifdef APPLE_COMMON_CRYPTO BASIC_CFLAGS += -DSHA1_APPLE else DC_SHA1 := YesPlease +ifdef DC_SHA1_SUBMODULE + LIB_OBJS += sha1collisiondetection/lib/sha1.o + LIB_OBJS += sha1collisiondetection/lib/ubc_check.o + BASIC_CFLAGS += -DDC_SHA1_SUBMODULE +else LIB_OBJS += sha1dc/sha1.o LIB_OBJS += sha1dc/ubc_check.o +endif BASIC_CFLAGS += \ -DSHA1_DC \ -DSHA1DC_NO_STANDARD_INCLUDES \ diff --git a/hash.h b/hash.h index a11fc9233f..bef3e630a0 100644 --- a/hash.h +++ b/hash.h @@ -8,7 +8,11 @@ #elif defined(SHA1_OPENSSL) #include #elif defined(SHA1_DC) +#ifdef DC_SHA1_SUBMODULE +#include "sha1collisiondetection/lib/sha1.h" +#else #include "sha1dc/sha1.h" +#endif #else /* SHA1_BLK */ #include "block-sha1/sha1.h" #endif diff --git a/sha1collisiondetection b/sha1collisiondetection new file mode 160000 index 0000000000..9d3a0b3783 --- /dev/null +++ b/sha1collisiondetection @@ -0,0 +1 @@ +Subproject commit 9d3a0b3783afab335a1819543d039cf2980577bb -- 2.13.1.611.g7e3b11ae1 ``` ## Ævar Arnfjörð Bjarmason, 2017-06-27 20:33 Subject: [PATCH v2 3/3] sha1collisiondetection: automatically enable when submodule is populated Message-ID: <20170627203348.19827-4-avarab@gmail.com> URL: https://gitlist.dev/e/20170627203348.19827-4-avarab%40gmail.com In-Reply-To: <20170627203348.19827-1-avarab@gmail.com> ``` From: Junio C Hamano If a user wants to experiment with the version of collision detecting sha1 from the submodule, the user needed to not just populate the submodule but also needed to turn the knob. A Makefile trick is easy enough to do so, so let's do this. When somebody with a copy of the submodule populated wants not to use it, that can be done by overriding it in config.mak or from the command line. Signed-off-by: Junio C Hamano Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Makefile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Makefile b/Makefile index f0cac1f246..3baa1eac0b 100644 --- a/Makefile +++ b/Makefile @@ -1009,6 +1009,10 @@ EXTLIBS = GIT_USER_AGENT = git/$(GIT_VERSION) +ifeq ($(wildcard sha1collisiondetection/lib/sha1.h),sha1collisiondetection/lib/sha1.h) +DC_SHA1_SUBMODULE = auto +endif + include config.mak.uname -include config.mak.autogen -include config.mak -- 2.13.1.611.g7e3b11ae1 ``` ## Ævar Arnfjörð Bjarmason, 2017-06-27 20:33 Subject: [PATCH v2 0/3] update sha1dc from PR #36 Message-ID: <20170627203348.19827-1-avarab@gmail.com> URL: https://gitlist.dev/e/20170627203348.19827-1-avarab%40gmail.com In-Reply-To: <20170627121718.12078-1-avarab@gmail.com> ``` A v2 addressing comments to the v1, tbdiff below. Just fixes the subject line on 1/3 & gets rid of the redundant _BIG_ENDIAN detection for Solaris, we can just use __sparc (and indeed we did before this). Junio C Hamano (1): sha1collisiondetection: automatically enable when submodule is populated Ævar Arnfjörð Bjarmason (2): sha1dc: correct endian detection for Solaris (and others?) sha1dc: optionally use sha1collisiondetection as a submodule .gitmodules | 4 +++ Makefile | 16 +++++++++++ hash.h | 4 +++ sha1collisiondetection | 1 + sha1dc/sha1.c | 73 +++++++++++++++++++++++++++++++++++++------------- 5 files changed, 79 insertions(+), 19 deletions(-) create mode 100644 .gitmodules create mode 160000 sha1collisiondetection @@ -1,6 +1,6 @@ Author: Ævar Arnfjörð Bjarmason - sha1dc: update from my PR #36 + sha1dc: correct endian detection for Solaris (and others?) Update sha1dc from my PR #36[1] which'll hopefully be integrated by upstream soon. @@ -96,22 +96,15 @@ + +#if defined(SHA1DC_ON_INTEL_LIKE_PROCESSOR) +/* -+ * As a last resort before we fall back on _BIG_ENDIAN or whatever -+ * else we're not 100% sure about below, we blacklist specific -+ * processors here. We could add more, see -+ * e.g. https://wiki.debian.org/ArchitectureSpecificsMemo ++ * As a last resort before we do anything else we're not 100% sure ++ * about below, we blacklist specific processors here. We could add ++ * more, see e.g. https://wiki.debian.org/ArchitectureSpecificsMemo + */ +#else /* Not under GCC-alike or glibc or or */ + -+#ifdef _BIG_ENDIAN -+/* -+ * Solaris / illumos defines either _LITTLE_ENDIAN or _BIG_ENDIAN in -+ * . -+ */ -+#define SHA1DC_BIGENDIAN -+#else ++/* We do nothing more here for now */ +/*#error "Uncomment this to see if you fall through all the detection"*/ -+#endif /* Big Endian because of _BIG_ENDIAN (Solaris)*/ ++ +#endif /* !SHA1DC_ON_INTEL_LIKE_PROCESSOR */ +#endif /* Big Endian under whitelist of processors */ +#endif /* Big Endian under glibc */ 2: 63a5d0cb2a ! 2: 28750d3c24 sha1dc: optionally use sha1collisiondetection as a submodule @@ -79,4 +79,4 @@ --- /dev/null +++ b/sha1collisiondetection @@ -+Subproject commit 56ab30c4c998e1e7f3075705087a2f0c4c4202d7 ++Subproject commit 9d3a0b3783afab335a1819543d039cf2980577bb -- 2.13.1.611.g7e3b11ae1 ``` ## Junio C Hamano, 2017-06-27 21:37 Subject: Re: [PATCH v2 0/3] update sha1dc from PR #36 Message-ID: URL: https://gitlist.dev/e/xmqqh8z1yuve.fsf%40gitster.mtv.corp.google.com In-Reply-To: <20170627203348.19827-1-avarab@gmail.com> ``` Ævar Arnfjörð Bjarmason writes: > A v2 addressing comments to the v1, tbdiff below. Just fixes the > subject line on 1/3 & gets rid of the redundant _BIG_ENDIAN detection > for Solaris, we can just use __sparc (and indeed we did before this). Thanks. Let's have this tested by minority platform folks to help the upstream digest your pull request ;-) ``` ## Junio C Hamano, 2017-06-27 21:38 Subject: Re: [PATCH v2 0/3] update sha1dc from PR #36 Message-ID: URL: https://gitlist.dev/e/xmqqfuelyuuu.fsf%40gitster.mtv.corp.google.com In-Reply-To: <20170627203348.19827-1-avarab@gmail.com> ``` Ævar Arnfjörð Bjarmason writes: > A v2 addressing comments to the v1, tbdiff below. Just fixes the > subject line on 1/3 & gets rid of the redundant _BIG_ENDIAN detection > for Solaris, we can just use __sparc (and indeed we did before this). Thanks. Let's have this tested by minority platform folks to help the upstream digest your pull request. The change since the last round makes perfect sense to me. Thanks. ``` ## Ævar Arnfjörð Bjarmason, 2017-06-27 22:24 Subject: Re: [PATCH v2 0/3] update sha1dc from PR #36 Message-ID: <87mv8tjchi.fsf@gmail.com> URL: https://gitlist.dev/e/87mv8tjchi.fsf%40gmail.com In-Reply-To: ``` On Tue, Jun 27 2017, Junio C. Hamano jotted: > Ævar Arnfjörð Bjarmason writes: > >> A v2 addressing comments to the v1, tbdiff below. Just fixes the >> subject line on 1/3 & gets rid of the redundant _BIG_ENDIAN detection >> for Solaris, we can just use __sparc (and indeed we did before this). > > Thanks. Let's have this tested by minority platform folks to help > the upstream digest your pull request. The change since the last > round makes perfect sense to me. I got someone on #cygwin on Freenode to test this for me (compile git + t0013-sha1dc.sh), it passed, ditto for 2x people testing on Solaris SPARC. Updated a comment on the PR with that: https://github.com/cr-marcstevens/sha1collisiondetection/pull/36#issuecomment-311502713 I'm pretty sure this un-breaks everything & doesn't regress on cases where we fixed things earlier. ``` ## Ævar Arnfjörð Bjarmason, 2017-06-28 21:42 Subject: Re: [PATCH v2 0/3] update sha1dc from PR #36 Message-ID: <87h8yzkcv5.fsf@gmail.com> URL: https://gitlist.dev/e/87h8yzkcv5.fsf%40gmail.com In-Reply-To: ``` On Tue, Jun 27 2017, Junio C. Hamano jotted: > Ævar Arnfjörð Bjarmason writes: > >> A v2 addressing comments to the v1, tbdiff below. Just fixes the >> subject line on 1/3 & gets rid of the redundant _BIG_ENDIAN detection >> for Solaris, we can just use __sparc (and indeed we did before this). > > Thanks. Let's have this tested by minority platform folks to help > the upstream digest your pull request. The change since the last > round makes perfect sense to me. Just a brief update. This has been found not to work on some BSDs (with older GCC), but there's a patch being discussed at https://github.com/cr-marcstevens/sha1collisiondetection/pull/34 The discussion is sorted out, just waiting on n-soda to submit a patch (or to tell me he'd like me to), then the known issues will be fixed & it should be merged quickly, then I'll submit a v3. ``` ## Junio C Hamano, 2017-06-28 22:02 Subject: Re: [PATCH v2 0/3] update sha1dc from PR #36 Message-ID: URL: https://gitlist.dev/e/xmqqh8yzvkis.fsf%40gitster.mtv.corp.google.com In-Reply-To: <87h8yzkcv5.fsf@gmail.com> ``` Ævar Arnfjörð Bjarmason writes: > On Tue, Jun 27 2017, Junio C. Hamano jotted: > >> Ævar Arnfjörð Bjarmason writes: >> >>> A v2 addressing comments to the v1, tbdiff below. Just fixes the >>> subject line on 1/3 & gets rid of the redundant _BIG_ENDIAN detection >>> for Solaris, we can just use __sparc (and indeed we did before this). >> >> Thanks. Let's have this tested by minority platform folks to help >> the upstream digest your pull request. The change since the last >> round makes perfect sense to me. > > Just a brief update. This has been found not to work on some BSDs (with > older GCC), but there's a patch being discussed at > https://github.com/cr-marcstevens/sha1collisiondetection/pull/34 > > The discussion is sorted out, just waiting on n-soda to submit a patch > (or to tell me he'd like me to), then the known issues will be fixed & > it should be merged quickly, then I'll submit a v3. Thanks. It's good that I learned to pretend that I didn't hear anything like I'm pretty sure this un-breaks everything & doesn't regress on cases where we fixed things earlier. from anybody, and instead wait a bit longer ;-) ```