{"thread":{"id":"48973","subject":"git broken for AIX somewhere between 2.13.2 and 2.13.3","startedAt":"2018-07-29T17:09:16Z","lastAt":"2018-08-02T21:32:23Z","messageCount":25,"participants":["Michael","brian m. carlson","Ævar Arnfjörð Bjarmason","Andreas Schwab","Junio C Hamano","Daniel Shumow","Michael Felt","Eric Wong","Michael Felt (aixtools)","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"353827","messageId":"fb935882-25b1-db5f-d765-50dab297f733@felt.demon.nl","threadId":"48973","inReplyTo":null,"subject":"git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"Michael","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-29T16:44:26Z","receivedAt":"2018-07-29T17:09:16Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"Hi,\n\nSeveral years ago I had downloaded and packaged git-2.10.1 with no real \nissues, and it has been working fine. Deciding it was time for an update \nI downloaded git-2.18.0 and built from scratch.\n\nAfter testing lots of version - the the last 2.12 version worked; \ngit-2.13.2 works but git-2.13.3 does not.\n\nroot@x066:[/tmp/xxx]git clone git@github.com:aixtools/hello-world.git\nroot@x066:[/tmp/xxx]git --version\ngit version 2.13.2\nroot@x066:[/tmp/xxx]ls -l\ntotal 0\ndrwxr-xr-x    3 root     system          256 Jul 29 16:37 hello-world\n\nroot@x066:[/tmp/xxx]git --version\ngit version 2.13.3\nroot@x066:[/tmp/xxx]git clone git@github.com:aixtools/hello-world.git\nCloning into 'hello-world'...\nremote: Counting objects: 3, done.\nremote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3\nReceiving objects: 100% (3/3), done.\nfatal: pack is corrupted (SHA1 mismatch)\nfatal: index-pack failed\n\np.s. - what surprises me re: git-2.13.2 - no messages about 'Cloning \ninto ...', which version 2.13.1 did give.\n\nI guess a bisect is the next step - between version 2.13.2 and 2.13.3. \nOther suggestions welcome!\n\nMichael\n\n"},{"id":"353828","messageId":"20180729181006.GC945730@genre.crustytoothpaste.net","threadId":"48973","inReplyTo":"fb935882-25b1-db5f-d765-50dab297f733@felt.demon.nl","subject":"Re: git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-07-29T18:10:06Z","receivedAt":"2018-07-29T18:10:17Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Jul 29, 2018 at 06:44:26PM +0200, Michael wrote:\n> root@x066:[/tmp/xxx]git --version\n> git version 2.13.3\n> root@x066:[/tmp/xxx]git clone git@github.com:aixtools/hello-world.git\n> Cloning into 'hello-world'...\n> remote: Counting objects: 3, done.\n> remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3\n> Receiving objects: 100% (3/3), done.\n> fatal: pack is corrupted (SHA1 mismatch)\n> fatal: index-pack failed\n> \n> p.s. - what surprises me re: git-2.13.2 - no messages about 'Cloning into\n> ...', which version 2.13.1 did give.\n> \n> I guess a bisect is the next step - between version 2.13.2 and 2.13.3. Other\n> suggestions welcome!\n\nAre you using SHA1DC on that system, and does compiling with another\nSHA-1 implementation help? There was a change to the SHA1DC code big\nendian detection in that commit, which might be the cause of your\nproblems if you're using a POWER or PowerPC system.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"353835","messageId":"466c7a59-0535-e72c-6c90-5288a53b59a0@felt.demon.nl","threadId":"48973","inReplyTo":"20180729181006.GC945730@genre.crustytoothpaste.net","subject":"Re: git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"Michael","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-29T19:46:38Z","receivedAt":"2018-07-29T19:46:47Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"On 29/07/2018 20:10, brian m. carlson wrote:\n> On Sun, Jul 29, 2018 at 06:44:26PM +0200, Michael wrote:\n>> root@x066:[/tmp/xxx]git --version\n>> git version 2.13.3\n>> root@x066:[/tmp/xxx]git clone git@github.com:aixtools/hello-world.git\n>> Cloning into 'hello-world'...\n>> remote: Counting objects: 3, done.\n>> remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3\n>> Receiving objects: 100% (3/3), done.\n>> fatal: pack is corrupted (SHA1 mismatch)\n>> fatal: index-pack failed\n>>\n>> p.s. - what surprises me re: git-2.13.2 - no messages about 'Cloning into\n>> ...', which version 2.13.1 did give.\n>>\n>> I guess a bisect is the next step - between version 2.13.2 and 2.13.3. Other\n>> suggestions welcome!\n> Are you using SHA1DC on that system, and does compiling with another\n> SHA-1 implementation help? There was a change to the SHA1DC code big\n> endian detection in that commit, which might be the cause of your\n> problems if you're using a POWER or PowerPC system.\n\nI was thinking it might be an 'endian' issue. So, yes - AIX runs on \nPOWER, only as BigEndian.\n\ngit bisect returns:\n\nmichael@x071:[/data/prj/aixtools/git/github/git-master]git bisect bad\nBisecting: 1 revision left to test after this (roughly 1 step)\n[35049a2343948f686861e176a8c395f9f67da7b6] Merge branch \n'aw/contrib-subtree-doc-asciidoctor' into maint\nmichael@x071:[/data/prj/aixtools/git/github/git-master]git bisect good\nBisecting: 0 revisions left to test after this (roughly 0 steps)\n[9936c1b52a39fa14fca04f937df3e75f7498ac66] sha1dc: update from upstream\n\n\nmichael@x071:[/data/prj/aixtools/git/github/git-master]git bisect bad\n9936c1b52a39fa14fca04f937df3e75f7498ac66 is the first bad commit\ncommit 9936c1b52a39fa14fca04f937df3e75f7498ac66\nAuthor: Ævar Arnfjörð Bjarmason \nDate:   Sat Jul 1 22:05:45 2017 +0000\n\n     sha1dc: update from upstream\n\n     Update sha1dc from the latest version by the upstream maintainer[1].\n\n     See commit 6b851e536b (\"sha1dc: update from upstream\", 2017-06-06) for\n     the last update.\n\n     This solves the Big Endian detection on Solaris reported against\n     v2.13.2[2], hopefully without any regressions. A version of this has\n     been tested on two Solaris SPARC installations, Cygwin (by jturney on\n     cygwin@Freenode), and on numerous more boring systems (mainly\n     linux/x86_64). See [3] for a discussion of the implementation and\n     platform-specific issues.\n\n     See commit a0103914c2 (\"sha1dc: update from upstream\", 2017-05-20) and\n     6b851e536b (\"sha1dc: update from upstream\", 2017-06-06) for previous\n     attempts in the 2.13 series to address various compile-time feature\n     detection in this library.\n\n     1. \nhttps://github.com/cr-marcstevens/sha1collisiondetection/commit/19d97bf5af05312267c2e874ee6bcf584d9e9681 \n\n     2. \n\n(https://public-inbox.org/git/CAKKM46tHq13XiW5C8sux3=PZ1VHSu_npG8ExfWwcPD7rkZkyRQ@mail.gmail.com/) \n\n     3. https://github.com/cr-marcstevens/sha1collisiondetection/pull/34\n\n     Signed-off-by: Ævar Arnfjörð Bjarmason \n     Signed-off-by: Junio C Hamano \n\n:040000 040000 a84797967fb742e4ca9618a641d53ce3a6c6589b \n32efa656d78901da961e4a47d84b6d82fede064b M      sha1dc\n\n"},{"id":"353836","messageId":"dfe374bf-d9de-8dad-6ec9-4edfa3e9b12b@felt.demon.nl","threadId":"48973","inReplyTo":"20180729192753.GD945730@genre.crustytoothpaste.net","subject":"Re: git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"Michael","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-29T19:48:43Z","receivedAt":"2018-07-29T19:48:50Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"On 29/07/2018 21:27, brian m. carlson wrote:\n> On Sun, Jul 29, 2018 at 08:59:39PM +0200, Michael wrote:\n>> On 29/07/2018 20:10, brian m. carlson wrote:\n>>> Are you using SHA1DC on that system, and does compiling with another\n>>> SHA-1 implementation help? There was a change to the SHA1DC code big\n>>> endian detection in that commit, which might be the cause of your\n>>> problems if you're using a POWER or PowerPC system.\n>> I was thinking it might be an 'endian' issue. So, yes - AIX runs on POWER,\n>> only as BigEndian.\n> Well, that explains it. I would recommend submitting a patch to\n> https://github.com/cr-marcstevens/sha1collisiondetection, and the we can\n> pull in the updated submodule with that fix.\nNot sure I am smart enough to do that. I'll have to download, build, and \nsee what it says.\n>\n> In the mean time, you could build using OpenSSL or the block SHA-1\n> implementation, and switch back once things are in a good state. I do\n> recommend using SHA1DC for things long term, though, as attacks on SHA-1\n> are only going to get better.\nAny suggestions on where/how to do this?\n\nroot@x066:[/data/prj/aixtools/git/git-2.13.2]./configure --help | grep \n-i sha\n   --sharedstatedir=DIR    modifiable architecture-independent data \n[PREFIX/com]\n   --datarootdir=DIR       read-only arch.-independent data root \n[PREFIX/share]\n\nroot@x066:[/data/prj/aixtools/git/git-2.13.2]./configure --help | grep ssl\n   --with-openssl          use OpenSSL library (default is YES)\n                           ARG can be prefix for openssl library and \nheaders\n\n\n"},{"id":"353837","messageId":"87zhy9u7rz.fsf@evledraar.gmail.com","threadId":"48973","inReplyTo":"466c7a59-0535-e72c-6c90-5288a53b59a0@felt.demon.nl","subject":"Re: git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-07-29T20:05:52Z","receivedAt":"2018-07-29T20:05:59Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jul 29 2018, Michael wrote:\n\n> On 29/07/2018 20:10, brian m. carlson wrote:\n>> On Sun, Jul 29, 2018 at 06:44:26PM +0200, Michael wrote:\n>>> root@x066:[/tmp/xxx]git --version\n>>> git version 2.13.3\n>>> root@x066:[/tmp/xxx]git clone git@github.com:aixtools/hello-world.git\n>>> Cloning into 'hello-world'...\n>>> remote: Counting objects: 3, done.\n>>> remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3\n>>> Receiving objects: 100% (3/3), done.\n>>> fatal: pack is corrupted (SHA1 mismatch)\n>>> fatal: index-pack failed\n>>>\n>>> p.s. - what surprises me re: git-2.13.2 - no messages about 'Cloning into\n>>> ...', which version 2.13.1 did give.\n>>>\n>>> I guess a bisect is the next step - between version 2.13.2 and 2.13.3. Other\n>>> suggestions welcome!\n>> Are you using SHA1DC on that system, and does compiling with another\n>> SHA-1 implementation help? There was a change to the SHA1DC code big\n>> endian detection in that commit, which might be the cause of your\n>> problems if you're using a POWER or PowerPC system.\n>\n> I was thinking it might be an 'endian' issue. So, yes - AIX runs on\n> POWER, only as BigEndian.\n>\n> git bisect returns:\n>\n> michael@x071:[/data/prj/aixtools/git/github/git-master]git bisect bad\n> Bisecting: 1 revision left to test after this (roughly 1 step)\n> [35049a2343948f686861e176a8c395f9f67da7b6] Merge branch\n> 'aw/contrib-subtree-doc-asciidoctor' into maint\n> michael@x071:[/data/prj/aixtools/git/github/git-master]git bisect good\n> Bisecting: 0 revisions left to test after this (roughly 0 steps)\n> [9936c1b52a39fa14fca04f937df3e75f7498ac66] sha1dc: update from upstream\n>\n>\n> michael@x071:[/data/prj/aixtools/git/github/git-master]git bisect bad\n> 9936c1b52a39fa14fca04f937df3e75f7498ac66 is the first bad commit\n> commit 9936c1b52a39fa14fca04f937df3e75f7498ac66\n> Author: Ævar Arnfjörð Bjarmason \n> Date: Sat Jul 1 22:05:45 2017 +0000\n>\n> sha1dc: update from upstream\n>\n> Update sha1dc from the latest version by the upstream maintainer[1].\n>\n> See commit 6b851e536b (\"sha1dc: update from upstream\", 2017-06-06) for\n> the last update.\n>\n> This solves the Big Endian detection on Solaris reported against\n> v2.13.2[2], hopefully without any regressions. A version of this has\n> been tested on two Solaris SPARC installations, Cygwin (by jturney on\n> cygwin@Freenode), and on numerous more boring systems (mainly\n> linux/x86_64). See [3] for a discussion of the implementation and\n> platform-specific issues.\n>\n> See commit a0103914c2 (\"sha1dc: update from upstream\", 2017-05-20) and\n> 6b851e536b (\"sha1dc: update from upstream\", 2017-06-06) for previous\n> attempts in the 2.13 series to address various compile-time feature\n> detection in this library.\n>\n>\n> 1. https://github.com/cr-marcstevens/sha1collisiondetection/commit/19d97bf5af05312267c2e874ee6bcf584d9e9681\n>\n>\n> 2. \n> (https://public-inbox.org/git/CAKKM46tHq13XiW5C8sux3=PZ1VHSu_npG8ExfWwcPD7rkZkyRQ@mail.gmail.com/)\n>\n> 3. https://github.com/cr-marcstevens/sha1collisiondetection/pull/34\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Junio C Hamano \n>\n> :040000 040000 a84797967fb742e4ca9618a641d53ce3a6c6589b\n> 32efa656d78901da961e4a47d84b6d82fede064b M sha1dc\n\nSorry about that. As can be seen from those PRs and the \"git log\"\ndetecting whether something is big endian or not can be quite tricky, we\nfigured out how to do it on both BSD and Solaris, but apparently broke\nAIX as a result.\n\nYou should be able to define -DSHA1DC_FORCE_LITTLEENDIAN or\n-DSHA1DC_FORCE_BIGENDIAN (looks like you'll need the latter) to get the\nlatest version to compile, but I and upstream cr-marcstevens would be\nvery interested to know from someone who knows AIX how this broke.\n\nAlso, to you and anyone else with access to AIX: I'd be happy to figure\nthese issues out pro-actively if you give me a login to an AIX\nmachine. I promise not to do anything except compile/debug/test git on\nit.\n\nI used to have access to an AIX box through a previous job ages ago,\nit's a very interesting OS and like with Solaris it's easy to discover a\nlot of portability issues.\n"},{"id":"353838","messageId":"20180729200623.GF945730@genre.crustytoothpaste.net","threadId":"48973","inReplyTo":"dfe374bf-d9de-8dad-6ec9-4edfa3e9b12b@felt.demon.nl","subject":"Re: git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-07-29T20:06:24Z","receivedAt":"2018-07-29T20:06:34Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Jul 29, 2018 at 09:48:43PM +0200, Michael wrote:\n> On 29/07/2018 21:27, brian m. carlson wrote:\n> > Well, that explains it. I would recommend submitting a patch to\n> > https://github.com/cr-marcstevens/sha1collisiondetection, and the we can\n> > pull in the updated submodule with that fix.\n> Not sure I am smart enough to do that. I'll have to download, build, and see\n> what it says.\n\nThe issue is that somewhere in lib/sha1.c, you need to cause\nSHA1DC_BIGENDIAN to be set. That means you need to figure out what\ncompiler macro might indicate that. I can tell you that a POWER- or\nPowerPC-specific one is going to be a bad choice unless it includes the\nendianness, since those chips come in little-endian versions as well.\n\n_AIX might be a fine choice if you know that it only ever runs on\nbig-endian chips.\n\n> > In the mean time, you could build using OpenSSL or the block SHA-1\n> > implementation, and switch back once things are in a good state. I do\n> > recommend using SHA1DC for things long term, though, as attacks on SHA-1\n> > are only going to get better.\n> Any suggestions on where/how to do this?\n> \n> root@x066:[/data/prj/aixtools/git/git-2.13.2]./configure --help | grep -i\n> sha\n>   --sharedstatedir=DIR    modifiable architecture-independent data\n> [PREFIX/com]\n>   --datarootdir=DIR       read-only arch.-independent data root\n> [PREFIX/share]\n> \n> root@x066:[/data/prj/aixtools/git/git-2.13.2]./configure --help | grep ssl\n>   --with-openssl          use OpenSSL library (default is YES)\n>                           ARG can be prefix for openssl library and headers\n\nIf you're using configure, you can use --with-openssl, or\n--with-openssl=PREFIX if your OpenSSL isn't in the standard location but\nis instead in PREFIX.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"353839","messageId":"701d9f4b-efbd-c584-4bec-bddb51b11d96@felt.demon.nl","threadId":"48973","inReplyTo":"20180729200623.GF945730@genre.crustytoothpaste.net","subject":"Re: git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"Michael","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-29T20:50:10Z","receivedAt":"2018-07-29T20:50:18Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"On 29/07/2018 22:06, brian m. carlson wrote:\n> On Sun, Jul 29, 2018 at 09:48:43PM +0200, Michael wrote:\n>> On 29/07/2018 21:27, brian m. carlson wrote:\n>>> Well, that explains it. I would recommend submitting a patch to\n>>> https://github.com/cr-marcstevens/sha1collisiondetection, and the we can\n>>> pull in the updated submodule with that fix.\n>> Not sure I am smart enough to do that. I'll have to download, build, and see\n>> what it says.\n> The issue is that somewhere in lib/sha1.c, you need to cause\n> SHA1DC_BIGENDIAN to be set. That means you need to figure out what\n> compiler macro might indicate that.\nI remember - roughly - a few decades back - having an assignment to \nwrite code to determine endianness. PDP and VAC were different iirc, and \nmany other micro-processors besides the 8088/8086/z85/68k/etc..\n\nIf you are looking for a compiler macro as a way to determine this - \nmaybe you have one for gcc, but not for xlc. I do not know it - currently :)\n\n> I can tell you that a POWER- or\n> PowerPC-specific one is going to be a bad choice unless it includes the\n> endianness, since those chips come in little-endian versions as well.\nActually, the POWER8 and POWER9 (and I expect all future versions) \nsupport both. There are not separate chips. Per virtual machine - a mode \nis determined during boot (virtual power on) e.g., SLES11 only ran in \nBigEndian and SLES12 only runs in LittleEndian. afaik, RHEL was \nsupplying both BE and LE distributions. AIX, as an OS, only runs in BE \nmode, and I expect IBM i (was os/400) is also only running in BE.\n>\n> _AIX might be a fine choice if you know that it only ever runs on\n> big-endian chips.\nDo you mean just testing for _AIX. That would be very very easy - yes!\n>>> In the mean time, you could build using OpenSSL or the block SHA-1\n>>> implementation, and switch back once things are in a good state. I do\n>>> recommend using SHA1DC for things long term, though, as attacks on SHA-1\n>>> are only going to get better.\n>> Any suggestions on where/how to do this?\n>>\n>> root@x066:[/data/prj/aixtools/git/git-2.13.2]./configure --help | grep -i\n>> sha\n>>   --sharedstatedir=DIR    modifiable architecture-independent data\n>> [PREFIX/com]\n>>   --datarootdir=DIR       read-only arch.-independent data root\n>> [PREFIX/share]\n>>\n>> root@x066:[/data/prj/aixtools/git/git-2.13.2]./configure --help | grep ssl\n>>   --with-openssl          use OpenSSL library (default is YES)\n>>                           ARG can be prefix for openssl library and headers\n> If you're using configure, you can use --with-openssl, or\n> --with-openssl=PREFIX if your OpenSSL isn't in the standard location but\n> is instead in PREFIX.\nI'll look in configure to see if it is not finding openssl. I was \nassuming it was found - as everything else using GNU \"auto\" tools find \nit okay. i.e., /var/lib/libssl.a, etc..\n\nTomorrow!\n\n\n"},{"id":"353846","messageId":"87in4x4t5y.fsf@igel.home","threadId":"48973","inReplyTo":"87zhy9u7rz.fsf@evledraar.gmail.com","subject":"Re: git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2018-07-29T21:40:41Z","receivedAt":"2018-07-29T21:40:47Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Jul 29 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> Also, to you and anyone else with access to AIX: I'd be happy to figure\n> these issues out pro-actively if you give me a login to an AIX\n> machine. I promise not to do anything except compile/debug/test git on\n> it.\n\nThe GCC compile farm has a machine\nrunning AIX, and is free to use for anyone working on free software.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"353853","messageId":"ddf55879-8c53-a172-adc6-c8942f1a1818@felt.demon.nl","threadId":"48973","inReplyTo":"87in4x4t5y.fsf@igel.home","subject":"Re: git broken for AIX somewhere between 2.13.2 and 2.13.3","fromName":"Michael","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-30T06:22:12Z","receivedAt":"2018-07-30T06:22:30Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"On 29/07/2018 23:40, Andreas Schwab wrote:\n> On Jul 29 2018, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Also, to you and anyone else with access to AIX: I'd be happy to figure\n>> these issues out pro-actively if you give me a login to an AIX\n>> machine. I promise not to do anything except compile/debug/test git on\n>> it.\n> The GCC compile farm has a machine\n> running AIX, and is free to use for anyone working on free software.\nMy goal is less \"to work on\", more, \"to package\" and/or \"to work with\". \nMost others, including IBM, seem to use gcc as compiler - which is fine. \nHowever, on AIX I often see side-effects introduced by the GNU run-time \nenvironment needed on top of the xlc.rte provided as part of AIX.\n\nIn any case - my testing is using the xlc complier - so there are syntax \ndifferences. the compiler has many modes - by default I use 'xlc_r' that \nhas the following default settings:\nxlc_r:  use        = DEFLT_C\n         crt        = /lib/crt0.o\n         mcrt       = /lib/mcrt0.o\n         gcrt       = /lib/gcrt0.o\n         libraries  = -L/usr/vac/lib,-lxlopt,-lxlipa,-lxl,-lpthreads,-lc\n         proflibs   = -L/lib/profiled,-L/usr/lib/profiled\n         hdlibs     = -L/usr/vac/lib,-lhmd\n         options    = \n-qlanglvl=extc99,-qcpluscmt,-qkeyword=inline,-qalias=ansi,-qthreaded,-D_THREAD_SAFE,-D__VACPP_MULTI__\n\nDEFLT_C (for the curious, the default options is perhaps most \ninteresting) is:\n* common definitions\n\nDEFLT_C:\n         use           =DEFLT\n         xlurt_cfg_path=/usr/vac/urt\n         xlurt_cfg_name=urt_client.cfg\n\nDEFLT_CPP:\n         use           =DEFLT\n         xlurt_cfg_path=/usr/vacpp/urt\n         xlurt_cfg_name=urt_client.cfg\n\nDEFLT:\n         cppcomp   = /usr/vacpp/exe/xlCentry\n         ccomp     = /usr/vac/exe/xlcentry\n         code      = /usr/vac/exe/xlCcode\n         cpp       = /usr/vac/exe/xlCcpp\n         munch     = /usr/vacpp/exe/munch\n         dis       = /usr/vac/exe/dis\n         xlC       = /usr/vac/bin/xlc\n         list      = /usr/vac/exe/xllist\n         xslt      = /usr/vac/exe/XALAN\n         transforms = /usr/vac/listings\n         listlibs  = /usr/vac/lib\n         cppinc    = /usr/vacpp/include\n         ipa       = /usr/vac/exe/ipa\n         cppfilt   = /usr/vacpp/bin/c++filt\n         bolt      = /usr/vac/exe/bolt\n         as        = /bin/as\n         ld        = /bin/ld\n         artool    = /bin/ar\n         options   = \n-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_IBMR2,-D_POWER\n         options32 = -bpT:0x10000000,-bpD:0x20000000\n         options32_bmaxdata = -bpT:0x10000000,-bpD:0x30000000\n         options64 = -bpT:0x100000000,-bpD:0x110000000\n         ldopt     = \"b:o:e:u:R:H:Y:Z:L:T:A:k:j:\"\n         hdlibs    = -L/usr/vac/lib,-lhmd\n         xlCcopt   = \n-qlanglvl=extc99,-qcpluscmt,-qkeyword=inline,-qalias=ansi\n         crt_64    = /lib/crt0_64.o\n         mcrt_64   = /lib/mcrt0_64.o\n         gcrt_64   = /lib/gcrt0_64.o\n         smplibraries = -lxlsmp\n         palibraries  = -L/usr/vatools/lib,-lpahooks\n         resexp     = /usr/vacpp/lib/res.exp\n         genexports = /usr/vac/bin/CreateExportList\n         vac_path   = /usr/vac\n         vacpp_path = /usr/vacpp\n         xlcmp_path = /usr/vac:/usr/vacpp\n         xlc_c_stdinc   = /usr/vac/include:/usr/include\n         xlc_cpp_stdinc = /usr/vacpp/include:/usr/include\n         xlurt_msg_cat_name=vacumsg.cat\n         __GNUC_MINOR__ = 3\n\n\n>\n> Andreas.\n>\n\n"},{"id":"353860","messageId":"87wotdt649.fsf@evledraar.gmail.com","threadId":"48973","inReplyTo":"701d9f4b-efbd-c584-4bec-bddb51b11d96@felt.demon.nl","subject":"Is detecting endianness at compile-time unworkable?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-07-30T09:39:18Z","receivedAt":"2018-07-30T09:39:23Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jul 29 2018, Michael wrote:\n\n> On 29/07/2018 22:06, brian m. carlson wrote:\n>> On Sun, Jul 29, 2018 at 09:48:43PM +0200, Michael wrote:\n>>> On 29/07/2018 21:27, brian m. carlson wrote:\n>>>> Well, that explains it. I would recommend submitting a patch to\n>>>> https://github.com/cr-marcstevens/sha1collisiondetection, and the we can\n>>>> pull in the updated submodule with that fix.\n>>> Not sure I am smart enough to do that. I'll have to download, build, and see\n>>> what it says.\n>> The issue is that somewhere in lib/sha1.c, you need to cause\n>> SHA1DC_BIGENDIAN to be set. That means you need to figure out what\n>> compiler macro might indicate that.\n> I remember - roughly - a few decades back - having an assignment to\n> write code to determine endianness. PDP and VAC were different iirc,\n> and many other micro-processors besides the 8088/8086/z85/68k/etc..\n>\n> If you are looking for a compiler macro as a way to determine this -\n> maybe you have one for gcc, but not for xlc. I do not know it - currently :)\n\nI'm not familiar with AIX, but from searching around I found this\nporting manual from IBM:\nhttp://www.redbooks.ibm.com/redbooks/pdfs/sg246034.pdf\n\nThere they suggest either defining your own macros, or testing the\nmemory layout at runtime (see section \"2.2.2.3 Technique 3: Testing\nmemory layout\" and surrounding sections).\n\nPerhaps it's worth taking a step back here and thinking about whether\nthis whole thing is unworkable. It was hard enough to get this to work\non the combination of Linux, *BSD and Solaris, but I suspect we'll run\ninto increasingly obscure platforms where this is hard or impossible\n(AIX, HP/UX etc.)\n\nThe reason we're in this hole is because we use this\nsha1collisiondetection library to do SHA-1, and the reason we have\nissues with it specifically (not OpenSSL et al) is because its only\nmethod of detecting endianness is at compile time.\n\nThis didn't use to be the case, it was changed in this commit:\nhttps://github.com/cr-marcstevens/sha1collisiondetection/commit/d597672\n\nDan Shumow: Since the commit message doesn't say why, can you elaborate\na bit on why this was done, i.e. is determining this at runtime harmful\nfor performance? If not, perhaps it would be best to bring this back, at\nleast as an option.\n\nAnd, as an aside, the reason we can't easily make it better ourselves is\nbecause the build process for git.git doesn't have a facility to run\ncode to detect this type of stuff (the configure script is always\noptional). So we can't just run this test ourselves.\n\nJunio: I've barked up that particular tree before in\nhttps://public-inbox.org/git/87a7x3kmh5.fsf@evledraar.gmail.com/ and I\nwon't bore you all by repeating myself, except to say that this is yet\nanother case where I wish we had a hard dependency on some way of doing\nchecks via compiled code in our build system.\n"},{"id":"353882","messageId":"xmqq600wkc4i.fsf@gitster-ct.c.googlers.com","threadId":"48973","inReplyTo":"87wotdt649.fsf@evledraar.gmail.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-30T14:54:21Z","receivedAt":"2018-07-30T14:54:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> And, as an aside, the reason we can't easily make it better ourselves is\n> because the build process for git.git doesn't have a facility to run\n> code to detect this type of stuff (the configure script is always\n> optional). So we can't just run this test ourselves.\n\nIt won't help those who cross-compile anyway. I thought we declared\n\"we make a reasonable effort to guess the target endianness from the\nsystem header by inspecting usual macros, but will not aim to cover\nevery system on the planet---instead there is a knob to tweak it for\nthose on exotic platforms\" last time we discussed this?\n"},{"id":"353929","messageId":"xmqqh8kgfuba.fsf@gitster-ct.c.googlers.com","threadId":"48973","inReplyTo":"xmqq600wkc4i.fsf@gitster-ct.c.googlers.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-30T18:32:41Z","receivedAt":"2018-07-30T18:32:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> And, as an aside, the reason we can't easily make it better ourselves is\n>> because the build process for git.git doesn't have a facility to run\n>> code to detect this type of stuff (the configure script is always\n>> optional). So we can't just run this test ourselves.\n>\n> It won't help those who cross-compile anyway. I thought we declared\n> \"we make a reasonable effort to guess the target endianness from the\n> system header by inspecting usual macros, but will not aim to cover\n> every system on the planet---instead there is a knob to tweak it for\n> those on exotic platforms\" last time we discussed this?\n\nWell, having said all that, I do not think I personally mind if\n./configure learned to include a \"compile small program and run it\nto determine byte order on the build machine\" as part of \"we make a\nreasonable effort\" as long as it cleanly excludes cross building\ncase (and the result is made overridable just in case we misdetect\nthe \"cross-ness\" of the build).\n\n"},{"id":"353930","messageId":"CADMpRy7f_Zr7Ay7bU_5eBS+7yO-58EVsEGE0k9qxefS-TO+z8w@mail.gmail.com","threadId":"48973","inReplyTo":"xmqqh8kgfuba.fsf@gitster-ct.c.googlers.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Daniel Shumow","fromEmail":"shumow@gmail.com","sentAt":"2018-07-30T18:39:52Z","receivedAt":"2018-07-30T18:39:55Z","isPatch":false,"sender":{"key":"shumow@gmail.com","avatar":null},"body":"The change was definitely made for performance. Removing the if\nstatements, conditioned upon endianess was an approx 10% improvement, which\nwas very important to getting this library accepted into git.\n\nThanks,\nDan\n\n\nOn Mon, Jul 30, 2018 at 11:32 AM, Junio C Hamano wrote:\n\n> Junio C Hamano writes:\n>\n> > Ævar Arnfjörð Bjarmason writes:\n> >\n> >> And, as an aside, the reason we can't easily make it better ourselves is\n> >> because the build process for git.git doesn't have a facility to run\n> >> code to detect this type of stuff (the configure script is always\n> >> optional). So we can't just run this test ourselves.\n> >\n> > It won't help those who cross-compile anyway. I thought we declared\n> > \"we make a reasonable effort to guess the target endianness from the\n> > system header by inspecting usual macros, but will not aim to cover\n> > every system on the planet---instead there is a knob to tweak it for\n> > those on exotic platforms\" last time we discussed this?\n>\n> Well, having said all that, I do not think I personally mind if\n> ./configure learned to include a \"compile small program and run it\n> to determine byte order on the build machine\" as part of \"we make a\n> reasonable effort\" as long as it cleanly excludes cross building\n> case (and the result is made overridable just in case we misdetect\n> the \"cross-ness\" of the build).\n>\n>\n"},{"id":"354007","messageId":"599fc1eb-d295-fe63-4d86-168d569c876c@felt.demon.nl","threadId":"48973","inReplyTo":"CADMpRy7f_Zr7Ay7bU_5eBS+7yO-58EVsEGE0k9qxefS-TO+z8w@mail.gmail.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Michael Felt","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-31T10:06:11Z","receivedAt":"2018-07-31T10:06:31Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"I have just replied to \nhttps://github.com/cr-marcstevens/sha1collisiondetection/pull/42\n\nI checked a gcc compiler on AIX, and I have the defines for vac.\n\nI do not have access yet to SLES or RHEL (or Ubuntu), just a \"free \nDebian\" on my Power6.\n\n* my conclusions|recommendations:\n\na) AIX is always Big Endian, the define _AIX can be used to determine if AIX\n\nb) POWER7 and earlier are always Big Endian\n\nc) assuming lscpu is always available on Linux systems a command (in \nconfigure?) could be used:\n\nroot@x074:/usr/bin# lscpu | grep -i endian\nByte Order:            Big Endian\n\nd) some linux systems (in any case latest versions of RHEL and SLES \nenterprise) should have a file named lparcfg in /proc \n(/proc/{powerppc|ppc64|ppc64le|ppc64el}/lparcfg - and it might be in \nthat file. Need to get onto a (POWER8|POWER9) system to check.\n\nHope this helps:\n\ndetails re: define of _AIX\n\nroot@x068:[/data/httpd/gcc]gcc -dM -E - < /dev/null | grep AIX | head -1\n#define _AIX 1\n\nmichael@x071:[/home/michael]/usr/bin/grep -p DEFLT: \n/etc/vac.cfg.[567][123] | grep options\\\n         options   = \n-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_IBMR2,-D_POWER\n         options   = \n-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_IBMR2,-D_POWER\n         options   = \n-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_AIX71,-D_IBMR2,-D_POWER\n         options   = \n-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_AIX71,-D_AIX72,-D_IBMR2,-D_POWER\n\nmichael@x071:[/home/michael]ls /etc/vac.cfg.[567][123]\n/etc/vac.cfg.53  /etc/vac.cfg.61  /etc/vac.cfg.71  /etc/vac.cfg.72\n\n\nOn 7/30/2018 8:39 PM, Daniel Shumow wrote:\n> The change was definitely made for performance. Removing the if \n> statements, conditioned upon endianess was an approx 10% improvement, \n> which was very important to getting this library accepted into git.\n>\n> Thanks,\n> Dan\n>\n>\n> On Mon, Jul 30, 2018 at 11:32 AM, Junio C Hamano > wrote:\n>\n> Junio C Hamano > writes:\n>\n> > Ævar Arnfjörð Bjarmason > writes:\n> >\n> >> And, as an aside, the reason we can't easily make it better\n> ourselves is\n> >> because the build process for git.git doesn't have a facility\n> to run\n> >> code to detect this type of stuff (the configure script is always\n> >> optional). So we can't just run this test ourselves.\n> >\n> > It won't help those who cross-compile anyway.  I thought we declared\n> > \"we make a reasonable effort to guess the target endianness from the\n> > system header by inspecting usual macros, but will not aim to cover\n> > every system on the planet---instead there is a knob to tweak it for\n> > those on exotic platforms\" last time we discussed this?\n>\n> Well, having said all that, I do not think I personally mind if\n> ./configure learned to include a \"compile small program and run it\n> to determine byte order on the build machine\" as part of \"we make a\n> reasonable effort\" as long as it cleanly excludes cross building\n> case (and the result is made overridable just in case we misdetect\n> the \"cross-ness\" of the build).\n>\n>\n\n"},{"id":"354011","messageId":"93056823-2740-d072-1ebd-46b440b33d7e@felt.demon.nl","threadId":"48973","inReplyTo":"87wotdt649.fsf@evledraar.gmail.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Michael Felt","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-31T10:39:36Z","receivedAt":"2018-07-31T10:39:52Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"A small step back...\n\n\nOn 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Sun, Jul 29 2018, Michael wrote:\n>\n>> On 29/07/2018 22:06, brian m. carlson wrote:\n>>> On Sun, Jul 29, 2018 at 09:48:43PM +0200, Michael wrote:\n>>>> On 29/07/2018 21:27, brian m. carlson wrote:\n>>>>> Well, that explains it. I would recommend submitting a patch to\n>>>>> https://github.com/cr-marcstevens/sha1collisiondetection, and the we can\n>>>>> pull in the updated submodule with that fix.\n>>>> Not sure I am smart enough to do that. I'll have to download, build, and see\n>>>> what it says.\n>>> The issue is that somewhere in lib/sha1.c, you need to cause\n>>> SHA1DC_BIGENDIAN to be set. That means you need to figure out what\n>>> compiler macro might indicate that.\n>> I remember - roughly - a few decades back - having an assignment to\n>> write code to determine endianness. PDP and VAC were different iirc,\n>> and many other micro-processors besides the 8088/8086/z85/68k/etc..\n>>\n>> If you are looking for a compiler macro as a way to determine this -\n>> maybe you have one for gcc, but not for xlc. I do not know it - currently :)\n> I'm not familiar with AIX, but from searching around I found this\n> porting manual from IBM:\n> http://www.redbooks.ibm.com/redbooks/pdfs/sg246034.pdf\nThis is from July 2001 - when AIX 5L, for Linux affinity, was new. AIX \nwas (nearly) the #1 posix system, and linux was a minor player in the \ndata center (in or out (now as IAAS)). IMHO, the recommendations made in \n2001 are probably no longer applicable (64-bit was fairly new, e.g., \nrather than common).\n>\n> There they suggest either defining your own macros, or testing the\n> memory layout at runtime (see section \"2.2.2.3 Technique 3: Testing\n> memory layout\" and surrounding sections).\n>\n> Perhaps it's worth taking a step back here and thinking about whether\n> this whole thing is unworkable. It was hard enough to get this to work\n> on the combination of Linux, *BSD and Solaris, but I suspect we'll run\n> into increasingly obscure platforms where this is hard or impossible\n> (AIX, HP/UX etc.)\n>\n> The reason we're in this hole is because we use this\n> sha1collisiondetection library to do SHA-1, and the reason we have\n> issues with it specifically (not OpenSSL et al) is because its only\n> method of detecting endianness is at compile time.\nCannot speak for the \"others\", but as I have mentioned before - as AIX \nin only on POWER it is also only Big Endian - so a compiletime #if \ntesting for _AIX will work fine\n> This didn't use to be the case, it was changed in this commit:\n> https://github.com/cr-marcstevens/sha1collisiondetection/commit/d597672\n>\n> Dan Shumow: Since the commit message doesn't say why, can you elaborate\n> a bit on why this was done, i.e. is determining this at runtime harmful\n> for performance? If not, perhaps it would be best to bring this back, at\n> least as an option.\n>\n> And, as an aside, the reason we can't easily make it better ourselves is\n> because the build process for git.git doesn't have a facility to run\n> code to detect this type of stuff (the configure script is always\n> optional). So we can't just run this test ourselves.\nOn AIX - I am required to run configure, and frankly, I am amazed that \nnot everyone is running it. Among other things I an modifying the prefix \n(to /opt) and many of the others to different /var/git/* areas as I do \nnot want to \"polute\" the BOS (base OS) and/or other packages/packagers. \nOfficially, according the Linux FHS-3.0 I sould be using /opt/aixtools \nas prefix.\n\nFYI: my current build process is:\nwget git-{version}.tar.xz\nxz -dc git-{version}.tar.xz\ncd git-{version}.tar.xz\nmv Makefile Makefile.git #my build scripts only run configure when \nMakefile does not exist\n./configure ...\nln Makefile.git Makefile\nmake\n\nI am amazed, as it rarely happens (maybe git is my first encounter) - \nthat configure does not create a Makefile. This also complicates \nbuilding git \"out of tree\".\n>\n> Junio: I've barked up that particular tree before in\n> https://public-inbox.org/git/87a7x3kmh5.fsf@evledraar.gmail.com/ and I\n> won't bore you all by repeating myself, except to say that this is yet\n> another case where I wish we had a hard dependency on some way of doing\n> checks via compiled code in our build system.\nFor AIX: again - the determination is simple. If _AIX is set to 1 then \nuse BigEndian, or, use:\nmichael@x071:[/home/michael]uname\nAIX\n  i.e., something like:\n$(uname) == \"AIX\" && BigEndian=1\n"},{"id":"354020","messageId":"73cd8ad3-f093-b82e-9726-b27f2214629d@felt.demon.nl","threadId":"48973","inReplyTo":"87wotdt649.fsf@evledraar.gmail.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Michael Felt","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-31T12:32:42Z","receivedAt":"2018-07-31T12:33:00Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"\n\nOn 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:\n> The reason we're in this hole is because we use this\n> sha1collisiondetection library to do SHA-1, and the reason we have\n> issues with it specifically (not OpenSSL et al) is because its only\n> method of detecting endianness is at compile time.\nWhen using gcc (no xlc available for Linux on Power)\n\nPOWER6 (Big Endian by definition)\nroot@x068:[/data/httpd/gcc]gcc -dM -E - < /dev/null | grep -i end\n#define __ORDER_LITTLE_ENDIAN__ 1234\n#define __BIG_ENDIAN__ 1\n#define __FLOAT_WORD_ORDER__ __ORDER_BIG_ENDIAN__\n#define __ORDER_PDP_ENDIAN__ 3412\n#define _BIG_ENDIAN 1\n#define __ORDER_BIG_ENDIAN__ 4321\n#define __BYTE_ORDER__ __ORDER_BIG_ENDIAN__\n\nSLES12 on POWER8\nsuse12test:~ # gcc -dM -E - < /dev/null | grep -i end\n#define __ORDER_LITTLE_ENDIAN__ 1234\n#define _LITTLE_ENDIAN 1\n#define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__\n#define __ORDER_PDP_ENDIAN__ 3412\n#define __LITTLE_ENDIAN__ 1\n#define __ORDER_BIG_ENDIAN__ 4321\n#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__\n\n*So, for compile time tests, when gcc is the compiler it seems the\nfollowing defines are available**\n**__BIG_ENDIAN__, _BIG_ENDIAN,  __LITTLE__ENDIAN__, _LITTLE_ENDIAN**\n**or something based on the value of __BYTE_ORDER__*\n\nI'll see if I can find something similar for xlc, but will only be able\nto test xlc on AIX.\n\n>\n> This didn't use to be the case, it was changed in this commit:\n> https://github.com/cr-marcstevens/sha1collisiondetection/commit/d597672\n>\n> Dan Shumow: Since the commit message doesn't say why, can you elaborate\n> a bit on why this was done, i.e. is determining this at runtime harmful\n> for performance? If not, perhaps it would be best to bring this back, at\n> least as an option.\n\n"},{"id":"354034","messageId":"287cdba8-19c1-2fe8-4aff-d0385b38e92c@felt.demon.nl","threadId":"48973","inReplyTo":"87wotdt649.fsf@evledraar.gmail.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Michael Felt","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-31T14:01:03Z","receivedAt":"2018-07-31T14:01:22Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"I hope a I have a \"leap forward\"\n\n\nOn 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:\n> Perhaps it's worth taking a step back here and thinking about whether\n> this whole thing is unworkable. It was hard enough to get this to work\n> on the combination of Linux, *BSD and Solaris, but I suspect we'll run\n> into increasingly obscure platforms where this is hard or impossible\n> (AIX, HP/UX etc.)\nWhile I still cannot say for HP/UX it does seem there is a potential\nsolution based on the status for _LITTLE_ENDIAN and _BIG_ENDIAN. At\nleast, gcc on POWER and xlc on POWER provides one or the other - and my\nhope is that gcc on other platforms also provides them.\n\nFor \"other\" compilers that do not provide them - a modification to\nCFLAGS to define one or the other should make \"make\" work.\n\nDetails (note - I am not a programmer, so by definition at least one of\nmy \"macros\" will be wrong :)\n\nAIX and xlc\nroot@x066:[/]xlc   -qshowmacros -E /dev/null | grep -i endi\n1506-297 (S) Unable to open input file null. A file or directory in the\npath name does not exist..\n#define __HHW_BIG_ENDIAN__ 1\n#define __BIG_ENDIAN__ 1\n#define __THW_BIG_ENDIAN__ 1\n#define _BIG_ENDIAN 1\n\nOn SLES12 (le) and xlc\nsuse12test:~/images/littleEndian/sles # xlc -qshowmacros -dM -E x.c |\ngrep -i endi\n#define _LITTLE_ENDIAN 1\n#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__\n#define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__\n#define __LITTLE_ENDIAN__ 1\n#define __ORDER_BIG_ENDIAN__ 4321\n#define __ORDER_LITTLE_ENDIAN__ 1234\n#define __ORDER_PDP_ENDIAN__ 3412\n#define __VEC_ELEMENT_REG_ORDER__ __ORDER_LITTLE_ENDIAN__\n\n\nBased on what I can see on gcc on POWER and xlc on POWER I think an\napproach (simplified) can be:\n\n#if undefined(_BIG_ENDIAN) && undef(_LITTLE_ENDIAN)\n#error \"one of _BIG_ENDIAN or _LITTLE_ENDIAN must be defined. Try adding\nthe correct value to CFLAGS\"\n#else defined(_BIG_ENDIAN) && defined(_LITTLE_ENDIAN)\n#error \"Only one of _BIG_ENDIAN and _LITTLE_ENDIAN may be defined, not both\"\n#endif\n\nAnd then logic based on the value set.\nThis should also make cross-compile possible by unsetting an incorrect\ndefault and setting the correct value.\n\np.s. Is there a setting I need to set somewhere so I receive a copy of\nthe email sent after it is received by the list. I could send myself a\ncopy, but I much prefer it comes from the maillist - as verification it\nwas received.\n"},{"id":"354036","messageId":"87sh3ztrcb.fsf@evledraar.gmail.com","threadId":"48973","inReplyTo":"287cdba8-19c1-2fe8-4aff-d0385b38e92c@felt.demon.nl","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-07-31T14:25:24Z","receivedAt":"2018-07-31T14:25:35Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 31 2018, Michael Felt wrote:\n\n> I hope a I have a \"leap forward\"\n>\n>\n> On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:\n>> Perhaps it's worth taking a step back here and thinking about whether\n>> this whole thing is unworkable. It was hard enough to get this to work\n>> on the combination of Linux, *BSD and Solaris, but I suspect we'll run\n>> into increasingly obscure platforms where this is hard or impossible\n>> (AIX, HP/UX etc.)\n> While I still cannot say for HP/UX it does seem there is a potential\n> solution based on the status for _LITTLE_ENDIAN and _BIG_ENDIAN. At\n> least, gcc on POWER and xlc on POWER provides one or the other - and my\n> hope is that gcc on other platforms also provides them.\n\nYeah with GCC this is relatively easy, see\nhttps://github.com/cr-marcstevens/sha1collisiondetection/blame/c3e1304/lib/sha1.c#L29-L115\n\n> For \"other\" compilers that do not provide them - a modification to\n> CFLAGS to define one or the other should make \"make\" work.\n>\n> Details (note - I am not a programmer, so by definition at least one of\n> my \"macros\" will be wrong :)\n>\n> AIX and xlc\n> root@x066:[/]xlc -qshowmacros -E /dev/null | grep -i endi\n> 1506-297 (S) Unable to open input file null. A file or directory in the\n> path name does not exist..\n> #define __HHW_BIG_ENDIAN__ 1\n> #define __BIG_ENDIAN__ 1\n> #define __THW_BIG_ENDIAN__ 1\n> #define _BIG_ENDIAN 1\n>\n> On SLES12 (le) and xlc\n> suse12test:~/images/littleEndian/sles # xlc -qshowmacros -dM -E x.c |\n> grep -i endi\n> #define _LITTLE_ENDIAN 1\n> #define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__\n> #define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__\n> #define __LITTLE_ENDIAN__ 1\n> #define __ORDER_BIG_ENDIAN__ 4321\n> #define __ORDER_LITTLE_ENDIAN__ 1234\n> #define __ORDER_PDP_ENDIAN__ 3412\n> #define __VEC_ELEMENT_REG_ORDER__ __ORDER_LITTLE_ENDIAN__\n>\n>\n> Based on what I can see on gcc on POWER and xlc on POWER I think an\n> approach (simplified) can be:\n>\n> #if undefined(_BIG_ENDIAN) && undef(_LITTLE_ENDIAN)\n> #error \"one of _BIG_ENDIAN or _LITTLE_ENDIAN must be defined. Try adding\n> the correct value to CFLAGS\"\n> #else defined(_BIG_ENDIAN) && defined(_LITTLE_ENDIAN)\n> #error \"Only one of _BIG_ENDIAN and _LITTLE_ENDIAN may be defined, not both\"\n> #endif\n>\n> And then logic based on the value set.\n> This should also make cross-compile possible by unsetting an incorrect\n> default and setting the correct value.\n\n...the real trick is using these macros outside of GCC / glibc and on\nolder GCC versions. See the github link above, you basically end up with\na whitelist of how it looks on different systems / compilers. Sometimes\nboth are defined, sometimes only both etc.\n\nIt can be done, but as that code shows it's somewhat complex macro soup\nto get right.\n\n> p.s. Is there a setting I need to set somewhere so I receive a copy of\n> the email sent after it is received by the list. I could send myself a\n> copy, but I much prefer it comes from the maillist - as verification it\n> was received.\n\nYou should get that, but maybe your mailer ignores Message-Ids it\nalready has, but you can go to https://public-inbox.org/git/ and search\nfor the Message-Id or your name to see E-Mails you've sent that made it\nto the list, e.g.:\nhttps://public-inbox.org/git/?q=aixtools%40felt.demon.nl\n"},{"id":"354115","messageId":"325d1a6c-276a-9ffa-f9c5-72cbdecb6175@felt.demon.nl","threadId":"48973","inReplyTo":"87sh3ztrcb.fsf@evledraar.gmail.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Michael","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-07-31T20:06:14Z","receivedAt":"2018-07-31T20:06:29Z","isPatch":false,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"On 31/07/2018 16:25, Ævar Arnfjörð Bjarmason wrote:\n> ...the real trick is using these macros outside of GCC / glibc and on\n> older GCC versions. See the github link above, you basically end up with\n> a whitelist of how it looks on different systems / compilers. Sometimes\n> both are defined, sometimes only both etc.\n>\n> It can be done, but as that code shows it's somewhat complex macro soup\n> to get right.\n\nFYI - the gcc I was using is 4.7.4.\n\nAnd, the reason I suggest the test for both not being defined is so that \n'make' stops and whoever is running make just sets one or the other. Let \nthem 'file a bug' When they come with a compiler that does not work - \nand find out what could be used.\n\nFor example, _AIX is the same as _BIG_ENDIAN. In the meantime, the code \nto test is simple.\n\nEither one of _BIG_ENDIAN or _LITTLE_ENDIAN is provided by the compiler \nor the builder supplies one of the two using CFLAGS. I assume there is \nalso a \"undefine\" flag, maybe -U - so hopefully a -U and a -D \ncombination could be used for cross-compiling.\n\nre: my mailer blocking things - it would only be for this list, as other \nlists come through with no extra work from me. At least I am not aware \nof anything special I could do.\n\n"},{"id":"354146","messageId":"20180801013524.4u5ioghtc2trhsqd@dcvr","threadId":"48973","inReplyTo":"xmqqh8kgfuba.fsf@gitster-ct.c.googlers.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2018-08-01T01:35:24Z","receivedAt":"2018-08-01T01:42:54Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano wrote:\n> Well, having said all that, I do not think I personally mind if\n> ./configure learned to include a \"compile small program and run it\n> to determine byte order on the build machine\" as part of \"we make a\n> reasonable effort\" as long as it cleanly excludes cross building\n> case (and the result is made overridable just in case we misdetect\n> the \"cross-ness\" of the build).\n\nNo need to run the program for cross-compiles, grepping a object\nfile seems to work for autoconf:\n\n\tgit clone https://git.sv.gnu.org/git/autoconf.git\n\t$PAGER autoconf/lib/autoconf/c.m4\n\t# look for \"BIGenDianSyS\"\n"},{"id":"354148","messageId":"87pnz2tv38.fsf@evledraar.gmail.com","threadId":"48973","inReplyTo":"xmqq600wkc4i.fsf@gitster-ct.c.googlers.com","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-08-01T07:16:43Z","receivedAt":"2018-08-01T07:16:48Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 30 2018, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> And, as an aside, the reason we can't easily make it better ourselves is\n>> because the build process for git.git doesn't have a facility to run\n>> code to detect this type of stuff (the configure script is always\n>> optional). So we can't just run this test ourselves.\n>\n> It won't help those who cross-compile anyway.\n\nI was being unclear, what I mean by having \"a hard dependency on some\nway of doing checks via compiled code in our build system\" is that we\nwould do some equivalent of this:\n\n diff --git a/Makefile b/Makefile\n index 08e5c54549..b021b6e1b6 100644\n --- a/Makefile\n +++ b/Makefile\n @@ -1107,7 +1107,7 @@ DC_SHA1_SUBMODULE = auto\n endif\n\n include config.mak.uname\n --include config.mak.autogen\n +include config.mak.autogen\n -include config.mak\n\n ifdef DEVELOPER\n\nAnd document that in order to build git you needed to do './configure &&\nmake' instead of just 'make', and we'd error out by default if\nconfig.mak.autogen wasn't there.\n\nNow obviously that would need some sort of escape hatch. I.e. you could\ninvoke 'make' like this:\n\n make CONFIGURE_MAK_AUTOGEN_FILE=some-file\n\nThat's how you would do cross-compilation, you'd arrange to run\n'./configure' on some system, save the output, and ferry over this\n'some-file' to where you're building git, or you would manually prepare\na file that had all the settings we'd expect to have been set already\nset.\n\nNow, whether we do this with autoconf or not is just an implementation\ndetail. Looking at this some more I think since we already use the\n$(shell) construct we could just have some 'configure-detect' Makefile\ntarget which would compile various test programs, and we'd use their\noutput to set various settings, a sort of home-grown autoconf (because\npeople are bound to have objections to a hard dependency on it...).\n\n> I thought we declared \"we make a reasonable effort to guess the target\n> endianness from the system header by inspecting usual macros, but will\n> not aim to cover every system on the planet---instead there is a knob\n> to tweak it for those on exotic platforms\" last time we discussed\n> this?\n\nYes, but I think it's worth re-visiting that decision, which was made\nwith the constraints that we don't have a build system that can do\nchecks via compiled code, so we need this hack in the first place\ninstead of things Just Working.\n\nAnd as I pointed out in the linked E-Mail this also impacts us in other\nways, and will cause other issues in the future, so it's worth thinking\nabout if this is the right path to take.\n"},{"id":"354149","messageId":"87o9emtuf6.fsf@evledraar.gmail.com","threadId":"48973","inReplyTo":"93056823-2740-d072-1ebd-46b440b33d7e@felt.demon.nl","subject":"Re: Is detecting endianness at compile-time unworkable?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-08-01T07:31:09Z","receivedAt":"2018-08-01T07:31:14Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 31 2018, Michael Felt wrote:\n\n> For AIX: again - the determination is simple. If _AIX is set to 1 then\n> use BigEndian, or, use:\n> michael@x071:[/home/michael]uname\n> AIX\n> i.e., something like:\n> $(uname) == \"AIX\" && BigEndian=1\n\nIn lieu of some \"let's test this with a compile-test\" solution, that\nseems like a viable way forward for now. Can you please test the merge\nrequest I have at\nhttps://github.com/cr-marcstevens/sha1collisiondetection/pull/45 by\nmanually applying that change to the relevant file in sha1dc/ and\nbuilding/testing git on AIX?\n\nIt should work, but as noted in the MR please test it so we can make\nsure, and then (if you have a GitHub account) comment on the MR saying\nit works for you.\n"},{"id":"354322","messageId":"20180802205044.23631-1-avarab@gmail.com","threadId":"48973","inReplyTo":"87o9emtuf6.fsf@evledraar.gmail.com","subject":"[PATCH] sha1dc: update from upstream","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-08-02T20:50:44Z","receivedAt":"2018-08-02T20:51:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Update sha1dc from the latest version by the upstream\nmaintainer[1]. See 2db87328ef (\"Merge branch 'ab/sha1dc'\", 2017-07-10)\nfor the last update.\n\nThis fixes an issue where AIX was wrongly detected as a Little-endian\ninstead of a Big-endian system. See [2][3][4].\n\n1. https://github.com/cr-marcstevens/sha1collisiondetection/commit/232357eb2ea0397388254a4b188333a227bf5b10\n2. https://github.com/cr-marcstevens/sha1collisiondetection/pull/45\n3. https://github.com/cr-marcstevens/sha1collisiondetection/pull/42\n4. https://public-inbox.org/git/20180729200623.GF945730@genre.crustytoothpaste.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nOn Wed, Aug 01 2018, Ævar Arnfjörð Bjarmason wrote:\n> https://github.com/cr-marcstevens/sha1collisiondetection/pull/45\n>[...]\n> It should work, but as noted in the MR please test it so we can make\n> sure, and then (if you have a GitHub account) comment on the MR saying\n> it works for you.\n\nThis got merged upstream, and as noted in that upstream PR I've\npersonally tested this on AIX under both GCC and IBM's xlc on the GCC\ncompile farm, it works.\n\n sha1collisiondetection | 2 +-\n sha1dc/sha1.c | 12 +++++++++++-\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/sha1collisiondetection b/sha1collisiondetection\nindex 19d97bf5af..232357eb2e 160000\n--- a/sha1collisiondetection\n+++ b/sha1collisiondetection\n@@ -1 +1 @@\n-Subproject commit 19d97bf5af05312267c2e874ee6bcf584d9e9681\n+Subproject commit 232357eb2ea0397388254a4b188333a227bf5b10\ndiff --git a/sha1dc/sha1.c b/sha1dc/sha1.c\nindex 25eded1399..df0630bc6d 100644\n--- a/sha1dc/sha1.c\n+++ b/sha1dc/sha1.c\n@@ -93,13 +93,23 @@\n #define SHA1DC_BIGENDIAN\n \n /* Not under GCC-alike or glibc or *BSD or newlib or */\n+#elif (defined(_AIX))\n+\n+/*\n+ * Defines Big Endian on a whitelist of OSs that are known to be Big\n+ * Endian-only. See\n+ * https://public-inbox.org/git/93056823-2740-d072-1ebd-46b440b33d7e@felt.demon.nl/\n+ */\n+#define SHA1DC_BIGENDIAN\n+\n+/* Not under GCC-alike or glibc or *BSD or newlib or or */\n #elif defined(SHA1DC_ON_INTEL_LIKE_PROCESSOR)\n /*\n * As a last resort before we do anything else we're not 100% sure\n * about below, we blacklist specific processors here. We could add\n * more, see e.g. https://wiki.debian.org/ArchitectureSpecificsMemo\n */\n-#else /* Not under GCC-alike or glibc or *BSD or newlib or or */\n+#else /* Not under GCC-alike or glibc or *BSD or newlib or or or */\n \n /* We do nothing more here for now */\n /*#error \"Uncomment this to see if you fall through all the detection\"*/\n-- \n2.18.0.345.g5c9ce644c3\n\n"},{"id":"354332","messageId":"496FCCBF-078F-404C-869C-720D2100CC45@felt.demon.nl","threadId":"48973","inReplyTo":"20180802205044.23631-1-avarab@gmail.com","subject":"Re: [PATCH] sha1dc: update from upstream","fromName":"Michael Felt (aixtools)","fromEmail":"aixtools@felt.demon.nl","sentAt":"2018-08-02T21:29:25Z","receivedAt":"2018-08-02T21:29:43Z","isPatch":true,"sender":{"key":"aixtools@felt.demon.nl","avatar":null},"body":"\n\nSent from my iPhone\n\n> On 2 Aug 2018, at 22:50, Ævar Arnfjörð Bjarmason wrote:\n> \n> Update sha1dc from the latest version by the upstream\n> maintainer[1]. See 2db87328ef (\"Merge branch 'ab/sha1dc'\", 2017-07-10)\n> for the last update.\n> \n> This fixes an issue where AIX was wrongly detected as a Little-endian\n> instead of a Big-endian system. See [2][3][4].\n> \n> 1. https://github.com/cr-marcstevens/sha1collisiondetection/commit/232357eb2ea0397388254a4b188333a227bf5b10\n> 2. https://github.com/cr-marcstevens/sha1collisiondetection/pull/45\n> 3. https://github.com/cr-marcstevens/sha1collisiondetection/pull/42\n> 4. https://public-inbox.org/git/20180729200623.GF945730@genre.crustytoothpaste.net/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> \n>> On Wed, Aug 01 2018, Ævar Arnfjörð Bjarmason wrote:\n>> https://github.com/cr-marcstevens/sha1collisiondetection/pull/45\n>> [...]\n>> It should work, but as noted in the MR please test it so we can make\n>> sure, and then (if you have a GitHub account) comment on the MR saying\n>> it works for you.\n> \n> This got merged upstream, and as noted in that upstream PR I've\n> personally tested this on AIX under both GCC and IBM's xlc on the GCC\n> compile farm, it works.\n> \nThanks. I already have git 2.18 in use with the manual patch. \n> sha1collisiondetection | 2 +-\n> sha1dc/sha1.c | 12 +++++++++++-\n> 2 files changed, 12 insertions(+), 2 deletions(-)\n> \n> diff --git a/sha1collisiondetection b/sha1collisiondetection\n> index 19d97bf5af..232357eb2e 160000\n> --- a/sha1collisiondetection\n> +++ b/sha1collisiondetection\n> @@ -1 +1 @@\n> -Subproject commit 19d97bf5af05312267c2e874ee6bcf584d9e9681\n> +Subproject commit 232357eb2ea0397388254a4b188333a227bf5b10\n> diff --git a/sha1dc/sha1.c b/sha1dc/sha1.c\n> index 25eded1399..df0630bc6d 100644\n> --- a/sha1dc/sha1.c\n> +++ b/sha1dc/sha1.c\n> @@ -93,13 +93,23 @@\n> #define SHA1DC_BIGENDIAN\n> \n> /* Not under GCC-alike or glibc or *BSD or newlib or */\n> +#elif (defined(_AIX))\n> +\n> +/*\n> + * Defines Big Endian on a whitelist of OSs that are known to be Big\n> + * Endian-only. See\n> + * https://public-inbox.org/git/93056823-2740-d072-1ebd-46b440b33d7e@felt.demon.nl/\n> + */\n> +#define SHA1DC_BIGENDIAN\n> +\n> +/* Not under GCC-alike or glibc or *BSD or newlib or or */\n> #elif defined(SHA1DC_ON_INTEL_LIKE_PROCESSOR)\n> /*\n> * As a last resort before we do anything else we're not 100% sure\n> * about below, we blacklist specific processors here. We could add\n> * more, see e.g. https://wiki.debian.org/ArchitectureSpecificsMemo\n> */\n> -#else /* Not under GCC-alike or glibc or *BSD or newlib or or */\n> +#else /* Not under GCC-alike or glibc or *BSD or newlib or or or */\n> \n> /* We do nothing more here for now */\n> /*#error \"Uncomment this to see if you fall through all the detection\"*/\n> -- \n> 2.18.0.345.g5c9ce644c3\n> \n\n"},{"id":"354333","messageId":"CAGZ79kaS_YazchNG+N=gpwnewe+oSdwbdYusFMWpWygbrSqUQQ@mail.gmail.com","threadId":"48973","inReplyTo":"20180802205044.23631-1-avarab@gmail.com","subject":"Re: [PATCH] sha1dc: update from upstream","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-08-02T21:32:08Z","receivedAt":"2018-08-02T21:32:23Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Aug 2, 2018 at 1:51 PM Ævar Arnfjörð Bjarmason wrote:\n\n> diff --git a/sha1collisiondetection b/sha1collisiondetection\n> index 19d97bf5af..232357eb2e 160000\n> --- a/sha1collisiondetection\n> +++ b/sha1collisiondetection\n> @@ -1 +1 @@\n> -Subproject commit 19d97bf5af05312267c2e874ee6bcf584d9e9681\n> +Subproject commit 232357eb2ea0397388254a4b188333a227bf5b10\n\nOfftopic:\nI wonder if we want to extend diffs a little here and similar to a diffstat\nthat is not technically needed, but rather aiding the reviewers eyes.\nSo maybe we could add a concise shortlog, e.g. after this line:\n\n19d97bf5..232357eb (2 dots indicating it is a fast forward instead of 3 dots)\n3 commits from \"Merge pull request #37 from avar/fixup-pull-request-34\"\nto \"Merge pull request #45 from avar/aix-big-endian-detection\"\n"}]}