{"thread":{"id":"45965","subject":"Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default","startedAt":"2017-05-15T12:49:41Z","lastAt":"2017-06-01T14:03:15Z","messageCount":17,"participants":["Ævar Arnfjörð Bjarmason","Marc Stevens","Jeff King","Michael Kebe","Junio C Hamano","Johannes Schindelin","Stefan Beller","Brandon Williams","demerphq"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"319781","messageId":"CACBZZX6nmKK8af0-UpjCKWV4R+hV-uk2xWXVA5U+_UQ3VXU03g@mail.gmail.com","threadId":"45965","inReplyTo":null,"subject":"Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-15T12:49:13Z","receivedAt":"2017-05-15T12:49:41Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since 2.13.0 just running \"git status\" on a newly init'd repo on\nSolaris SPARC[1] segfaults. Michael (CC'd) reported this issue on #git\nand I helped him debug it.\n\nJust compiling with BLK_SHA1=YesPlease solves the issue.\n\nThere are at least two different issues with DC_SHA1 here:\n\n * We don't properly detect that this platform is big endian. The\ncheck at the top of sha1dc/sha1.c needs to test for _BIG_ENDIAN. This\ncomes from sys/isa_defs.h which (I'm told by #solaris) is included on\nSolaris by default, at least by stdio.h.\n\nHacking the endian detection makes t0013-sha1dc.sh pass.\n\n * Even with that & the test passing just a plain \"git init x && cd x\n&& touch A && git add A && git commit\" will segfault.\n\nThis is some bug in the sha1dc code, presumably some big endian issue\nthat's not resolved by the change above. Backtrace for that (censored\nactual author info):\n\n Program received signal SIGSEGV, Segmentation fault.\n [Switching to Thread 1 (LWP 1)]\n 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5,\nW=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291\n 291 SHA1COMPRESS_FULL_ROUND1_STEP_LOAD(a, b, c, d, e, m,\nW, 0, temp);\n (gdb) bt\n #0 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268,\nm=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291\n #1 0x00300b60 in sha1_process (ctx=0xffbf8260, block=0x398ad5) at\nsha1dc/sha1.c:1616\n #2 0x00301188 in SHA1DCUpdate (ctx=0xffbf8260,\n buf=0x398ad5 \"deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\\nauthor Au\nThor 123456789 +0000\\ncommitter Au Thor\n 123456789\n +0000\\n\\nBlah Blah\"..., len=220)\n at sha1dc/sha1.c:1731\n #3 0x0030168c in git_SHA1DCUpdate (ctx=0xffbf8260, vdata=0x398aa0,\nlen=273) at sha1dc/sha1.c:1808\n #4 0x002a6f7c in write_sha1_file_prepare (buf=0x398aa0, len=273,\ntype=0x959c8 \"commit\", sha1=0xffbfd630 \"\",\n hdr=0xffbf8c28 \"commit 273\", hdrlen=0xffbf8c24) at sha1_file.c:3207\n #5 0x002a71ac in hash_sha1_file (buf=0x398aa0, len=273, type=0x959c8\n\"commit\", sha1=0xffbfd630 \"\") at sha1_file.c:3266\n #6 0x002a25f8 in check_sha1_signature (sha1=0xffbfdbb8\n\"\\375\\067\\356\\337\\002\", map=0x398aa0, size=273, type=0x959c8 \"commit\")\n at sha1_file.c:1644\n #7 0x0022816c in parse_object (sha1=0xffbfdbb8\n\"\\375\\067\\356\\337\\002\") at object.c:269\n #8 0x0027c258 in get_reference (revs=0xffbfdc88, name=0xa87f0\n\"HEAD\", sha1=0xffbfdbb8 \"\\375\\067\\356\\337\\002\", flags=0)\n at revision.c:196\n #9 0x00284714 in setup_revisions (argc=0, argv=0x0, revs=0xffbfdc88,\nopt=0xffbfdc74) at revision.c:2295\n #10 0x002ee4d8 in wt_status_collect_changes_index (s=0x348ea8\n) at wt-status.c:585\n #11 0x002eeae8 in wt_status_collect (s=0x348ea8 ) at wt-status.c:701\n #12 0x000db4fc in cmd_status (argc=0, argv=0xffbfe6fc, prefix=0x0) at\nbuiltin/commit.c:1393\n #13 0x000acc24 in run_builtin (p=0x340ccc , argc=1,\nargv=0xffbfe6fc) at git.c:371\n #14 0x000acf98 in handle_builtin (argc=1, argv=0xffbfe6fc) at git.c:572\n #15 0x000ad1a4 in run_argv (argcp=0xffbfe66c, argv=0xffbfe670) at git.c:624\n #16 0x000ad3fc in cmd_main (argc=1, argv=0xffbfe6fc) at git.c:701\n #17 0x001747a0 in main (argc=5, argv=0xffbfe6ec) at common-main.c:43\n\n1. Relevant part of uname -a: \"SunOS 5.11 11.3 sun4v sparc sun4v\".\n"},{"id":"319786","messageId":"006301d2cd83$663b5520$32b1ff60$@marc-stevens.nl","threadId":"45965","inReplyTo":"CACBZZX6nmKK8af0-UpjCKWV4R+hV-uk2xWXVA5U+_UQ3VXU03g@mail.gmail.com","subject":"RE: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default","fromName":"Marc Stevens","fromEmail":"marc@marc-stevens.nl","sentAt":"2017-05-15T13:58:57Z","receivedAt":"2017-05-15T13:59:06Z","isPatch":false,"sender":{"key":"marc@marc-stevens.nl","avatar":null},"body":"Hi Aevar,\n\nThank you for notifying us of this issue.\nBig endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems.\nOur github repo does check correct functioning, including an endianness mistake, with 'make test'.\nBut I guess this is not included for SHA1DC in Git.\n\nAnyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.\n\nI don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue.\nBut I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd. \nCan you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform?\nThis is not problem for x86, but maybe for your platform it is?\nIf it is then we should always copy buffer contents to the sha1context to avoid this issue.\n\nBest regards,\nMarc Stevens\n\n-----Oorspronkelijk bericht-----\nVan: Ævar Arnfjörð Bjarmason [mailto:avarab@gmail.com] \nVerzonden: maandag 15 mei 2017 14:49\nAan: Git Mailing List \nCC: michael.kebe@gmail.com; Jeff King ; Marc Stevens \nOnderwerp: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default\n\nSince 2.13.0 just running \"git status\" on a newly init'd repo on Solaris SPARC[1] segfaults. Michael (CC'd) reported this issue on #git and I helped him debug it.\n\nJust compiling with BLK_SHA1=YesPlease solves the issue.\n\nThere are at least two different issues with DC_SHA1 here:\n\n * We don't properly detect that this platform is big endian. The check at the top of sha1dc/sha1.c needs to test for _BIG_ENDIAN. This comes from sys/isa_defs.h which (I'm told by #solaris) is included on Solaris by default, at least by stdio.h.\n\nHacking the endian detection makes t0013-sha1dc.sh pass.\n\n * Even with that & the test passing just a plain \"git init x && cd x && touch A && git add A && git commit\" will segfault.\n\nThis is some bug in the sha1dc code, presumably some big endian issue that's not resolved by the change above. Backtrace for that (censored actual author info):\n\n Program received signal SIGSEGV, Segmentation fault.\n [Switching to Thread 1 (LWP 1)]\n 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291\n 291 SHA1COMPRESS_FULL_ROUND1_STEP_LOAD(a, b, c, d, e, m,\nW, 0, temp);\n (gdb) bt\n #0 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291\n #1 0x00300b60 in sha1_process (ctx=0xffbf8260, block=0x398ad5) at\nsha1dc/sha1.c:1616\n #2 0x00301188 in SHA1DCUpdate (ctx=0xffbf8260,\n buf=0x398ad5 \"deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\\nauthor Au Thor 123456789 +0000\\ncommitter Au Thor 123456789\n +0000\\n\\nBlah Blah\"..., len=220)\n at sha1dc/sha1.c:1731\n #3 0x0030168c in git_SHA1DCUpdate (ctx=0xffbf8260, vdata=0x398aa0,\nlen=273) at sha1dc/sha1.c:1808\n #4 0x002a6f7c in write_sha1_file_prepare (buf=0x398aa0, len=273,\ntype=0x959c8 \"commit\", sha1=0xffbfd630 \"\",\n hdr=0xffbf8c28 \"commit 273\", hdrlen=0xffbf8c24) at sha1_file.c:3207\n #5 0x002a71ac in hash_sha1_file (buf=0x398aa0, len=273, type=0x959c8 \"commit\", sha1=0xffbfd630 \"\") at sha1_file.c:3266\n #6 0x002a25f8 in check_sha1_signature (sha1=0xffbfdbb8 \"\\375\\067\\356\\337\\002\", map=0x398aa0, size=273, type=0x959c8 \"commit\")\n at sha1_file.c:1644\n #7 0x0022816c in parse_object (sha1=0xffbfdbb8\n\"\\375\\067\\356\\337\\002\") at object.c:269\n #8 0x0027c258 in get_reference (revs=0xffbfdc88, name=0xa87f0 \"HEAD\", sha1=0xffbfdbb8 \"\\375\\067\\356\\337\\002\", flags=0)\n at revision.c:196\n #9 0x00284714 in setup_revisions (argc=0, argv=0x0, revs=0xffbfdc88,\nopt=0xffbfdc74) at revision.c:2295\n #10 0x002ee4d8 in wt_status_collect_changes_index (s=0x348ea8\n) at wt-status.c:585\n #11 0x002eeae8 in wt_status_collect (s=0x348ea8 ) at wt-status.c:701\n #12 0x000db4fc in cmd_status (argc=0, argv=0xffbfe6fc, prefix=0x0) at\nbuiltin/commit.c:1393\n #13 0x000acc24 in run_builtin (p=0x340ccc , argc=1,\nargv=0xffbfe6fc) at git.c:371\n #14 0x000acf98 in handle_builtin (argc=1, argv=0xffbfe6fc) at git.c:572\n #15 0x000ad1a4 in run_argv (argcp=0xffbfe66c, argv=0xffbfe670) at git.c:624\n #16 0x000ad3fc in cmd_main (argc=1, argv=0xffbfe6fc) at git.c:701\n #17 0x001747a0 in main (argc=5, argv=0xffbfe6ec) at common-main.c:43\n\n1. Relevant part of uname -a: \"SunOS 5.11 11.3 sun4v sparc sun4v\".\n\n"},{"id":"319787","messageId":"CACBZZX5Q9paMbYWH47fdK9GuNrE=F=FwR__E1yZ32EOAMw_w6w@mail.gmail.com","threadId":"45965","inReplyTo":"006301d2cd83$663b5520$32b1ff60$@marc-stevens.nl","subject":"Re: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-15T14:13:58Z","receivedAt":"2017-05-15T14:14:25Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Mon, May 15, 2017 at 3:58 PM, Marc Stevens wrote:\n> Hi Aevar,\n>\n> Thank you for notifying us of this issue.\n> Big endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems.\n> Our github repo does check correct functioning, including an endianness mistake, with 'make test'.\n> But I guess this is not included for SHA1DC in Git.\n>\n> Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.\n>\n> I don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue.\n> But I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd.\n> Can you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform?\n> This is not problem for x86, but maybe for your platform it is?\n> If it is then we should always copy buffer contents to the sha1context to avoid this issue.\n\nI don't have access to the box in question, Michael was testing this\ncode for me. But unaligned access is probably the cause, although\naccording to some info I found online that should give a SIGBUS not a\nSIGSEGV, but that may have changed:\n\nhttps://bugs.python.org/issue12181\nhttps://github.com/magnumripper/JohnTheRipper/issues/2187\n\n> Best regards,\n> Marc Stevens\n>\n> -----Oorspronkelijk bericht-----\n> Van: Ævar Arnfjörð Bjarmason [mailto:avarab@gmail.com]\n> Verzonden: maandag 15 mei 2017 14:49\n> Aan: Git Mailing List \n> CC: michael.kebe@gmail.com; Jeff King ; Marc Stevens \n> Onderwerp: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default\n>\n> Since 2.13.0 just running \"git status\" on a newly init'd repo on Solaris SPARC[1] segfaults. Michael (CC'd) reported this issue on #git and I helped him debug it.\n>\n> Just compiling with BLK_SHA1=YesPlease solves the issue.\n>\n> There are at least two different issues with DC_SHA1 here:\n>\n> * We don't properly detect that this platform is big endian. The check at the top of sha1dc/sha1.c needs to test for _BIG_ENDIAN. This comes from sys/isa_defs.h which (I'm told by #solaris) is included on Solaris by default, at least by stdio.h.\n>\n> Hacking the endian detection makes t0013-sha1dc.sh pass.\n>\n> * Even with that & the test passing just a plain \"git init x && cd x && touch A && git add A && git commit\" will segfault.\n>\n> This is some bug in the sha1dc code, presumably some big endian issue that's not resolved by the change above. Backtrace for that (censored actual author info):\n>\n> Program received signal SIGSEGV, Segmentation fault.\n> [Switching to Thread 1 (LWP 1)]\n> 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291\n> 291 SHA1COMPRESS_FULL_ROUND1_STEP_LOAD(a, b, c, d, e, m,\n> W, 0, temp);\n> (gdb) bt\n> #0 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291\n> #1 0x00300b60 in sha1_process (ctx=0xffbf8260, block=0x398ad5) at\n> sha1dc/sha1.c:1616\n> #2 0x00301188 in SHA1DCUpdate (ctx=0xffbf8260,\n> buf=0x398ad5 \"deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\\nauthor Au Thor 123456789 +0000\\ncommitter Au Thor 123456789\n> +0000\\n\\nBlah Blah\"..., len=220)\n> at sha1dc/sha1.c:1731\n> #3 0x0030168c in git_SHA1DCUpdate (ctx=0xffbf8260, vdata=0x398aa0,\n> len=273) at sha1dc/sha1.c:1808\n> #4 0x002a6f7c in write_sha1_file_prepare (buf=0x398aa0, len=273,\n> type=0x959c8 \"commit\", sha1=0xffbfd630 \"\",\n> hdr=0xffbf8c28 \"commit 273\", hdrlen=0xffbf8c24) at sha1_file.c:3207\n> #5 0x002a71ac in hash_sha1_file (buf=0x398aa0, len=273, type=0x959c8 \"commit\", sha1=0xffbfd630 \"\") at sha1_file.c:3266\n> #6 0x002a25f8 in check_sha1_signature (sha1=0xffbfdbb8 \"\\375\\067\\356\\337\\002\", map=0x398aa0, size=273, type=0x959c8 \"commit\")\n> at sha1_file.c:1644\n> #7 0x0022816c in parse_object (sha1=0xffbfdbb8\n> \"\\375\\067\\356\\337\\002\") at object.c:269\n> #8 0x0027c258 in get_reference (revs=0xffbfdc88, name=0xa87f0 \"HEAD\", sha1=0xffbfdbb8 \"\\375\\067\\356\\337\\002\", flags=0)\n> at revision.c:196\n> #9 0x00284714 in setup_revisions (argc=0, argv=0x0, revs=0xffbfdc88,\n> opt=0xffbfdc74) at revision.c:2295\n> #10 0x002ee4d8 in wt_status_collect_changes_index (s=0x348ea8\n> ) at wt-status.c:585\n> #11 0x002eeae8 in wt_status_collect (s=0x348ea8 ) at wt-status.c:701\n> #12 0x000db4fc in cmd_status (argc=0, argv=0xffbfe6fc, prefix=0x0) at\n> builtin/commit.c:1393\n> #13 0x000acc24 in run_builtin (p=0x340ccc , argc=1,\n> argv=0xffbfe6fc) at git.c:371\n> #14 0x000acf98 in handle_builtin (argc=1, argv=0xffbfe6fc) at git.c:572\n> #15 0x000ad1a4 in run_argv (argcp=0xffbfe66c, argv=0xffbfe670) at git.c:624\n> #16 0x000ad3fc in cmd_main (argc=1, argv=0xffbfe6fc) at git.c:701\n> #17 0x001747a0 in main (argc=5, argv=0xffbfe6ec) at common-main.c:43\n>\n> 1. Relevant part of uname -a: \"SunOS 5.11 11.3 sun4v sparc sun4v\".\n>\n"},{"id":"319840","messageId":"20170515220939.vkgofpkdtpz7u26v@sigill.intra.peff.net","threadId":"45965","inReplyTo":"CACBZZX5Q9paMbYWH47fdK9GuNrE=F=FwR__E1yZ32EOAMw_w6w@mail.gmail.com","subject":"Re: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-05-15T22:09:39Z","receivedAt":"2017-05-15T22:09:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 15, 2017 at 04:13:58PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> On Mon, May 15, 2017 at 3:58 PM, Marc Stevens wrote:\n> > Hi Aevar,\n> >\n> > Thank you for notifying us of this issue.\n> > Big endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems.\n> > Our github repo does check correct functioning, including an endianness mistake, with 'make test'.\n> > But I guess this is not included for SHA1DC in Git.\n> >\n> > Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.\n> >\n> > I don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue.\n> > But I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd.\n> > Can you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform?\n> > This is not problem for x86, but maybe for your platform it is?\n> > If it is then we should always copy buffer contents to the sha1context to avoid this issue.\n> \n> I don't have access to the box in question, Michael was testing this\n> code for me. But unaligned access is probably the cause, although\n> according to some info I found online that should give a SIGBUS not a\n> SIGSEGV, but that may have changed:\n\nYeah, I would have expected SIGBUS there. If we have alignment issues,\nthough, I'd expect that ARM systems will experience problems.\n\nBlock-sha1 uses a macro which allows unaligned loads on platforms that\nsupport it, and otherwise does the endian conversion on the fly as we\nload the bytes into a local variable (which presumably happens all\nin-register). That may be faster than doing a mass copy of the buffer.\n\n-Peff\n"},{"id":"319891","messageId":"CAKKM46sS_5bVe5a6wNN7SdVoGvwmVxnLAZTxJ+tSftXfZKeGWg@mail.gmail.com","threadId":"45965","inReplyTo":"007001d2cd88$2b916180$82b42480$@marc-stevens.nl","subject":"Re: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default","fromName":"Michael Kebe","fromEmail":"michael.kebe@gmail.com","sentAt":"2017-05-16T05:43:17Z","receivedAt":"2017-05-16T05:43:44Z","isPatch":false,"sender":{"key":"michael.kebe@gmail.com","avatar":null},"body":"Hi Marc,\n\nworks like a charm!\n\nMichael\n\n\n2017-05-15 16:33 GMT+02:00 Marc Stevens :\n> Hi Michael,\n>\n>\n>\n> See the latest commit to the SHA1DC repo:\n>\n> https://github.com/cr-marcstevens/sha1collisiondetection/commit/33a694a9ee1b79c24be45f9eab5ac0e1aeeaf271\n>\n> Just apply this same patch to SHA1DC in Git.\n>\n>\n>\n> Best regards,\n>\n> Marc Stevens\n>\n>\n>\n> Van: Michael Kebe [mailto:michael.kebe@gmail.com]\n> Verzonden: maandag 15 mei 2017 16:22\n> Aan: Ævar Arnfjörð Bjarmason \n> CC: Jeff King ; Git Mailing List ; Marc\n> Stevens \n> Onderwerp: Re: Git 2.13.0 segfaults on Solaris SPARC due to\n> DC_SHA1=YesPlease being on by default\n>\n>\n>\n> Just give me patch or repo, I will test it tomorrow.\n>\n>\n>\n> Michael\n>\n>\n>\n>\n>\n> Am 15.05.2017 4:14 nachm. schrieb \"Ævar Arnfjörð Bjarmason\"\n> :\n>\n> On Mon, May 15, 2017 at 3:58 PM, Marc Stevens wrote:\n>> Hi Aevar,\n>>\n>> Thank you for notifying us of this issue.\n>> Big endianness is a tricky issue, also since I don't have access or\n>> accurate knowledge about all big endian systems.\n>> Our github repo does check correct functioning, including an endianness\n>> mistake, with 'make test'.\n>> But I guess this is not included for SHA1DC in Git.\n>>\n>> Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and\n>> will do so soon.\n>>\n>> I don't think the segfault is caused by buffer overflow, inproper access,\n>> or the endianness issue.\n>> But I did notice an unexpected issue: the message block pointer m=0x398ad5\n>> is odd.\n>> Can you confirm whether loading an uint32_t from an odd address triggers a\n>> hardware interrupt on your platform?\n>> This is not problem for x86, but maybe for your platform it is?\n>> If it is then we should always copy buffer contents to the sha1context to\n>> avoid this issue.\n>\n> I don't have access to the box in question, Michael was testing this\n> code for me. But unaligned access is probably the cause, although\n> according to some info I found online that should give a SIGBUS not a\n> SIGSEGV, but that may have changed:\n>\n> https://bugs.python.org/issue12181\n> https://github.com/magnumripper/JohnTheRipper/issues/2187\n>\n>> Best regards,\n>> Marc Stevens\n>>\n>> -----Oorspronkelijk bericht-----\n>> Van: Ævar Arnfjörð Bjarmason [mailto:avarab@gmail.com]\n>> Verzonden: maandag 15 mei 2017 14:49\n>> Aan: Git Mailing List \n>> CC: michael.kebe@gmail.com; Jeff King ; Marc Stevens\n>> \n>> Onderwerp: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease\n>> being on by default\n>>\n>> Since 2.13.0 just running \"git status\" on a newly init'd repo on Solaris\n>> SPARC[1] segfaults. Michael (CC'd) reported this issue on #git and I helped\n>> him debug it.\n>>\n>> Just compiling with BLK_SHA1=YesPlease solves the issue.\n>>\n>> There are at least two different issues with DC_SHA1 here:\n>>\n>> * We don't properly detect that this platform is big endian. The check at\n>> the top of sha1dc/sha1.c needs to test for _BIG_ENDIAN. This comes from\n>> sys/isa_defs.h which (I'm told by #solaris) is included on Solaris by\n>> default, at least by stdio.h.\n>>\n>> Hacking the endian detection makes t0013-sha1dc.sh pass.\n>>\n>> * Even with that & the test passing just a plain \"git init x && cd x &&\n>> touch A && git add A && git commit\" will segfault.\n>>\n>> This is some bug in the sha1dc code, presumably some big endian issue\n>> that's not resolved by the change above. Backtrace for that (censored actual\n>> author info):\n>>\n>> Program received signal SIGSEGV, Segmentation fault.\n>> [Switching to Thread 1 (LWP 1)]\n>> 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5,\n>> W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291\n>> 291 SHA1COMPRESS_FULL_ROUND1_STEP_LOAD(a, b, c, d, e, m,\n>> W, 0, temp);\n>> (gdb) bt\n>> #0 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5,\n>> W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291\n>> #1 0x00300b60 in sha1_process (ctx=0xffbf8260, block=0x398ad5) at\n>> sha1dc/sha1.c:1616\n>> #2 0x00301188 in SHA1DCUpdate (ctx=0xffbf8260,\n>> buf=0x398ad5 \"deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\\nauthor Au\n>> Thor 123456789 +0000\\ncommitter Au Thor\n>> 123456789\n>> +0000\\n\\nBlah Blah\"..., len=220)\n>> at sha1dc/sha1.c:1731\n>> #3 0x0030168c in git_SHA1DCUpdate (ctx=0xffbf8260, vdata=0x398aa0,\n>> len=273) at sha1dc/sha1.c:1808\n>> #4 0x002a6f7c in write_sha1_file_prepare (buf=0x398aa0, len=273,\n>> type=0x959c8 \"commit\", sha1=0xffbfd630 \"\",\n>> hdr=0xffbf8c28 \"commit 273\", hdrlen=0xffbf8c24) at sha1_file.c:3207\n>> #5 0x002a71ac in hash_sha1_file (buf=0x398aa0, len=273, type=0x959c8\n>> \"commit\", sha1=0xffbfd630 \"\") at sha1_file.c:3266\n>> #6 0x002a25f8 in check_sha1_signature (sha1=0xffbfdbb8\n>> \"\\375\\067\\356\\337\\002\", map=0x398aa0, size=273, type=0x959c8 \"commit\")\n>> at sha1_file.c:1644\n>> #7 0x0022816c in parse_object (sha1=0xffbfdbb8\n>> \"\\375\\067\\356\\337\\002\") at object.c:269\n>> #8 0x0027c258 in get_reference (revs=0xffbfdc88, name=0xa87f0 \"HEAD\",\n>> sha1=0xffbfdbb8 \"\\375\\067\\356\\337\\002\", flags=0)\n>> at revision.c:196\n>> #9 0x00284714 in setup_revisions (argc=0, argv=0x0, revs=0xffbfdc88,\n>> opt=0xffbfdc74) at revision.c:2295\n>> #10 0x002ee4d8 in wt_status_collect_changes_index (s=0x348ea8\n>> ) at wt-status.c:585\n>> #11 0x002eeae8 in wt_status_collect (s=0x348ea8 ) at\n>> wt-status.c:701\n>> #12 0x000db4fc in cmd_status (argc=0, argv=0xffbfe6fc, prefix=0x0) at\n>> builtin/commit.c:1393\n>> #13 0x000acc24 in run_builtin (p=0x340ccc , argc=1,\n>> argv=0xffbfe6fc) at git.c:371\n>> #14 0x000acf98 in handle_builtin (argc=1, argv=0xffbfe6fc) at git.c:572\n>> #15 0x000ad1a4 in run_argv (argcp=0xffbfe66c, argv=0xffbfe670) at\n>> git.c:624\n>> #16 0x000ad3fc in cmd_main (argc=1, argv=0xffbfe6fc) at git.c:701\n>> #17 0x001747a0 in main (argc=5, argv=0xffbfe6ec) at common-main.c:43\n>>\n>> 1. Relevant part of uname -a: \"SunOS 5.11 11.3 sun4v sparc\n>> sun4v\".\n>>\n"},{"id":"320022","messageId":"xmqq37c4xcr6.fsf_-_@gitster.mtv.corp.google.com","threadId":"45965","inReplyTo":"CAKKM46sS_5bVe5a6wNN7SdVoGvwmVxnLAZTxJ+tSftXfZKeGWg@mail.gmail.com","subject":"[PATCH] sha1dc: fix issues with a big endian platform","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-17T05:39:57Z","receivedAt":"2017-05-17T05:40:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"From: Marc Stevens \n\nSome big-endian platforms define _BIG_ENDIAN, which the test at the\nbeginning of file has missed. Also, when the input is not aligned,\nsome platforms trigger SIGBUS.\n\nThis change corresponds to 33a694a9 (\"Fix issues with a big endian\nplatform\", 2017-05-15) in the history of the upstream repository\nhttps://github.com/cr-marcstevens/sha1collisiondetection\n\n---\n\n * So here is my attempt to clarify the log message (I left the\n title as-is, but this change deals both with endianness and\n alignment requirement).\n\n Please look it over, and then sign-off your patch ;-)\n\n Thanks.\n\n P.S. I wonder how often \"buf\" is not aligned---could we somehow\n optimize out memcpy when it is not necessary, or is it not worth\n it?\n\n sha1dc/sha1.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/sha1dc/sha1.c b/sha1dc/sha1.c\nindex 35e9dd5bf4..ae25318c47 100644\n--- a/sha1dc/sha1.c\n+++ b/sha1dc/sha1.c\n@@ -20,7 +20,7 @@\n */\n #if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \\\n (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) || \\\n- defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \\\n+ defined(_BIG_ENDIAN) || defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \\\n defined(_MIPSEB) || defined(__MIPSEB) || defined(__MIPSEB__)\n \n #define SHA1DC_BIGENDIAN\t1\n@@ -1728,7 +1728,8 @@ void SHA1DCUpdate(SHA1_CTX* ctx, const char* buf, size_t len)\n \twhile (len >= 64)\n \t{\n \t\tctx->total += 64;\n-\t\tsha1_process(ctx, (uint32_t*)(buf));\n+\t\tmemcpy(ctx->buffer, buf, 64);\n+\t\tsha1_process(ctx, (uint32_t*)(ctx->buffer));\n \t\tbuf += 64;\n \t\tlen -= 64;\n \t}\n-- \n2.13.0-416-g4c6b804423\n\n"},{"id":"320025","messageId":"CACBZZX6cCjorisOn9qyrDH9L_MtKAsLLW=q1_htuaJ+E0QK9rg@mail.gmail.com","threadId":"45965","inReplyTo":"xmqq37c4xcr6.fsf_-_@gitster.mtv.corp.google.com","subject":"Re: [PATCH] sha1dc: fix issues with a big endian platform","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-17T06:30:12Z","receivedAt":"2017-05-17T06:30:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Wed, May 17, 2017 at 7:39 AM, Junio C Hamano wrote:\n> From: Marc Stevens \n>\n> Some big-endian platforms define _BIG_ENDIAN, which the test at the\n> beginning of file has missed. Also, when the input is not aligned,\n> some platforms trigger SIGBUS.\n>\n> This change corresponds to 33a694a9 (\"Fix issues with a big endian\n> platform\", 2017-05-15) in the history of the upstream repository\n> https://github.com/cr-marcstevens/sha1collisiondetection\n\nThen why not just update sha1dc from upstream instead of\ncherry-picking one patch from them? My update-sha1dc change does this:\nhttps://github.com/git/git/compare/master...avar:update-sha1dc\n\nThis follows the instructions Jeff detailed in 28dc98e343. I solved a\nfew merge conflicts, I think I got them right, but review of that\nespecially welcome.\n\nMaybe you just meant this patch for maint, but if that's the case it's\nnot clear from your commit message or E-Mail.\n\n> ---\n>\n> * So here is my attempt to clarify the log message (I left the\n> title as-is, but this change deals both with endianness and\n> alignment requirement).\n>\n> Please look it over, and then sign-off your patch ;-)\n>\n> Thanks.\n>\n> P.S. I wonder how often \"buf\" is not aligned---could we somehow\n> optimize out memcpy when it is not necessary, or is it not worth\n> it?\n>\n> sha1dc/sha1.c | 5 +++--\n> 1 file changed, 3 insertions(+), 2 deletions(-)\n>\n> diff --git a/sha1dc/sha1.c b/sha1dc/sha1.c\n> index 35e9dd5bf4..ae25318c47 100644\n> --- a/sha1dc/sha1.c\n> +++ b/sha1dc/sha1.c\n> @@ -20,7 +20,7 @@\n> */\n> #if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \\\n> (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) || \\\n> - defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \\\n> + defined(_BIG_ENDIAN) || defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \\\n> defined(_MIPSEB) || defined(__MIPSEB) || defined(__MIPSEB__)\n>\n> #define SHA1DC_BIGENDIAN 1\n> @@ -1728,7 +1728,8 @@ void SHA1DCUpdate(SHA1_CTX* ctx, const char* buf, size_t len)\n> while (len >= 64)\n> {\n> ctx->total += 64;\n> - sha1_process(ctx, (uint32_t*)(buf));\n> + memcpy(ctx->buffer, buf, 64);\n> + sha1_process(ctx, (uint32_t*)(ctx->buffer));\n> buf += 64;\n> len -= 64;\n> }\n> --\n> 2.13.0-416-g4c6b804423\n>\n"},{"id":"320029","messageId":"xmqqk25gvu12.fsf@gitster.mtv.corp.google.com","threadId":"45965","inReplyTo":"CACBZZX6cCjorisOn9qyrDH9L_MtKAsLLW=q1_htuaJ+E0QK9rg@mail.gmail.com","subject":"Re: [PATCH] sha1dc: fix issues with a big endian platform","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-17T07:09:45Z","receivedAt":"2017-05-17T07:09:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Wed, May 17, 2017 at 7:39 AM, Junio C Hamano wrote:\n>> From: Marc Stevens \n>>\n>> Some big-endian platforms define _BIG_ENDIAN, which the test at the\n>> beginning of file has missed. Also, when the input is not aligned,\n>> some platforms trigger SIGBUS.\n>>\n>> This change corresponds to 33a694a9 (\"Fix issues with a big endian\n>> platform\", 2017-05-15) in the history of the upstream repository\n>> https://github.com/cr-marcstevens/sha1collisiondetection\n>\n> Then why not just update sha1dc from upstream instead of\n> cherry-picking one patch from them?\n\nSee the original message upthread. I did the cherry-pick simply\nbecause that was what Marc instructed the patch recipient to do ;-).\n\n"},{"id":"320034","messageId":"20170517113824.31700-1-avarab@gmail.com","threadId":"45965","inReplyTo":"xmqqk25gvu12.fsf@gitster.mtv.corp.google.com","subject":"[PATCH/RFC 0/3] Use sha1collisiondetection as a submodule","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-17T11:38:21Z","receivedAt":"2017-05-17T11:38:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Wed, May 17, 2017 at 9:09 AM, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> On Wed, May 17, 2017 at 7:39 AM, Junio C Hamano wrote:\n>>> From: Marc Stevens \n>>>\n>>> Some big-endian platforms define _BIG_ENDIAN, which the test at the\n>>> beginning of file has missed. Also, when the input is not aligned,\n>>> some platforms trigger SIGBUS.\n>>>\n>>> This change corresponds to 33a694a9 (\"Fix issues with a big endian\n>>> platform\", 2017-05-15) in the history of the upstream repository\n>>> https://github.com/cr-marcstevens/sha1collisiondetection\n>>\n>> Then why not just update sha1dc from upstream instead of\n>> cherry-picking one patch from them?\n>\n> See the original message upthread. I did the cherry-pick simply\n> because that was what Marc instructed the patch recipient to do ;-).\n\nSince that patch is now in Marc's upstream code we can just update to\nthat.\n\nWhile we're at it let's see if Marc will take a patch so that we can\nuse his code as-is with some Makefile trickery of our own, instead of\nhaving to solve merge conflicts each time we update from him. I'll\nsubmit a pull request for that shortly.\n\nAnd since if and when that pull request gets accepted we're at the\npoint of being able to use the upstream code as-is & don't need to\nlocally modify it, we can just use a submodule to track it.\n\nÆvar Arnfjörð Bjarmason (3):\n sha1dc: update from my fork of upstream\n sha1dc: use sha1collisiondetection as a submodule\n sha1dc: remove our old copy of the sha1dc code\n\n .gitmodules | 4 +\n Makefile | 13 +-\n hash.h | 2 +-\n sha1collisiondetection | 1 +\n sha1dc/LICENSE.txt | 30 -\n sha1dc/sha1.c | 1809 ------------------------------------------------\n sha1dc/sha1.h | 122 ----\n sha1dc/ubc_check.c | 363 ----------\n sha1dc/ubc_check.h | 44 --\n sha1dc_git.c | 19 +\n sha1dc_git.h | 14 +\n 11 files changed, 49 insertions(+), 2372 deletions(-)\n create mode 100644 .gitmodules\n create mode 160000 sha1collisiondetection\n delete mode 100644 sha1dc/LICENSE.txt\n delete mode 100644 sha1dc/sha1.c\n delete mode 100644 sha1dc/sha1.h\n delete mode 100644 sha1dc/ubc_check.c\n delete mode 100644 sha1dc/ubc_check.h\n create mode 100644 sha1dc_git.c\n create mode 100644 sha1dc_git.h\n\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320035","messageId":"20170517113824.31700-2-avarab@gmail.com","threadId":"45965","inReplyTo":"20170517113824.31700-1-avarab@gmail.com","subject":"[PATCH/RFC 1/3] sha1dc: update from my fork of upstream","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-17T11:38:22Z","receivedAt":"2017-05-17T11:38:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Upgrade the sha1dc code from my fork of the project, which has one\npatch I'm hoping will be integrated upstream after this series\ndemonstrates its usefulness:\n\n https://github.com/cr-marcstevens/sha1collisiondetection/compare/master...avar:easier-inclusion-in-other-programs\n\nThat minor change allows us to in the future just use the upstream\ncode as-is, without any local modifications.\n\nNow all the local modifications we've done in this directory (see the\ngit history of sha1dc/) are done either with defines, or via the newly\nadded sha1dc_git.[ch] files.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 9 +++++-\n sha1dc/sha1.c | 79 ++++++++++++++++++++++++++---------------------\n sha1dc/sha1.h | 90 +++++++++++++++++++++++-------------------------------\n sha1dc/ubc_check.c | 9 ++++--\n sha1dc/ubc_check.h | 10 ++++--\n sha1dc_git.c | 19 ++++++++++++\n sha1dc_git.h | 14 +++++++++\n 7 files changed, 139 insertions(+), 91 deletions(-)\n create mode 100644 sha1dc_git.c\n create mode 100644 sha1dc_git.h\n\ndiff --git a/Makefile b/Makefile\nindex e35542e631..342466d83a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1414,7 +1414,14 @@ else\n \tDC_SHA1 := YesPlease\n \tLIB_OBJS += sha1dc/sha1.o\n \tLIB_OBJS += sha1dc/ubc_check.o\n-\tBASIC_CFLAGS += -DSHA1_DC\n+\tBASIC_CFLAGS += \\\n+\t\t-DSHA1_DC \\\n+\t\t-DSHA1DC_NO_STANDARD_INCLUDES \\\n+\t\t-DSHA1DC_INIT_SAFE_HASH_DEFAULT=0 \\\n+\t\t-DSHA1DC_CUSTOM_INCLUDE_SHA1_C=\"\\\"cache.h\\\"\" \\\n+\t\t-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_C=\"\\\"../sha1dc_git.c\\\"\" \\\n+\t\t-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_H=\"\\\"../sha1dc_git.h\\\"\" \\\n+\t\t-DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C=\"\\\"git-compat-util.h\\\"\"\n endif\n endif\n endif\ndiff --git a/sha1dc/sha1.c b/sha1dc/sha1.c\nindex 35e9dd5bf4..c53f68b844 100644\n--- a/sha1dc/sha1.c\n+++ b/sha1dc/sha1.c\n@@ -5,12 +5,26 @@\n * https://opensource.org/licenses/MIT\n ***/\n \n-#include \"cache.h\"\n-#include \"sha1dc/sha1.h\"\n-#include \"sha1dc/ubc_check.h\"\n+#ifndef SHA1DC_NO_STANDARD_INCLUDES\n+#include \n+#include \n+#include \n+#include \n+#endif\n+\n+#ifdef SHA1DC_CUSTOM_INCLUDE_SHA1_C\n+#include SHA1DC_CUSTOM_INCLUDE_SHA1_C\n+#endif\n \n+#ifndef SHA1DC_INIT_SAFE_HASH_DEFAULT\n+#define SHA1DC_INIT_SAFE_HASH_DEFAULT 1\n+#endif\n \n-/*\n+#include \"sha1.h\"\n+#include \"ubc_check.h\"\n+\n+\n+/* \n Because Little-Endian architectures are most common,\n we only set SHA1DC_BIGENDIAN if one of these conditions is met.\n Note that all MSFT platforms are little endian,\n@@ -18,14 +32,17 @@\n If you are compiling on a big endian platform and your compiler does not define one of these,\n you will have to add whatever macros your tool chain defines to indicate Big-Endianness.\n */\n-#if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \\\n+#ifdef SHA1DC_BIGENDIAN\n+#undef SHA1DC_BIGENDIAN\n+#endif\n+#if (!defined SHA1DC_FORCE_LITTLEENDIAN) && \\\n+ ((defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \\\n (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) || \\\n- defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \\\n- defined(_MIPSEB) || defined(__MIPSEB) || defined(__MIPSEB__)\n+ defined(_BIG_ENDIAN) || defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \\\n+ defined(_MIPSEB) || defined(__MIPSEB) || defined(__MIPSEB__) || defined(SHA1DC_FORCE_BIGENDIAN))\n+\n+#define SHA1DC_BIGENDIAN\n \n-#define SHA1DC_BIGENDIAN\t1\n-#else\n-#undef SHA1DC_BIGENDIAN\n #endif /*ENDIANNESS SELECTION*/\n \n #define rotate_right(x,n) (((x)>>(n))|((x)<<(32-(n))))\n@@ -36,11 +53,11 @@\n \n #define sha1_mix(W, t) (rotate_left(W[t - 3] ^ W[t - 8] ^ W[t - 14] ^ W[t - 16], 1))\n \n-#if defined(SHA1DC_BIGENDIAN)\n+#ifdef SHA1DC_BIGENDIAN\n \t#define sha1_load(m, t, temp) { temp = m[t]; }\n #else\n \t#define sha1_load(m, t, temp) { temp = m[t]; sha1_bswap32(temp); }\n-#endif /* !defined(SHA1DC_BIGENDIAN) */\n+#endif\n \n #define sha1_store(W, t, x)\t*(volatile uint32_t *)&W[t] = x\n \n@@ -869,6 +886,11 @@ static void sha1recompress_fast_ ## t (uint32_t ihvin[5], uint32_t ihvout[5], co\n \tihvout[0] = ihvin[0] + a; ihvout[1] = ihvin[1] + b; ihvout[2] = ihvin[2] + c; ihvout[3] = ihvin[3] + d; ihvout[4] = ihvin[4] + e; \\\n }\n \n+#ifdef _MSC_VER\n+#pragma warning(push)\n+#pragma warning(disable: 4127) /* Complier complains about the checks in the above macro being constant. */\n+#endif\n+\n #ifdef DOSTORESTATE0\n SHA1_RECOMPRESS(0)\n #endif\n@@ -1189,6 +1211,10 @@ SHA1_RECOMPRESS(78)\n SHA1_RECOMPRESS(79)\n #endif\n \n+#ifdef _MSC_VER\n+#pragma warning(pop)\n+#endif\n+\n static void sha1_recompression_step(uint32_t step, uint32_t ihvin[5], uint32_t ihvout[5], const uint32_t me2[80], const uint32_t state[5])\n {\n \tswitch (step)\n@@ -1606,7 +1632,7 @@ static void sha1_process(SHA1_CTX* ctx, const uint32_t block[16])\n \tunsigned i, j;\n \tuint32_t ubc_dv_mask[DVMASKSIZE] = { 0xFFFFFFFF };\n \tuint32_t ihvtmp[5];\n-\n+\t\n \tctx->ihv1[0] = ctx->ihv[0];\n \tctx->ihv1[1] = ctx->ihv[1];\n \tctx->ihv1[2] = ctx->ihv[2];\n@@ -1662,7 +1688,7 @@ void SHA1DCInit(SHA1_CTX* ctx)\n \tctx->ihv[3] = 0x10325476;\n \tctx->ihv[4] = 0xC3D2E1F0;\n \tctx->found_collision = 0;\n-\tctx->safe_hash = 0;\n+\tctx->safe_hash = SHA1DC_INIT_SAFE_HASH_DEFAULT;\n \tctx->ubc_check = 1;\n \tctx->detect_coll = 1;\n \tctx->reduced_round_coll = 0;\n@@ -1728,7 +1754,8 @@ void SHA1DCUpdate(SHA1_CTX* ctx, const char* buf, size_t len)\n \twhile (len >= 64)\n \t{\n \t\tctx->total += 64;\n-\t\tsha1_process(ctx, (uint32_t*)(buf));\n+\t\tmemcpy(ctx->buffer, buf, 64);\n+\t\tsha1_process(ctx, (uint32_t*)(ctx->buffer));\n \t\tbuf += 64;\n \t\tlen -= 64;\n \t}\n@@ -1788,22 +1815,6 @@ int SHA1DCFinal(unsigned char output[20], SHA1_CTX *ctx)\n \treturn ctx->found_collision;\n }\n \n-void git_SHA1DCFinal(unsigned char hash[20], SHA1_CTX *ctx)\n-{\n-\tif (!SHA1DCFinal(hash, ctx))\n-\t\treturn;\n-\tdie(\"SHA-1 appears to be part of a collision attack: %s\",\n-\t sha1_to_hex(hash));\n-}\n-\n-void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *vdata, unsigned long len)\n-{\n-\tconst char *data = vdata;\n-\t/* We expect an unsigned long, but sha1dc only takes an int */\n-\twhile (len > INT_MAX) {\n-\t\tSHA1DCUpdate(ctx, data, INT_MAX);\n-\t\tdata += INT_MAX;\n-\t\tlen -= INT_MAX;\n-\t}\n-\tSHA1DCUpdate(ctx, data, len);\n-}\n+#ifdef SHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_C\n+#include SHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_C\n+#endif\ndiff --git a/sha1dc/sha1.h b/sha1dc/sha1.h\nindex bd8bd928fb..285161ee35 100644\n--- a/sha1dc/sha1.h\n+++ b/sha1dc/sha1.h\n@@ -4,6 +4,7 @@\n * See accompanying file LICENSE.txt or copy at\n * https://opensource.org/licenses/MIT\n ***/\n+\n #ifndef SHA1DC_SHA1_H\n #define SHA1DC_SHA1_H\n \n@@ -11,36 +12,30 @@\n extern \"C\" {\n #endif\n \n-/* uses SHA-1 message expansion to expand the first 16 words of W[] to 80 words */\n-/* void sha1_message_expansion(uint32_t W[80]); */\n-\n-/* sha-1 compression function; first version takes a message block pre-parsed as 16 32-bit integers, second version takes an already expanded message) */\n-/* void sha1_compression(uint32_t ihv[5], const uint32_t m[16]);\n-void sha1_compression_W(uint32_t ihv[5], const uint32_t W[80]); */\n+#ifndef SHA1DC_NO_STANDARD_INCLUDES\n+#include \n+#endif\n \n-/* same as sha1_compression_W, but additionally store intermediate states */\n+/* sha-1 compression function that takes an already expanded message, and additionally store intermediate states */\n /* only stores states ii (the state between step ii-1 and step ii) when DOSTORESTATEii is defined in ubc_check.h */\n void sha1_compression_states(uint32_t[5], const uint32_t[16], uint32_t[80], uint32_t[80][5]);\n \n /*\n-// function type for sha1_recompression_step_T (uint32_t ihvin[5], uint32_t ihvout[5], const uint32_t me2[80], const uint32_t state[5])\n-// where 0 <= T < 80\n-// me2 is an expanded message (the expansion of an original message block XOR'ed with a disturbance vector's message block difference)\n-// state is the internal state (a,b,c,d,e) before step T of the SHA-1 compression function while processing the original message block\n-// the function will return:\n-// ihvin: the reconstructed input chaining value\n-// ihvout: the reconstructed output chaining value\n+// Function type for sha1_recompression_step_T (uint32_t ihvin[5], uint32_t ihvout[5], const uint32_t me2[80], const uint32_t state[5]).\n+// Where 0 <= T < 80\n+// me2 is an expanded message (the expansion of an original message block XOR'ed with a disturbance vector's message block difference.)\n+// state is the internal state (a,b,c,d,e) before step T of the SHA-1 compression function while processing the original message block.\n+// The function will return:\n+// ihvin: The reconstructed input chaining value.\n+// ihvout: The reconstructed output chaining value.\n */\n typedef void(*sha1_recompression_type)(uint32_t*, uint32_t*, const uint32_t*, const uint32_t*);\n \n-/* table of sha1_recompression_step_0, ... , sha1_recompression_step_79 */\n-/* extern sha1_recompression_type sha1_recompression_step[80];*/\n-\n-/* a callback function type that can be set to be called when a collision block has been found: */\n+/* A callback function type that can be set to be called when a collision block has been found: */\n /* void collision_block_callback(uint64_t byteoffset, const uint32_t ihvin1[5], const uint32_t ihvin2[5], const uint32_t m1[80], const uint32_t m2[80]) */\n typedef void(*collision_block_callback)(uint64_t, const uint32_t*, const uint32_t*, const uint32_t*, const uint32_t*);\n \n-/* the SHA-1 context */\n+/* The SHA-1 context. */\n typedef struct {\n \tuint64_t total;\n \tuint32_t ihv[5];\n@@ -59,30 +54,34 @@ typedef struct {\n \tuint32_t states[80][5];\n } SHA1_CTX;\n \n-/* initialize SHA-1 context */\n+/* Initialize SHA-1 context. */\n void SHA1DCInit(SHA1_CTX*);\n \n /*\n-// function to enable safe SHA-1 hashing:\n-// collision attacks are thwarted by hashing a detected near-collision block 3 times\n-// think of it as extending SHA-1 from 80-steps to 240-steps for such blocks:\n-// the best collision attacks against SHA-1 have complexity about 2^60,\n-// thus for 240-steps an immediate lower-bound for the best cryptanalytic attacks would 2^180\n-// an attacker would be better off using a generic birthday search of complexity 2^80\n-//\n-// enabling safe SHA-1 hashing will result in the correct SHA-1 hash for messages where no collision attack was detected\n-// but it will result in a different SHA-1 hash for messages where a collision attack was detected\n-// this will automatically invalidate SHA-1 based digital signature forgeries\n-// enabled by default\n+ Function to enable safe SHA-1 hashing:\n+ Collision attacks are thwarted by hashing a detected near-collision block 3 times.\n+ Think of it as extending SHA-1 from 80-steps to 240-steps for such blocks:\n+ The best collision attacks against SHA-1 have complexity about 2^60,\n+ thus for 240-steps an immediate lower-bound for the best cryptanalytic attacks would be 2^180.\n+ An attacker would be better off using a generic birthday search of complexity 2^80.\n+ \n+ Enabling safe SHA-1 hashing will result in the correct SHA-1 hash for messages where no collision attack was detected,\n+ but it will result in a different SHA-1 hash for messages where a collision attack was detected.\n+ This will automatically invalidate SHA-1 based digital signature forgeries.\n+ Enabled by default.\n */\n void SHA1DCSetSafeHash(SHA1_CTX*, int);\n \n-/* function to disable or enable the use of Unavoidable Bitconditions (provides a significant speed up) */\n-/* enabled by default */\n+/*\n+ Function to disable or enable the use of Unavoidable Bitconditions (provides a significant speed up).\n+ Enabled by default\n+ */\n void SHA1DCSetUseUBC(SHA1_CTX*, int);\n \n-/* function to disable or enable the use of Collision Detection */\n-/* enabled by default */\n+/*\n+ Function to disable or enable the use of Collision Detection.\n+ Enabled by default.\n+ */\n void SHA1DCSetUseDetectColl(SHA1_CTX*, int);\n \n /* function to disable or enable the detection of reduced-round SHA-1 collisions */\n@@ -98,25 +97,14 @@ void SHA1DCUpdate(SHA1_CTX*, const char*, size_t);\n \n /* obtain SHA-1 hash from SHA-1 context */\n /* returns: 0 = no collision detected, otherwise = collision found => warn user for active attack */\n-int SHA1DCFinal(unsigned char[20], SHA1_CTX*);\n+int SHA1DCFinal(unsigned char[20], SHA1_CTX*); \n \n-/*\n- * Same as SHA1DCFinal, but convert collision attack case into a verbose die().\n- */\n-void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);\n-\n-/*\n- * Same as SHA1DCUpdate, but adjust types to match git's usual interface.\n- */\n-void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len);\n-\n-#define platform_SHA_CTX SHA1_CTX\n-#define platform_SHA1_Init SHA1DCInit\n-#define platform_SHA1_Update git_SHA1DCUpdate\n-#define platform_SHA1_Final git_SHA1DCFinal\n+#ifdef SHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_H\n+#include SHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_H\n+#endif\n \n #if defined(__cplusplus)\n }\n #endif\n \n-#endif /* SHA1DC_SHA1_H */\n+#endif\ndiff --git a/sha1dc/ubc_check.c b/sha1dc/ubc_check.c\nindex 089dd4743d..0614926648 100644\n--- a/sha1dc/ubc_check.c\n+++ b/sha1dc/ubc_check.c\n@@ -24,8 +24,13 @@\n // ubc_check has been verified against ubc_check_verify using the 'ubc_check_test' program in the tools section\n */\n \n-#include \"git-compat-util.h\"\n-#include \"sha1dc/ubc_check.h\"\n+#ifndef SHA1DC_NO_STANDARD_INCLUDES\n+#include \n+#endif\n+#ifdef SHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C\n+#include SHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C\n+#endif\n+#include \"ubc_check.h\"\n \n static const uint32_t DV_I_43_0_bit \t= (uint32_t)(1) << 0;\n static const uint32_t DV_I_44_0_bit \t= (uint32_t)(1) << 1;\ndiff --git a/sha1dc/ubc_check.h b/sha1dc/ubc_check.h\nindex b64c306d77..f6bb62547a 100644\n--- a/sha1dc/ubc_check.h\n+++ b/sha1dc/ubc_check.h\n@@ -20,13 +20,17 @@\n // thus one needs to do the recompression check for each DV that has its bit set\n */\n \n-#ifndef UBC_CHECK_H\n-#define UBC_CHECK_H\n+#ifndef SHA1DC_UBC_CHECK_H\n+#define SHA1DC_UBC_CHECK_H\n \n #if defined(__cplusplus)\n extern \"C\" {\n #endif\n \n+#ifndef SHA1DC_NO_STANDARD_INCLUDES\n+#include \n+#endif\n+\n #define DVMASKSIZE 1\n typedef struct { int dvType; int dvK; int dvB; int testt; int maski; int maskb; uint32_t dm[80]; } dv_info_t;\n extern dv_info_t sha1_dvs[];\n@@ -41,4 +45,4 @@ void ubc_check(const uint32_t W[80], uint32_t dvmask[DVMASKSIZE]);\n }\n #endif\n \n-#endif /* UBC_CHECK_H */\n+#endif\ndiff --git a/sha1dc_git.c b/sha1dc_git.c\nnew file mode 100644\nindex 0000000000..e59c1d8103\n--- /dev/null\n+++ b/sha1dc_git.c\n@@ -0,0 +1,19 @@\n+void git_SHA1DCFinal(unsigned char hash[20], SHA1_CTX *ctx)\n+{\n+\tif (!SHA1DCFinal(hash, ctx))\n+\t\treturn;\n+\tdie(\"SHA-1 appears to be part of a collision attack: %s\",\n+\t sha1_to_hex(hash));\n+}\n+\n+void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *vdata, unsigned long len)\n+{\n+\tconst char *data = vdata;\n+\t/* We expect an unsigned long, but sha1dc only takes an int */\n+\twhile (len > INT_MAX) {\n+\t\tSHA1DCUpdate(ctx, data, INT_MAX);\n+\t\tdata += INT_MAX;\n+\t\tlen -= INT_MAX;\n+\t}\n+\tSHA1DCUpdate(ctx, data, len);\n+}\ndiff --git a/sha1dc_git.h b/sha1dc_git.h\nnew file mode 100644\nindex 0000000000..801b5a1fcb\n--- /dev/null\n+++ b/sha1dc_git.h\n@@ -0,0 +1,14 @@\n+/*\n+ * Same as SHA1DCFinal, but convert collision attack case into a verbose die().\n+ */\n+void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);\n+\n+/*\n+ * Same as SHA1DCUpdate, but adjust types to match git's usual interface.\n+ */\n+void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len);\n+\n+#define platform_SHA_CTX SHA1_CTX\n+#define platform_SHA1_Init SHA1DCInit\n+#define platform_SHA1_Update git_SHA1DCUpdate\n+#define platform_SHA1_Final git_SHA1DCFinal\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320036","messageId":"20170517113824.31700-3-avarab@gmail.com","threadId":"45965","inReplyTo":"20170517113824.31700-1-avarab@gmail.com","subject":"[PATCH/RFC 2/3] sha1dc: use sha1collisiondetection as a submodule","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-17T11:38:23Z","receivedAt":"2017-05-17T11:38:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Now that we don't need local modifications to the code anymore with my\navar/sha1collisiondetection:easier-inclusion-in-other-programs branch\nwe can use the upstream code-as is.\n\nIf and when this change gets accepted by upstream we should of course\npoint to the https://github.com/cr-marcstevens/sha1collisiondetection\nmaster branch instead.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .gitmodules | 4 ++++\n Makefile | 8 ++++----\n hash.h | 2 +-\n sha1collisiondetection | 1 +\n 4 files changed, 10 insertions(+), 5 deletions(-)\n create mode 100644 .gitmodules\n create mode 160000 sha1collisiondetection\n\ndiff --git a/.gitmodules b/.gitmodules\nnew file mode 100644\nindex 0000000000..b60238a087\n--- /dev/null\n+++ b/.gitmodules\n@@ -0,0 +1,4 @@\n+[submodule \"sha1collisiondetection\"]\n+\tpath = sha1collisiondetection\n+\turl = https://github.com/avar/sha1collisiondetection.git\n+\tbranch = easier-inclusion-in-other-programs\ndiff --git a/Makefile b/Makefile\nindex 342466d83a..63223fd02f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1412,15 +1412,15 @@ ifdef APPLE_COMMON_CRYPTO\n \tBASIC_CFLAGS += -DSHA1_APPLE\n else\n \tDC_SHA1 := YesPlease\n-\tLIB_OBJS += sha1dc/sha1.o\n-\tLIB_OBJS += sha1dc/ubc_check.o\n+\tLIB_OBJS += sha1dc/lib/sha1.o\n+\tLIB_OBJS += sha1dc/lib/ubc_check.o\n \tBASIC_CFLAGS += \\\n \t\t-DSHA1_DC \\\n \t\t-DSHA1DC_NO_STANDARD_INCLUDES \\\n \t\t-DSHA1DC_INIT_SAFE_HASH_DEFAULT=0 \\\n \t\t-DSHA1DC_CUSTOM_INCLUDE_SHA1_C=\"\\\"cache.h\\\"\" \\\n-\t\t-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_C=\"\\\"../sha1dc_git.c\\\"\" \\\n-\t\t-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_H=\"\\\"../sha1dc_git.h\\\"\" \\\n+\t\t-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_C=\"\\\"../../sha1dc_git.c\\\"\" \\\n+\t\t-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_H=\"\\\"../../sha1dc_git.h\\\"\" \\\n \t\t-DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C=\"\\\"git-compat-util.h\\\"\"\n endif\n endif\ndiff --git a/hash.h b/hash.h\nindex a11fc9233f..e585237136 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -8,7 +8,7 @@\n #elif defined(SHA1_OPENSSL)\n #include \n #elif defined(SHA1_DC)\n-#include \"sha1dc/sha1.h\"\n+#include \"sha1dc/lib/sha1.h\"\n #else /* SHA1_BLK */\n #include \"block-sha1/sha1.h\"\n #endif\ndiff --git a/sha1collisiondetection b/sha1collisiondetection\nnew file mode 160000\nindex 0000000000..5423dd8bcf\n--- /dev/null\n+++ b/sha1collisiondetection\n@@ -0,0 +1 @@\n+Subproject commit 5423dd8bcf3cdbd029dfa08d5d5f6dc044f7ac6d\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320087","messageId":"alpine.DEB.2.21.1.1705171722440.3610@virtualbox","threadId":"45965","inReplyTo":"xmqq37c4xcr6.fsf_-_@gitster.mtv.corp.google.com","subject":"Re: [PATCH] sha1dc: fix issues with a big endian platform","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-05-17T15:26:32Z","receivedAt":"2017-05-17T15:27:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 17 May 2017, Junio C Hamano wrote:\n\n> diff --git a/sha1dc/sha1.c b/sha1dc/sha1.c\n> index 35e9dd5bf4..ae25318c47 100644\n> --- a/sha1dc/sha1.c\n> +++ b/sha1dc/sha1.c\n> @@ -20,7 +20,7 @@\n> */\n> #if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \\\n> (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) || \\\n> - defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \\\n> + defined(_BIG_ENDIAN) || defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) || \\\n> defined(_MIPSEB) || defined(__MIPSEB) || defined(__MIPSEB__)\n> \n> #define SHA1DC_BIGENDIAN\t1\n> @@ -1728,7 +1728,8 @@ void SHA1DCUpdate(SHA1_CTX* ctx, const char* buf, size_t len)\n> \twhile (len >= 64)\n> \t{\n> \t\tctx->total += 64;\n> -\t\tsha1_process(ctx, (uint32_t*)(buf));\n> +\t\tmemcpy(ctx->buffer, buf, 64);\n> +\t\tsha1_process(ctx, (uint32_t*)(ctx->buffer));\n> \t\tbuf += 64;\n\nThere is actually a discussion going on about this. See\n\nhttps://github.com/cr-marcstevens/sha1collisiondetection/commit/33a694a9ee1b79c24be45f9eab5ac0e1aeeaf271#commitcomment-22158243\n\nfor details.\n\nThe fixup commit under discussion is here:\n\nhttps://github.com/cr-marcstevens/sha1collisiondetection/commit/40f07a0c12d525b24ac1235ee8a81bbf33957ca5\n\nCiao,\ndscho\n"},{"id":"320099","messageId":"CAGZ79kZC98CxA69QjmX2s_SU6z1CSgKgwZeqvwiMRAQc6+S3xg@mail.gmail.com","threadId":"45965","inReplyTo":"20170517113824.31700-1-avarab@gmail.com","subject":"Re: [PATCH/RFC 0/3] Use sha1collisiondetection as a submodule","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T18:52:12Z","receivedAt":"2017-05-17T18:52:18Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 17, 2017 at 4:38 AM, Ævar Arnfjörð Bjarmason\n wrote:\n> On Wed, May 17, 2017 at 9:09 AM, Junio C Hamano wrote:\n>> Ævar Arnfjörð Bjarmason writes:\n>>\n>>> On Wed, May 17, 2017 at 7:39 AM, Junio C Hamano wrote:\n>>>> From: Marc Stevens \n>>>>\n>>>> Some big-endian platforms define _BIG_ENDIAN, which the test at the\n>>>> beginning of file has missed. Also, when the input is not aligned,\n>>>> some platforms trigger SIGBUS.\n>>>>\n>>>> This change corresponds to 33a694a9 (\"Fix issues with a big endian\n>>>> platform\", 2017-05-15) in the history of the upstream repository\n>>>> https://github.com/cr-marcstevens/sha1collisiondetection\n>>>\n>>> Then why not just update sha1dc from upstream instead of\n>>> cherry-picking one patch from them?\n>>\n>> See the original message upthread. I did the cherry-pick simply\n>> because that was what Marc instructed the patch recipient to do ;-).\n>\n> Since that patch is now in Marc's upstream code we can just update to\n> that.\n>\n> While we're at it let's see if Marc will take a patch so that we can\n> use his code as-is with some Makefile trickery of our own, instead of\n> having to solve merge conflicts each time we update from him. I'll\n> submit a pull request for that shortly.\n>\n> And since if and when that pull request gets accepted we're at the\n> point of being able to use the upstream code as-is & don't need to\n> locally modify it, we can just use a submodule to track it.\n\nAs someone who works on submodules: YAY! This sounds\nwonderful in the sense that more git developers experience the\nsharp edges (if any) of submodules and would fix them.\n\nWith the reset work on submodules (checkout, reset,\nls-files, grep) and more in the works (better push/pull)\nwe may be close to prime time.\n\nHowever we do distribute tarballs (well, Junio does),\nwhich is not yet supported to include submodules.\n\nI did follow the SHA1DC discussion just from the sideline,\nhow crucial is that library for us?\n\nAlso now that we discuss having submodules:\nWould we just point the submodule URL to github\\\nand call it a day?\n\nWe could make a friendly fork of it and put it next to all the repositories\nof https://git-blame.blogspot.com/p/git-public-repositories.html\nand then use relative URLs in the .gitmodules file.\n\nOn a tangent, in an off-list discussion we discussed having\nthe git-annex tests as an optional submodule instead of an\n\"external\" test, but in conclusion we considered that idea not\nideal to implement (because our tests are like contracts, we should\nactually write our own tests and not rely on downstream to write\ntests for us)\n\nlooking forward for a discussion here,\nStefan\n"},{"id":"320101","messageId":"CACBZZX6Gad32eZX6BTamNK016wu3HHLdYDB6JkUPNv=+mB3PKg@mail.gmail.com","threadId":"45965","inReplyTo":"CAGZ79kZC98CxA69QjmX2s_SU6z1CSgKgwZeqvwiMRAQc6+S3xg@mail.gmail.com","subject":"Re: [PATCH/RFC 0/3] Use sha1collisiondetection as a submodule","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-17T19:45:16Z","receivedAt":"2017-05-17T19:45:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Wed, May 17, 2017 at 8:52 PM, Stefan Beller wrote:\n> On Wed, May 17, 2017 at 4:38 AM, Ævar Arnfjörð Bjarmason\n> wrote:\n>> On Wed, May 17, 2017 at 9:09 AM, Junio C Hamano wrote:\n>>> Ævar Arnfjörð Bjarmason writes:\n>>>\n>>>> On Wed, May 17, 2017 at 7:39 AM, Junio C Hamano wrote:\n>>>>> From: Marc Stevens \n>>>>>\n>>>>> Some big-endian platforms define _BIG_ENDIAN, which the test at the\n>>>>> beginning of file has missed. Also, when the input is not aligned,\n>>>>> some platforms trigger SIGBUS.\n>>>>>\n>>>>> This change corresponds to 33a694a9 (\"Fix issues with a big endian\n>>>>> platform\", 2017-05-15) in the history of the upstream repository\n>>>>> https://github.com/cr-marcstevens/sha1collisiondetection\n>>>>\n>>>> Then why not just update sha1dc from upstream instead of\n>>>> cherry-picking one patch from them?\n>>>\n>>> See the original message upthread. I did the cherry-pick simply\n>>> because that was what Marc instructed the patch recipient to do ;-).\n>>\n>> Since that patch is now in Marc's upstream code we can just update to\n>> that.\n>>\n>> While we're at it let's see if Marc will take a patch so that we can\n>> use his code as-is with some Makefile trickery of our own, instead of\n>> having to solve merge conflicts each time we update from him. I'll\n>> submit a pull request for that shortly.\n>>\n>> And since if and when that pull request gets accepted we're at the\n>> point of being able to use the upstream code as-is & don't need to\n>> locally modify it, we can just use a submodule to track it.\n>\n> As someone who works on submodules: YAY! This sounds\n> wonderful in the sense that more git developers experience the\n> sharp edges (if any) of submodules and would fix them.\n\nYeah I agree git.git should dogfood submodules, and this seemed like a\nperfect opportunity for thaht.\n\nAs noted later in your mail everything may not be ready, so when I\nsubmit a non-RFC version of this (after upstream pulls my changes,\nhopefully), the 2nd and 3rd patches can just be dropped.\n\n> With the reset work on submodules (checkout, reset,\n> ls-files, grep) and more in the works (better push/pull)\n> we may be close to prime time.\n>\n> However we do distribute tarballs (well, Junio does),\n> which is not yet supported to include submodules.\n\nOuch. So there's no non-manual way with git-archive to make a tarball\nrelease of a git project using submodules?\n\n> I did follow the SHA1DC discussion just from the sideline,\n> how crucial is that library for us?\n\nSince 2.13.0 it's git's default sha1 implementation.\n\n> Also now that we discuss having submodules:\n> Would we just point the submodule URL to github\\\n> and call it a day?\n>\n> We could make a friendly fork of it and put it next to all the repositories\n> of https://git-blame.blogspot.com/p/git-public-repositories.html\n> and then use relative URLs in the .gitmodules file.\n\nWouldn't we lose the ability to just \"pull\" the module to see if\nupstream changed, i.e. now we'd need to manually munge the URL first.\n\nIf so having a fetchurl be a relative URL similar to git-push's\npushurl would solve it wouldn't it? I.e. a project like git could say\n\"here's my mirrors\" different from \"here's my upstream\".\n"},{"id":"320102","messageId":"CAGZ79kZXgjC84vsXtvPu_=ApJOkAwUjV1BCS2iETowF-sSrrvw@mail.gmail.com","threadId":"45965","inReplyTo":"CACBZZX6Gad32eZX6BTamNK016wu3HHLdYDB6JkUPNv=+mB3PKg@mail.gmail.com","subject":"Re: [PATCH/RFC 0/3] Use sha1collisiondetection as a submodule","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T19:57:18Z","receivedAt":"2017-05-17T19:57:40Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 17, 2017 at 12:45 PM, Ævar Arnfjörð Bjarmason\n wrote:\n> On Wed, May 17, 2017 at 8:52 PM, Stefan Beller wrote:\n>> On Wed, May 17, 2017 at 4:38 AM, Ævar Arnfjörð Bjarmason\n>> wrote:\n>>> On Wed, May 17, 2017 at 9:09 AM, Junio C Hamano wrote:\n>>>> Ævar Arnfjörð Bjarmason writes:\n>>>>\n>>>>> On Wed, May 17, 2017 at 7:39 AM, Junio C Hamano wrote:\n>>>>>> From: Marc Stevens \n>>>>>>\n>>>>>> Some big-endian platforms define _BIG_ENDIAN, which the test at the\n>>>>>> beginning of file has missed. Also, when the input is not aligned,\n>>>>>> some platforms trigger SIGBUS.\n>>>>>>\n>>>>>> This change corresponds to 33a694a9 (\"Fix issues with a big endian\n>>>>>> platform\", 2017-05-15) in the history of the upstream repository\n>>>>>> https://github.com/cr-marcstevens/sha1collisiondetection\n>>>>>\n>>>>> Then why not just update sha1dc from upstream instead of\n>>>>> cherry-picking one patch from them?\n>>>>\n>>>> See the original message upthread. I did the cherry-pick simply\n>>>> because that was what Marc instructed the patch recipient to do ;-).\n>>>\n>>> Since that patch is now in Marc's upstream code we can just update to\n>>> that.\n>>>\n>>> While we're at it let's see if Marc will take a patch so that we can\n>>> use his code as-is with some Makefile trickery of our own, instead of\n>>> having to solve merge conflicts each time we update from him. I'll\n>>> submit a pull request for that shortly.\n>>>\n>>> And since if and when that pull request gets accepted we're at the\n>>> point of being able to use the upstream code as-is & don't need to\n>>> locally modify it, we can just use a submodule to track it.\n>>\n>> As someone who works on submodules: YAY! This sounds\n>> wonderful in the sense that more git developers experience the\n>> sharp edges (if any) of submodules and would fix them.\n>\n> Yeah I agree git.git should dogfood submodules, and this seemed like a\n> perfect opportunity for thaht.\n>\n> As noted later in your mail everything may not be ready, so when I\n> submit a non-RFC version of this (after upstream pulls my changes,\n> hopefully), the 2nd and 3rd patches can just be dropped.\n>\n>> With the reset work on submodules (checkout, reset,\n>> ls-files, grep) and more in the works (better push/pull)\n>> we may be close to prime time.\n>>\n>> However we do distribute tarballs (well, Junio does),\n>> which is not yet supported to include submodules.\n>\n> Ouch. So there's no non-manual way with git-archive to make a tarball\n> release of a git project using submodules?\n\nYes. Ouch.\n\n>> We could make a friendly fork of it and put it next to all the repositories\n>> of https://git-blame.blogspot.com/p/git-public-repositories.html\n>> and then use relative URLs in the .gitmodules file.\n>\n> Wouldn't we lose the ability to just \"pull\" the module to see if\n> upstream changed, i.e. now we'd need to manually munge the URL first.\n\nI assumed more people would fetch git and trust the community rather\nthan people checking if SHA1DC is up-to-date, so I thought it is easier for\nthe minority to rewrite a URL.\n\n> If so having a fetchurl be a relative URL similar to git-push's\n> pushurl would solve it wouldn't it? I.e. a project like git could say\n> \"here's my mirrors\" different from \"here's my upstream\".\n\nI would think so.\n"},{"id":"320116","messageId":"20170518001152.GE185461@google.com","threadId":"45965","inReplyTo":"CAGZ79kZC98CxA69QjmX2s_SU6z1CSgKgwZeqvwiMRAQc6+S3xg@mail.gmail.com","subject":"Re: [PATCH/RFC 0/3] Use sha1collisiondetection as a submodule","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-05-18T00:11:52Z","receivedAt":"2017-05-18T00:12:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 05/17, Stefan Beller wrote:\n> On Wed, May 17, 2017 at 4:38 AM, Ævar Arnfjörð Bjarmason\n> wrote:\n> > On Wed, May 17, 2017 at 9:09 AM, Junio C Hamano wrote:\n> >> Ævar Arnfjörð Bjarmason writes:\n> >>\n> >>> On Wed, May 17, 2017 at 7:39 AM, Junio C Hamano wrote:\n> >>>> From: Marc Stevens \n> >>>>\n> >>>> Some big-endian platforms define _BIG_ENDIAN, which the test at the\n> >>>> beginning of file has missed. Also, when the input is not aligned,\n> >>>> some platforms trigger SIGBUS.\n> >>>>\n> >>>> This change corresponds to 33a694a9 (\"Fix issues with a big endian\n> >>>> platform\", 2017-05-15) in the history of the upstream repository\n> >>>> https://github.com/cr-marcstevens/sha1collisiondetection\n> >>>\n> >>> Then why not just update sha1dc from upstream instead of\n> >>> cherry-picking one patch from them?\n> >>\n> >> See the original message upthread. I did the cherry-pick simply\n> >> because that was what Marc instructed the patch recipient to do ;-).\n> >\n> > Since that patch is now in Marc's upstream code we can just update to\n> > that.\n> >\n> > While we're at it let's see if Marc will take a patch so that we can\n> > use his code as-is with some Makefile trickery of our own, instead of\n> > having to solve merge conflicts each time we update from him. I'll\n> > submit a pull request for that shortly.\n> >\n> > And since if and when that pull request gets accepted we're at the\n> > point of being able to use the upstream code as-is & don't need to\n> > locally modify it, we can just use a submodule to track it.\n> \n> As someone who works on submodules: YAY! This sounds\n> wonderful in the sense that more git developers experience the\n> sharp edges (if any) of submodules and would fix them.\n\nHaha, I wish I had your enthusiasm for this. That is a good argument\nfor us to include them in git.git, but I still feel apprehensive. There\nis still so much to do in submodule land! We'll make them great though!\n\n> \n> With the reset work on submodules (checkout, reset,\n> ls-files, grep) and more in the works (better push/pull)\n> we may be close to prime time.\n> \n> However we do distribute tarballs (well, Junio does),\n> which is not yet supported to include submodules.\n> \n> I did follow the SHA1DC discussion just from the sideline,\n> how crucial is that library for us?\n> \n> Also now that we discuss having submodules:\n> Would we just point the submodule URL to github\\\n> and call it a day?\n> \n> We could make a friendly fork of it and put it next to all the repositories\n> of https://git-blame.blogspot.com/p/git-public-repositories.html\n> and then use relative URLs in the .gitmodules file.\n> \n> On a tangent, in an off-list discussion we discussed having\n> the git-annex tests as an optional submodule instead of an\n> \"external\" test, but in conclusion we considered that idea not\n> ideal to implement (because our tests are like contracts, we should\n> actually write our own tests and not rely on downstream to write\n> tests for us)\n> \n> looking forward for a discussion here,\n> Stefan\n\n-- \nBrandon Williams\n"},{"id":"321285","messageId":"CANgJU+XQ4uk_OZ+bHxJ51qxAxM6Lq9yu-GNSqZ4BNiBU-8Zpqg@mail.gmail.com","threadId":"45965","inReplyTo":"20170515220939.vkgofpkdtpz7u26v@sigill.intra.peff.net","subject":"Re: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2017-06-01T14:03:08Z","receivedAt":"2017-06-01T14:03:15Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"On 16 May 2017 at 00:09, Jeff King wrote:\n> On Mon, May 15, 2017 at 04:13:58PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> On Mon, May 15, 2017 at 3:58 PM, Marc Stevens wrote:\n>> > Hi Aevar,\n>> >\n>> > Thank you for notifying us of this issue.\n>> > Big endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems.\n>> > Our github repo does check correct functioning, including an endianness mistake, with 'make test'.\n>> > But I guess this is not included for SHA1DC in Git.\n>> >\n>> > Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.\n>> >\n>> > I don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue.\n>> > But I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd.\n>> > Can you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform?\n>> > This is not problem for x86, but maybe for your platform it is?\n>> > If it is then we should always copy buffer contents to the sha1context to avoid this issue.\n>>\n>> I don't have access to the box in question, Michael was testing this\n>> code for me. But unaligned access is probably the cause, although\n>> according to some info I found online that should give a SIGBUS not a\n>> SIGSEGV, but that may have changed:\n>\n> Yeah, I would have expected SIGBUS there. If we have alignment issues,\n> though, I'd expect that ARM systems will experience problems.\n>\n> Block-sha1 uses a macro which allows unaligned loads on platforms that\n> support it, and otherwise does the endian conversion on the fly as we\n> load the bytes into a local variable (which presumably happens all\n> in-register). That may be faster than doing a mass copy of the buffer.\n\nI agree. It is fairly normal to use that kind of macro and not do a\nmemcpy with hash functions.\n\nIn fact many hash functions ONLY use that kind of macro, as decent\ncompilers will automagically convert the macro into an unaligned load\non platforms that support fast unaligned loads.\n\nThe only reason to expose the direct unaligned load is to make sure\nthat the hashing code is fast on such platforms even when compiled\nunder -g.\n\nYves\n\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"}]}