Re: [PATCH 0/4] faster SHA-1 collision detection
- From
Sebastian Thiel <sebastian.thiel@icloud.com>
- Date
- Oct 8, 2026, 06:21 UTC
- Message-ID
- <79ae606b-cf99-4867-9db3-bcd7ff03626d@icloud.com>
- In-Reply-To
- <CAP2yMaL51H1OAG25nQ0NuLQLb0wevd4CicCG1_ezsJfrZDqfUA@mail.gmail.com>
Thanks for reeling me in, Scott!
First of all, I am very happy to see that overall, everyone here is making an effort to find a way to speed up SHA-1dc again. It's so impactful!
It really did hurt when I finally had to add SHA-1dc to Gitoxide and see the performance of clones plummet. And it still hurts me knowing that an incredible amount of CPU time is wasted doing something that we now know can be done much faster. At GitHub scale, this must be more than a blip.
While it's my dream to one day have a GitHub action that uses `gix` to clone and safe even more power, I think Git is in a far better spot to achieve significant savings much sooner.
On 07.10.26 20:13, Scott Chacon wrote:
Show 53 quoted lines
> Hey, > > On Wed, Oct 7, 2026 at 7:23 PM Junio C Hamano <gitster@pobox.com> wrote: >>> This series ports the approach of Sam Reis's sha1dc Rust crate [1], >>> which gitoxide recently switched to [2], to C. >> >> Which means license-wise the original is compatible with us, I >> presume, as they are "Apache2 or MIT, your choice". >> >> How can you/we be sure, with respect to the current AI policy in >> SubmittingPatches (which by the way was vetted by SFC lawyers), that >> your "AI generated" code did not "borrow" from places that gets >> you/us into trouble? > > It's a good question. I actually just submitted a proposed update to > that policy based on SFC's updated guidelines, but either way, I > learned about this from Sam and have talked to him about the port and > he seemed excited about it. I can triple check, but I'm fairly > confident that he's fine with this and I am fine signing off on it > under the terms of the DCO language. > > Of course, he in turn used AI tooling to produce _his_ library, but > within the guidelines of the updated SFC guidelines. Johannes's > alternative series is the original Rust code of Sam that my agent > looked at to produce this (in addition to his blog post explaining > it), so I'm not sure how that might be materially different. > >>> The end result hashes roughly 2.7x faster on the Xeon and 2.85x faster >>> on the M5 Max. Single-threaded index-pack of git.git goes from 24.3s to >>> 12.7s on the Xeon, and from 16.1s to 8.7s on the M5 Max. >>> >>> Hashing throughput on the Xeon, in MiB/s: >>> >>> 16KiB 1MiB vs OpenSSL >>> OpenSSL SHA-1 (no detection) 1234 1129 1.00x >>> sha1dc/ (today) 435 450 2.67x >>> shani+avx2 (default here) 1002 901 1.24x >>> shani+sse2 1075 1008 1.13x >>> portable+avx2 553 654 1.96x >>> portable+sse2 603 681 1.84x >>> portable 466 565 2.29x >>> >>> In other words, currently collision detection costs about 1.5–2.5x on >>> top of the hashing itself today, but only about 0.2x with the series. >> >> Thanks for these numbers. > > It would have been better had I provided the same relative scale (it > should be 1.5-2.5x vs 1.2x, but whatever, you probably get it. It's > 20% overhead here vs 50%-150% overhead previously). > >>> [1] https://sam.dev/blog/faster-sha1-collision-detection >>> [2] https://github.com/GitoxideLabs/gitoxide/pull/3008 >> >> And the pointers to the original sources. > > CC'ing Sam (sha1dc rust guy) and Sebastian (Gitoxide) on this, just in > case they have an opinion but I'm pretty sure they would be more than > happy for this to be integrated. > > Scott