From: Jon Smirl Date: Tue, 04 Aug 2009 12:56:48 GMT Subject: Re: x86 SHA1: Faster than OpenSSL Message-ID: <9e4733910908040556t477dcb66u86a43e12148e3352@mail.gmail.com> In-Reply-To: <20090804050138.13256.qmail@science.horizon.com> On Tue, Aug 4, 2009 at 1:01 AM, George Spelvin wrote: >> Would there happen to be a SHA1 implementation around that can compute >> the SHA1 without first decompressing the data? Databases gain a lot of >> speed by using special algorithms that can directly operate on the >> compressed data. > > I can't imagine how.  In general, this requires that the compression > be carefully designed to be compatible with the algorithms, and SHA1 > is specifically designed to depend on every bit of the input in > an un-analyzable way. A simple start would be to feed each byte as it is decompressed directly into the sha code and avoid the intermediate buffer. Removing the buffer reduces cache pressure. > Also, git normally avoids hashing objects that it doesn't need > uncompressed for some other reason.  git-fsck is a notable exception, > but I think the idea of creating special optimized code paths for that > interferes with its reliability and robustness goals. Agreed that there is no real need for this, just something to play with if you are trying for a speed record. I'd much rather have a solution for the rebase problem where one side of the diff has moved to a different file and rebase can't figure it out. > -- Jon Smirl jonsmirl@gmail.com