Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1

From: Michal Suchánek <msuchanek@suse.de>
Date: Jan 16, 2023, 09:13 UTC
Message-ID: <20230116091346.GC16547@kitsune.suse.cz>
In-Reply-To: <b1984123-569a-c290-8048-158c1c5e08b4@selasky.org>

On Mon, Jan 16, 2023 at 08:17:58AM +0100, Hans Petter Selasky wrote:

Show 21 quoted lines

> On 1/15/23 14:53, Michal Suchánek wrote:
> > > Many people think that bit errors cannot happen because the memory uses ECC
> > > and the file system uses cryptographic hashes to verify the integrity of the
> > > data. But what many people forget about is that when copying data from
> > > memory to disk, typically using a DMA channel data is copied w/o any kind of
> > > integrity protection, because the integrity protection is not end-to-end.
> > > The integrity protection is only per-link.
> >
> > So long as all links have integrity protection it's end-to-end.
> >
> 
> Hi Michael,
> 
> You clearly don't see what this is about! Only if the same CRC mechanism is
> end-to-end, you don't have any good integrity mechanism at all!
> 
> Let me try to explain what this is about in very simple words. Because
> memcpy() does not copy the ECC CRC values along with the data, it is an
> unsafe memory copy mechanism, which may introduce bit-errors without
> noticing. It does not help to only have ECC RAM or for that sake protect the
> PCI links.

The ECC protects against 1bit errors - so long as only 1 bit is flipped along that path it is corrected.

If you have bigger errors ECC can sometimes detect them and your system crashes or whatever, and sometimes they go unnoticed.

It does not make sense to copy around that CRC. It is used to recover the corrupted bit, and when that data is copied to a new location a new CRC is calculated that can detect an error in that location. Copying that checksum around would only accumulate the errors.

Of course, that assumes that the corruption happens only in the cheaper external long-term storage, and data does not get corrupted as it goes through your CPU where it is stored only a few CPU cycles at a time. It is mostly the case but when you need extreme reliability system-level schemes that mitigate this possibility do exist.

Thanks

Michal