Re: SHA1 hash safety

Brian> (1) I _have_ seen real-life collisions with MD5, in the context Brian> of Document management systems containing ~10^6 ms-WORD Brian> documents.

Was this whole-document based, or was it blocked or otherwise chunked?

I'm wondering, because (SFAIK) the MS word on-disk format is some serialized version of one or more containers, possibly nested. If you're blocks are sized so that the first block is the same across multiple files, this could cause collisions -- but they're the good kind, that allow us to save disk space, so they're not a problem.

Are you saying that, within 1e7 documents, that you found two documents with the same MD5 hash yet different contents?

That's not an accusation, btw; I'm just trying to get clarity on the terminology. I'm fascinated by the idea of using this sort of content-addressable filesystem, but the chance of any collision at all wigs me out. I look at the probabilities, but still.

Thanks, t.