article: Using a rolling hash to break up binary files

Philip Oakley· Jul 31, 2014, 22:31 UTC · lore

I thought it worth bring to the list's attention a recent article on CodeProject that may be of interest to those looking at splitting binary files into deterministic hunks.

http://www.codeproject.com/Articles/801608/Using-a-rolling-hash-to-break-up-binary-files

It's based on Rabin and Karp's algorithm http://en.wikipedia.org/wiki/Rabin%E2%80%93Karp_algorithm.

-- Philip

Shawn Pearce· Aug 1, 2014, 03:26 UTC · re: Philip Oakley · lore

Re: article: Using a rolling hash to break up binary files

On Thu, Jul 31, 2014 at 3:31 PM, Philip Oakley <philipoakley@iee.org> wrote:

Show 8 quoted lines

> I thought it worth bring to the list's attention a  recent article on
> CodeProject that may be of interest to those looking at splitting binary
> files into deterministic hunks.
>
> http://www.codeproject.com/Articles/801608/Using-a-rolling-hash-to-break-up-binary-files
>
> It's based on Rabin and Karp's algorithm
> http://en.wikipedia.org/wiki/Rabin%E2%80%93Karp_algorithm.

If I remember right, this is how bup[1] works. Its certainly what we do for delta compressing files.

[1] https://github.com/bup/bup