Compression and dictionaries
- From
Jon Smirl <jonsmirl@gmail.com>
- Date
- Aug 14, 2006, 03:37 UTC
- Message-ID
- <9e4733910608132037t4297c3bbq9b0cd6ebaa03b979@mail.gmail.com>
>From what I remember from long ago most compression schemes build
dictionaries as a way of achieving significant compression. If so, since we zlib compress each entry in a pack individually, are there many copies of very similar dictionaries in the pack?
Some compression schemes support being initialized with a fixed dictionary and sharing it over all entries. A fixed dictionary could be built by analysing a large pack file. Sharing a compression dictionary would probably be a win for me since I have 1M+ entries in a pack.
Poking around in the zlib it appears that zlib supports precomputed dictionaries. http://www.zlib.net/manual.html
int deflateSetDictionary (z_streamp strm, const Bytef *dictionary, uInt dictLength);
-- Jon Smirl jonsmirl@gmail.com