Re: Compression and dictionaries

From: David Lang <dlang@digitalinsight.com>
Date: Aug 14, 2006, 16:32 UTC
Message-ID: <Pine.LNX.4.63.0608140930380.14796@qynat.qvtvafvgr.pbz>
In-Reply-To: <9e4733910608140915i728004c1p216bf3d74fcc6ab7@mail.gmail.com>

On Mon, 14 Aug 2006, Jon Smirl wrote:

Show 10 quoted lines

> On 8/14/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>> I still think that this is important to think through: Is it worth a
>> couple of kilobytes (I doubt that it would be as much as 1MB in _total_),
>> and be on the unsafe side?
>
> The only "unsafe" aspect I see to this is if the global dictionary
> doesn't contain any of the words in the documents being encoded. In
> that case the global dictionary will occupy the short huffman keys
> forcing longer internal keys.  The keys for the words in the document
> would be longer by a about a bit on average.

the other factor that was mentioned was that a single-bit corruption in the dictionary would make the entire pack file useless. if this is really a concern then just store multiple copies of the dictionary. on a pack with lots of files in it it can still be a significant win.

David Lang